[==========] Running 7 tests from 2 test suites.
[----------] Global test environment set-up.
[----------] 5 tests from LeaderRebalancerTest
[ RUN ] LeaderRebalancerTest.FunctionalTestForDivided
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20260809 15:44:46.169345 21289 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.20.202.126:37791
I20260809 15:44:46.170578 21289 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 15:44:46.171571 21289 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:44:46.189322 21304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:44:46.189323 21299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:44:46.189491 21297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:44:46.191365 21289 server_base.cc:1034] running on GCE node
I20260809 15:44:47.296154 21289 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260809 15:44:47.296366 21289 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260809 15:44:47.296530 21289 hybrid_clock.cc:648] HybridClock initialized: now 1786290287296509 us; error 0 us; skew 500 ppm
I20260809 15:44:47.297319 21289 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 15:44:47.307130 21289 webserver.cc:460] Webserver started at http://127.20.202.126:32881/ using document root <none> and password file <none>
I20260809 15:44:47.307991 21289 fs_manager.cc:359] Metadata directory not provided
I20260809 15:44:47.308195 21289 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:44:47.308612 21289 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 15:44:47.314674 21289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj2AJ_N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786290285951082-21289-0/minicluster-data/master-0-root/instance:
uuid: "5f69f1903e1d4f6f932244dd3880569c"
format_stamp: "Formatted at 2026-08-09 15:44:47 on dist-test-slave-gjrz"
I20260809 15:44:47.321960 21289 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260809 15:44:47.326817 21312 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:44:47.327843 21289 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 15:44:47.328161 21289 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj2AJ_N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786290285951082-21289-0/minicluster-data/master-0-root
uuid: "5f69f1903e1d4f6f932244dd3880569c"
format_stamp: "Formatted at 2026-08-09 15:44:47 on dist-test-slave-gjrz"
I20260809 15:44:47.328464 21289 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj2AJ_N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786290285951082-21289-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskj2AJ_N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786290285951082-21289-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskj2AJ_N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786290285951082-21289-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:44:47.365340 21289 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:44:47.366690 21289 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 15:44:47.367110 21289 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:44:47.435395 21289 rpc_server.cc:307] RPC server started. Bound to: 127.20.202.126:37791
I20260809 15:44:47.435475 21401 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.202.126:37791 every 8 connection(s)
I20260809 15:44:47.441991 21405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:44:47.464928 21405 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f69f1903e1d4f6f932244dd3880569c: Bootstrap starting.
I20260809 15:44:47.470184 21405 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5f69f1903e1d4f6f932244dd3880569c: Neither blocks nor log segments found. Creating new log.
I20260809 15:44:47.471921 21405 log.cc:824] T 00000000000000000000000000000000 P 5f69f1903e1d4f6f932244dd3880569c: Log is configured to *not* fsync() on all Append() calls
I20260809 15:44:47.476094 21405 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f69f1903e1d4f6f932244dd3880569c: No bootstrap required, opened a new log
I20260809 15:44:47.498912 21405 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5f69f1903e1d4f6f932244dd3880569c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f69f1903e1d4f6f932244dd3880569c" member_type: VOTER }
I20260809 15:44:47.499584 21405 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5f69f1903e1d4f6f932244dd3880569c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:44:47.499859 21405 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5f69f1903e1d4f6f932244dd3880569c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f69f1903e1d4f6f932244dd3880569c, State: Initialized, Role: FOLLOWER
I20260809 15:44:47.500761 21405 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5f69f1903e1d4f6f932244dd3880569c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f69f1903e1d4f6f932244dd3880569c" member_type: VOTER }
I20260809 15:44:47.501366 21405 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5f69f1903e1d4f6f932244dd3880569c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 15:44:47.501677 21405 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5f69f1903e1d4f6f932244dd3880569c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 15:44:47.502051 21405 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5f69f1903e1d4f6f932244dd3880569c [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:44:47.506668 21405 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5f69f1903e1d4f6f932244dd3880569c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f69f1903e1d4f6f932244dd3880569c" member_type: VOTER }
I20260809 15:44:47.507401 21405 leader_election.cc:302] T 00000000000000000000000000000000 P 5f69f1903e1d4f6f932244dd3880569c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5f69f1903e1d4f6f932244dd3880569c; no voters:
I20260809 15:44:47.509457 21405 leader_election.cc:288] T 00000000000000000000000000000000 P 5f69f1903e1d4f6f932244dd3880569c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 15:44:47.509752 21410 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5f69f1903e1d4f6f932244dd3880569c [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:44:47.511864 21410 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5f69f1903e1d4f6f932244dd3880569c [term 1 LEADER]: Becoming Leader. State: Replica: 5f69f1903e1d4f6f932244dd3880569c, State: Running, Role: LEADER
I20260809 15:44:47.512672 21410 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5f69f1903e1d4f6f932244dd3880569c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f69f1903e1d4f6f932244dd3880569c" member_type: VOTER }
I20260809 15:44:47.513614 21405 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5f69f1903e1d4f6f932244dd3880569c [sys.catalog]: configured and running, proceeding with master startup.
I20260809 15:44:47.521746 21412 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f69f1903e1d4f6f932244dd3880569c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5f69f1903e1d4f6f932244dd3880569c. Latest consensus state: current_term: 1 leader_uuid: "5f69f1903e1d4f6f932244dd3880569c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f69f1903e1d4f6f932244dd3880569c" member_type: VOTER } }
I20260809 15:44:47.522394 21412 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f69f1903e1d4f6f932244dd3880569c [sys.catalog]: This master's current role is: LEADER
I20260809 15:44:47.523967 21411 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f69f1903e1d4f6f932244dd3880569c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5f69f1903e1d4f6f932244dd3880569c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f69f1903e1d4f6f932244dd3880569c" member_type: VOTER } }
I20260809 15:44:47.524502 21411 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f69f1903e1d4f6f932244dd3880569c [sys.catalog]: This master's current role is: LEADER
I20260809 15:44:47.528313 21426 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 15:44:47.539544 21426 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 15:44:47.541270 21289 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260809 15:44:47.554350 21426 catalog_manager.cc:1347] Generated new cluster ID: dd6c78a3e4194679ae0232c814d43e8b
I20260809 15:44:47.554611 21426 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 15:44:47.569970 21426 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 15:44:47.572412 21426 catalog_manager.cc:1504] Loading token signing keys...
I20260809 15:44:47.588871 21426 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5f69f1903e1d4f6f932244dd3880569c: Generated new TSK 0
I20260809 15:44:47.589670 21426 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 15:44:47.607918 21289 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:44:47.616714 21439 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:44:47.617823 21443 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:44:47.621410 21289 server_base.cc:1034] running on GCE node
W20260809 15:44:47.621788 21446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:44:47.622576 21289 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260809 15:44:47.622763 21289 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260809 15:44:47.622961 21289 hybrid_clock.cc:648] HybridClock initialized: now 1786290287622925 us; error 0 us; skew 500 ppm
I20260809 15:44:47.623543 21289 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 15:44:47.627663 21289 webserver.cc:460] Webserver started at http://127.20.202.65:44777/ using document root <none> and password file <none>
I20260809 15:44:47.628167 21289 fs_manager.cc:359] Metadata directory not provided
I20260809 15:44:47.628348 21289 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:44:47.628592 21289 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 15:44:47.629773 21289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj2AJ_N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786290285951082-21289-0/minicluster-data/ts-0-root/instance:
uuid: "faa9b2a805cd43eca626c68bc7e7168f"
format_stamp: "Formatted at 2026-08-09 15:44:47 on dist-test-slave-gjrz"
I20260809 15:44:47.635063 21289 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260809 15:44:47.639168 21454 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:44:47.639958 21289 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 15:44:47.640209 21289 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj2AJ_N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786290285951082-21289-0/minicluster-data/ts-0-root
uuid: "faa9b2a805cd43eca626c68bc7e7168f"
format_stamp: "Formatted at 2026-08-09 15:44:47 on dist-test-slave-gjrz"
I20260809 15:44:47.640465 21289 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj2AJ_N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786290285951082-21289-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskj2AJ_N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786290285951082-21289-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskj2AJ_N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786290285951082-21289-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:44:47.668334 21289 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:44:47.669461 21289 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:44:47.671962 21289 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 15:44:47.676275 21289 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 15:44:47.676474 21289 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 15:44:47.676721 21289 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 15:44:47.676872 21289 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 15:44:47.771807 21289 rpc_server.cc:307] RPC server started. Bound to: 127.20.202.65:37685
I20260809 15:44:47.772013 21553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.202.65:37685 every 8 connection(s)
I20260809 15:44:47.778798 21289 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:44:47.788354 21560 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:44:47.790802 21561 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:44:47.798993 21289 server_base.cc:1034] running on GCE node
W20260809 15:44:47.799158 21565 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:44:47.800222 21289 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260809 15:44:47.800503 21289 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260809 15:44:47.800710 21289 hybrid_clock.cc:648] HybridClock initialized: now 1786290287800690 us; error 0 us; skew 500 ppm
I20260809 15:44:47.801456 21289 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 15:44:47.805238 21554 heartbeater.cc:347] Connected to a master server at 127.20.202.126:37791
I20260809 15:44:47.805652 21554 heartbeater.cc:464] Registering TS with master...
I20260809 15:44:47.805832 21289 webserver.cc:460] Webserver started at http://127.20.202.66:46057/ using document root <none> and password file <none>
I20260809 15:44:47.806377 21289 fs_manager.cc:359] Metadata directory not provided
I20260809 15:44:47.806566 21289 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:44:47.806660 21554 heartbeater.cc:511] Master 127.20.202.126:37791 requested a full tablet report, sending...
I20260809 15:44:47.806828 21289 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 15:44:47.808108 21289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj2AJ_N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786290285951082-21289-0/minicluster-data/ts-1-root/instance:
uuid: "6517ff96fe0a42a2bde064dc7f1ecc97"
format_stamp: "Formatted at 2026-08-09 15:44:47 on dist-test-slave-gjrz"
I20260809 15:44:47.809818 21341 ts_manager.cc:194] Registered new tserver with Master: faa9b2a805cd43eca626c68bc7e7168f (127.20.202.65:37685)
I20260809 15:44:47.812863 21341 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:54680
I20260809 15:44:47.814517 21289 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.002s
I20260809 15:44:47.820016 21573 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:44:47.820773 21289 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260809 15:44:47.821022 21289 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj2AJ_N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786290285951082-21289-0/minicluster-data/ts-1-root
uuid: "6517ff96fe0a42a2bde064dc7f1ecc97"
format_stamp: "Formatted at 2026-08-09 15:44:47 on dist-test-slave-gjrz"
I20260809 15:44:47.821250 21289 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj2AJ_N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786290285951082-21289-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskj2AJ_N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786290285951082-21289-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskj2AJ_N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786290285951082-21289-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:44:47.855386 21289 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:44:47.856431 21289 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:44:47.858734 21289 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 15:44:47.861460 21289 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 15:44:47.861629 21289 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 15:44:47.861845 21289 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 15:44:47.861995 21289 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 15:44:47.956318 21289 rpc_server.cc:307] RPC server started. Bound to: 127.20.202.66:42701
I20260809 15:44:47.956430 21683 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.202.66:42701 every 8 connection(s)
I20260809 15:44:47.962389 21289 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:44:47.970391 21695 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:44:47.973054 21696 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:44:47.978767 21688 heartbeater.cc:347] Connected to a master server at 127.20.202.126:37791
I20260809 15:44:47.979125 21688 heartbeater.cc:464] Registering TS with master...
I20260809 15:44:47.979961 21688 heartbeater.cc:511] Master 127.20.202.126:37791 requested a full tablet report, sending...
I20260809 15:44:47.982276 21341 ts_manager.cc:194] Registered new tserver with Master: 6517ff96fe0a42a2bde064dc7f1ecc97 (127.20.202.66:42701)
I20260809 15:44:47.984372 21341 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:54692
W20260809 15:44:47.985363 21702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:44:47.986601 21289 server_base.cc:1034] running on GCE node
I20260809 15:44:47.987558 21289 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260809 15:44:47.987735 21289 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260809 15:44:47.987907 21289 hybrid_clock.cc:648] HybridClock initialized: now 1786290287987892 us; error 0 us; skew 500 ppm
I20260809 15:44:47.988394 21289 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 15:44:47.990726 21289 webserver.cc:460] Webserver started at http://127.20.202.67:45517/ using document root <none> and password file <none>
I20260809 15:44:47.991199 21289 fs_manager.cc:359] Metadata directory not provided
I20260809 15:44:47.991429 21289 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:44:47.991683 21289 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 15:44:47.992666 21289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj2AJ_N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786290285951082-21289-0/minicluster-data/ts-2-root/instance:
uuid: "dee2d58fd60f4b39a553c007f76ac760"
format_stamp: "Formatted at 2026-08-09 15:44:47 on dist-test-slave-gjrz"
I20260809 15:44:47.997052 21289 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260809 15:44:48.000221 21708 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:44:48.000960 21289 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 15:44:48.001207 21289 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj2AJ_N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786290285951082-21289-0/minicluster-data/ts-2-root
uuid: "dee2d58fd60f4b39a553c007f76ac760"
format_stamp: "Formatted at 2026-08-09 15:44:47 on dist-test-slave-gjrz"
I20260809 15:44:48.001457 21289 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj2AJ_N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786290285951082-21289-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskj2AJ_N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786290285951082-21289-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskj2AJ_N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786290285951082-21289-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:44:48.024551 21289 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:44:48.025770 21289 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:44:48.027045 21289 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 15:44:48.029253 21289 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 15:44:48.029439 21289 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 15:44:48.029652 21289 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 15:44:48.029795 21289 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 15:44:48.116079 21289 rpc_server.cc:307] RPC server started. Bound to: 127.20.202.67:36447
I20260809 15:44:48.116195 21813 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.202.67:36447 every 8 connection(s)
I20260809 15:44:48.135543 21816 heartbeater.cc:347] Connected to a master server at 127.20.202.126:37791
I20260809 15:44:48.135946 21816 heartbeater.cc:464] Registering TS with master...
I20260809 15:44:48.136780 21816 heartbeater.cc:511] Master 127.20.202.126:37791 requested a full tablet report, sending...
I20260809 15:44:48.138772 21341 ts_manager.cc:194] Registered new tserver with Master: dee2d58fd60f4b39a553c007f76ac760 (127.20.202.67:36447)
I20260809 15:44:48.139801 21289 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.016635913s
I20260809 15:44:48.140493 21341 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:54700
I20260809 15:44:48.142517 21289 test_util.cc:268] Using random seed: 504834206
I20260809 15:44:48.191684 21341 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54716:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q""\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 15:44:48.195122 21341 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 15:44:48.306349 21498 tablet_service.cc:1467] Processing CreateTablet for tablet d203a1017c6b44d0a62bbc654ee21d8c (DEFAULT_TABLE table=test-workload [id=20fba09ac2564fa7a69b7cc5f458d990]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260809 15:44:48.307945 21498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d203a1017c6b44d0a62bbc654ee21d8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:44:48.309546 21497 tablet_service.cc:1467] Processing CreateTablet for tablet 8d9c4e17e76743439089a2a53c45d01d (DEFAULT_TABLE table=test-workload [id=20fba09ac2564fa7a69b7cc5f458d990]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260809 15:44:48.310135 21496 tablet_service.cc:1467] Processing CreateTablet for tablet d18bbcc7ebd14b479ddacb4aad814d47 (DEFAULT_TABLE table=test-workload [id=20fba09ac2564fa7a69b7cc5f458d990]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260809 15:44:48.310817 21497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d9c4e17e76743439089a2a53c45d01d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:44:48.311496 21496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d18bbcc7ebd14b479ddacb4aad814d47. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:44:48.313552 21622 tablet_service.cc:1467] Processing CreateTablet for tablet d203a1017c6b44d0a62bbc654ee21d8c (DEFAULT_TABLE table=test-workload [id=20fba09ac2564fa7a69b7cc5f458d990]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260809 15:44:48.314759 21622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d203a1017c6b44d0a62bbc654ee21d8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:44:48.316361 21617 tablet_service.cc:1467] Processing CreateTablet for tablet ae87ef2626584e6bacd9f1af2c8991d8 (DEFAULT_TABLE table=test-workload [id=20fba09ac2564fa7a69b7cc5f458d990]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260809 15:44:48.316494 21621 tablet_service.cc:1467] Processing CreateTablet for tablet 8d9c4e17e76743439089a2a53c45d01d (DEFAULT_TABLE table=test-workload [id=20fba09ac2564fa7a69b7cc5f458d990]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260809 15:44:48.317638 21617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae87ef2626584e6bacd9f1af2c8991d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:44:48.318387 21619 tablet_service.cc:1467] Processing CreateTablet for tablet d18bbcc7ebd14b479ddacb4aad814d47 (DEFAULT_TABLE table=test-workload [id=20fba09ac2564fa7a69b7cc5f458d990]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260809 15:44:48.320364 21618 tablet_service.cc:1467] Processing CreateTablet for tablet 4fe458f602674472bedadfdd3c7ee1e6 (DEFAULT_TABLE table=test-workload [id=20fba09ac2564fa7a69b7cc5f458d990]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260809 15:44:48.320466 21613 tablet_service.cc:1467] Processing CreateTablet for tablet ab976a6064754a689b7c88c5fa968b86 (DEFAULT_TABLE table=test-workload [id=20fba09ac2564fa7a69b7cc5f458d990]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260809 15:44:48.322464 21615 tablet_service.cc:1467] Processing CreateTablet for tablet 724982e80d3b455a91f8a564d374a65e (DEFAULT_TABLE table=test-workload [id=20fba09ac2564fa7a69b7cc5f458d990]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260809 15:44:48.322559 21491 tablet_service.cc:1467] Processing CreateTablet for tablet ab976a6064754a689b7c88c5fa968b86 (DEFAULT_TABLE table=test-workload [id=20fba09ac2564fa7a69b7cc5f458d990]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260809 15:44:48.322813 21609 tablet_service.cc:1467] Processing CreateTablet for tablet 6dd9896939284d9cb2c03e9c21a92dfa (DEFAULT_TABLE table=test-workload [id=20fba09ac2564fa7a69b7cc5f458d990]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260809 15:44:48.323894 21491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab976a6064754a689b7c88c5fa968b86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:44:48.324501 21610 tablet_service.cc:1467] Processing CreateTablet for tablet 2f46e8d793294ce384acbf2f5584ecd2 (DEFAULT_TABLE table=test-workload [id=20fba09ac2564fa7a69b7cc5f458d990]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260809 15:44:48.326452 21495 tablet_service.cc:1467] Processing CreateTablet for tablet 4fe458f602674472bedadfdd3c7ee1e6 (DEFAULT_TABLE table=test-workload [id=20fba09ac2564fa7a69b7cc5f458d990]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260809 15:44:48.327811 21495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4fe458f602674472bedadfdd3c7ee1e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:44:48.332544 21493 tablet_service.cc:1467] Processing CreateTablet for tablet ae87ef2626584e6bacd9f1af2c8991d8 (DEFAULT_TABLE table=test-workload [id=20fba09ac2564fa7a69b7cc5f458d990]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260809 15:44:48.333498 21489 tablet_service.cc:1467] Processing CreateTablet for tablet 6dd9896939284d9cb2c03e9c21a92dfa (DEFAULT_TABLE table=test-workload [id=20fba09ac2564fa7a69b7cc5f458d990]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260809 15:44:48.335356 21492 tablet_service.cc:1467] Processing CreateTablet for tablet 724982e80d3b455a91f8a564d374a65e (DEFAULT_TABLE table=test-workload [id=20fba09ac2564fa7a69b7cc5f458d990]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260809 15:44:48.337786 21490 tablet_service.cc:1467] Processing CreateTablet for tablet 2f46e8d793294ce384acbf2f5584ecd2 (DEFAULT_TABLE table=test-workload [id=20fba09ac2564fa7a69b7cc5f458d990]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260809 15:44:48.339560 21619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d18bbcc7ebd14b479ddacb4aad814d47. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:44:48.339684 21850 tablet_bootstrap.cc:492] T d203a1017c6b44d0a62bbc654ee21d8c P faa9b2a805cd43eca626c68bc7e7168f: Bootstrap starting.
I20260809 15:44:48.341842 21744 tablet_service.cc:1467] Processing CreateTablet for tablet 2f46e8d793294ce384acbf2f5584ecd2 (DEFAULT_TABLE table=test-workload [id=20fba09ac2564fa7a69b7cc5f458d990]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260809 15:44:48.342178 21755 tablet_service.cc:1467] Processing CreateTablet for tablet d203a1017c6b44d0a62bbc654ee21d8c (DEFAULT_TABLE table=test-workload [id=20fba09ac2564fa7a69b7cc5f458d990]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260809 15:44:48.343096 21618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4fe458f602674472bedadfdd3c7ee1e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:44:48.343145 21744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f46e8d793294ce384acbf2f5584ecd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:44:48.344394 21753 tablet_service.cc:1467] Processing CreateTablet for tablet 8d9c4e17e76743439089a2a53c45d01d (DEFAULT_TABLE table=test-workload [id=20fba09ac2564fa7a69b7cc5f458d990]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260809 15:44:48.345454 21613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab976a6064754a689b7c88c5fa968b86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:44:48.345669 21753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d9c4e17e76743439089a2a53c45d01d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:44:48.348009 21615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 724982e80d3b455a91f8a564d374a65e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:44:48.348476 21752 tablet_service.cc:1467] Processing CreateTablet for tablet d18bbcc7ebd14b479ddacb4aad814d47 (DEFAULT_TABLE table=test-workload [id=20fba09ac2564fa7a69b7cc5f458d990]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260809 15:44:48.333880 21493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae87ef2626584e6bacd9f1af2c8991d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:44:48.356534 21609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6dd9896939284d9cb2c03e9c21a92dfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:44:48.358429 21755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d203a1017c6b44d0a62bbc654ee21d8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:44:48.358564 21610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f46e8d793294ce384acbf2f5584ecd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:44:48.362607 21751 tablet_service.cc:1467] Processing CreateTablet for tablet 4fe458f602674472bedadfdd3c7ee1e6 (DEFAULT_TABLE table=test-workload [id=20fba09ac2564fa7a69b7cc5f458d990]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260809 15:44:48.366154 21621 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d9c4e17e76743439089a2a53c45d01d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:44:48.367300 21489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6dd9896939284d9cb2c03e9c21a92dfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:44:48.356688 21740 tablet_service.cc:1467] Processing CreateTablet for tablet 6dd9896939284d9cb2c03e9c21a92dfa (DEFAULT_TABLE table=test-workload [id=20fba09ac2564fa7a69b7cc5f458d990]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260809 15:44:48.360862 21747 tablet_service.cc:1467] Processing CreateTablet for tablet 724982e80d3b455a91f8a564d374a65e (DEFAULT_TABLE table=test-workload [id=20fba09ac2564fa7a69b7cc5f458d990]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260809 15:44:48.374292 21751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4fe458f602674472bedadfdd3c7ee1e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:44:48.382573 21490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f46e8d793294ce384acbf2f5584ecd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:44:48.404495 21747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 724982e80d3b455a91f8a564d374a65e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:44:48.381752 21749 tablet_service.cc:1467] Processing CreateTablet for tablet ae87ef2626584e6bacd9f1af2c8991d8 (DEFAULT_TABLE table=test-workload [id=20fba09ac2564fa7a69b7cc5f458d990]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260809 15:44:48.406544 21492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 724982e80d3b455a91f8a564d374a65e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:44:48.402752 21745 tablet_service.cc:1467] Processing CreateTablet for tablet ab976a6064754a689b7c88c5fa968b86 (DEFAULT_TABLE table=test-workload [id=20fba09ac2564fa7a69b7cc5f458d990]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260809 15:44:48.418602 21752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d18bbcc7ebd14b479ddacb4aad814d47. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:44:48.443665 21740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6dd9896939284d9cb2c03e9c21a92dfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:44:48.445243 21745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab976a6064754a689b7c88c5fa968b86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:44:48.447758 21749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae87ef2626584e6bacd9f1af2c8991d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:44:48.477553 21853 tablet_bootstrap.cc:492] T ae87ef2626584e6bacd9f1af2c8991d8 P 6517ff96fe0a42a2bde064dc7f1ecc97: Bootstrap starting.
I20260809 15:44:48.484488 21850 tablet_bootstrap.cc:654] T d203a1017c6b44d0a62bbc654ee21d8c P faa9b2a805cd43eca626c68bc7e7168f: Neither blocks nor log segments found. Creating new log.
I20260809 15:44:48.530279 21850 tablet_bootstrap.cc:492] T d203a1017c6b44d0a62bbc654ee21d8c P faa9b2a805cd43eca626c68bc7e7168f: No bootstrap required, opened a new log
W20260809 15:44:48.530366 21424 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260809 15:44:48.530943 21850 ts_tablet_manager.cc:1397] T d203a1017c6b44d0a62bbc654ee21d8c P faa9b2a805cd43eca626c68bc7e7168f: Time spent bootstrapping tablet: real 0.192s user 0.021s sys 0.011s
I20260809 15:44:48.542897 21850 raft_consensus.cc:348] T d203a1017c6b44d0a62bbc654ee21d8c P faa9b2a805cd43eca626c68bc7e7168f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } }
I20260809 15:44:48.558568 21850 raft_consensus.cc:374] T d203a1017c6b44d0a62bbc654ee21d8c P faa9b2a805cd43eca626c68bc7e7168f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:44:48.558861 21850 raft_consensus.cc:729] T d203a1017c6b44d0a62bbc654ee21d8c P faa9b2a805cd43eca626c68bc7e7168f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: faa9b2a805cd43eca626c68bc7e7168f, State: Initialized, Role: FOLLOWER
I20260809 15:44:48.563330 21850 consensus_queue.cc:260] T d203a1017c6b44d0a62bbc654ee21d8c P faa9b2a805cd43eca626c68bc7e7168f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } }
I20260809 15:44:48.580350 21856 tablet_bootstrap.cc:492] T 6dd9896939284d9cb2c03e9c21a92dfa P dee2d58fd60f4b39a553c007f76ac760: Bootstrap starting.
I20260809 15:44:48.588605 21853 tablet_bootstrap.cc:654] T ae87ef2626584e6bacd9f1af2c8991d8 P 6517ff96fe0a42a2bde064dc7f1ecc97: Neither blocks nor log segments found. Creating new log.
I20260809 15:44:48.600421 21554 heartbeater.cc:503] Master 127.20.202.126:37791 was elected leader, sending a full tablet report...
I20260809 15:44:48.607636 21850 ts_tablet_manager.cc:1428] T d203a1017c6b44d0a62bbc654ee21d8c P faa9b2a805cd43eca626c68bc7e7168f: Time spent starting tablet: real 0.074s user 0.002s sys 0.016s
I20260809 15:44:48.610549 21850 tablet_bootstrap.cc:492] T ab976a6064754a689b7c88c5fa968b86 P faa9b2a805cd43eca626c68bc7e7168f: Bootstrap starting.
I20260809 15:44:48.612634 21856 tablet_bootstrap.cc:654] T 6dd9896939284d9cb2c03e9c21a92dfa P dee2d58fd60f4b39a553c007f76ac760: Neither blocks nor log segments found. Creating new log.
I20260809 15:44:48.620406 21850 tablet_bootstrap.cc:654] T ab976a6064754a689b7c88c5fa968b86 P faa9b2a805cd43eca626c68bc7e7168f: Neither blocks nor log segments found. Creating new log.
I20260809 15:44:48.641201 21853 tablet_bootstrap.cc:492] T ae87ef2626584e6bacd9f1af2c8991d8 P 6517ff96fe0a42a2bde064dc7f1ecc97: No bootstrap required, opened a new log
I20260809 15:44:48.641608 21853 ts_tablet_manager.cc:1397] T ae87ef2626584e6bacd9f1af2c8991d8 P 6517ff96fe0a42a2bde064dc7f1ecc97: Time spent bootstrapping tablet: real 0.224s user 0.011s sys 0.013s
I20260809 15:44:48.642002 21856 tablet_bootstrap.cc:492] T 6dd9896939284d9cb2c03e9c21a92dfa P dee2d58fd60f4b39a553c007f76ac760: No bootstrap required, opened a new log
I20260809 15:44:48.642409 21856 ts_tablet_manager.cc:1397] T 6dd9896939284d9cb2c03e9c21a92dfa P dee2d58fd60f4b39a553c007f76ac760: Time spent bootstrapping tablet: real 0.073s user 0.007s sys 0.006s
I20260809 15:44:48.644119 21853 raft_consensus.cc:348] T ae87ef2626584e6bacd9f1af2c8991d8 P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } }
I20260809 15:44:48.644821 21853 raft_consensus.cc:374] T ae87ef2626584e6bacd9f1af2c8991d8 P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:44:48.645030 21850 tablet_bootstrap.cc:492] T ab976a6064754a689b7c88c5fa968b86 P faa9b2a805cd43eca626c68bc7e7168f: No bootstrap required, opened a new log
I20260809 15:44:48.645123 21853 raft_consensus.cc:729] T ae87ef2626584e6bacd9f1af2c8991d8 P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6517ff96fe0a42a2bde064dc7f1ecc97, State: Initialized, Role: FOLLOWER
I20260809 15:44:48.644927 21856 raft_consensus.cc:348] T 6dd9896939284d9cb2c03e9c21a92dfa P dee2d58fd60f4b39a553c007f76ac760 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } }
I20260809 15:44:48.645701 21856 raft_consensus.cc:374] T 6dd9896939284d9cb2c03e9c21a92dfa P dee2d58fd60f4b39a553c007f76ac760 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:44:48.645725 21850 ts_tablet_manager.cc:1397] T ab976a6064754a689b7c88c5fa968b86 P faa9b2a805cd43eca626c68bc7e7168f: Time spent bootstrapping tablet: real 0.035s user 0.004s sys 0.009s
I20260809 15:44:48.646111 21856 raft_consensus.cc:729] T 6dd9896939284d9cb2c03e9c21a92dfa P dee2d58fd60f4b39a553c007f76ac760 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dee2d58fd60f4b39a553c007f76ac760, State: Initialized, Role: FOLLOWER
I20260809 15:44:48.645925 21853 consensus_queue.cc:260] T ae87ef2626584e6bacd9f1af2c8991d8 P 6517ff96fe0a42a2bde064dc7f1ecc97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } }
I20260809 15:44:48.646814 21856 consensus_queue.cc:260] T 6dd9896939284d9cb2c03e9c21a92dfa P dee2d58fd60f4b39a553c007f76ac760 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } }
I20260809 15:44:48.648900 21850 raft_consensus.cc:348] T ab976a6064754a689b7c88c5fa968b86 P faa9b2a805cd43eca626c68bc7e7168f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } }
I20260809 15:44:48.649700 21850 raft_consensus.cc:374] T ab976a6064754a689b7c88c5fa968b86 P faa9b2a805cd43eca626c68bc7e7168f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:44:48.650043 21850 raft_consensus.cc:729] T ab976a6064754a689b7c88c5fa968b86 P faa9b2a805cd43eca626c68bc7e7168f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: faa9b2a805cd43eca626c68bc7e7168f, State: Initialized, Role: FOLLOWER
I20260809 15:44:48.650816 21850 consensus_queue.cc:260] T ab976a6064754a689b7c88c5fa968b86 P faa9b2a805cd43eca626c68bc7e7168f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } }
I20260809 15:44:48.653311 21850 ts_tablet_manager.cc:1428] T ab976a6064754a689b7c88c5fa968b86 P faa9b2a805cd43eca626c68bc7e7168f: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260809 15:44:48.654188 21850 tablet_bootstrap.cc:492] T d18bbcc7ebd14b479ddacb4aad814d47 P faa9b2a805cd43eca626c68bc7e7168f: Bootstrap starting.
I20260809 15:44:48.660342 21688 heartbeater.cc:503] Master 127.20.202.126:37791 was elected leader, sending a full tablet report...
I20260809 15:44:48.663911 21853 ts_tablet_manager.cc:1428] T ae87ef2626584e6bacd9f1af2c8991d8 P 6517ff96fe0a42a2bde064dc7f1ecc97: Time spent starting tablet: real 0.022s user 0.008s sys 0.004s
I20260809 15:44:48.665135 21853 tablet_bootstrap.cc:492] T d18bbcc7ebd14b479ddacb4aad814d47 P 6517ff96fe0a42a2bde064dc7f1ecc97: Bootstrap starting.
I20260809 15:44:48.670786 21853 tablet_bootstrap.cc:654] T d18bbcc7ebd14b479ddacb4aad814d47 P 6517ff96fe0a42a2bde064dc7f1ecc97: Neither blocks nor log segments found. Creating new log.
I20260809 15:44:48.672003 21816 heartbeater.cc:503] Master 127.20.202.126:37791 was elected leader, sending a full tablet report...
I20260809 15:44:48.672464 21856 ts_tablet_manager.cc:1428] T 6dd9896939284d9cb2c03e9c21a92dfa P dee2d58fd60f4b39a553c007f76ac760: Time spent starting tablet: real 0.030s user 0.011s sys 0.019s
I20260809 15:44:48.673071 21850 tablet_bootstrap.cc:654] T d18bbcc7ebd14b479ddacb4aad814d47 P faa9b2a805cd43eca626c68bc7e7168f: Neither blocks nor log segments found. Creating new log.
I20260809 15:44:48.673283 21856 tablet_bootstrap.cc:492] T ab976a6064754a689b7c88c5fa968b86 P dee2d58fd60f4b39a553c007f76ac760: Bootstrap starting.
I20260809 15:44:48.678504 21866 raft_consensus.cc:482] T 6dd9896939284d9cb2c03e9c21a92dfa P dee2d58fd60f4b39a553c007f76ac760 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:44:48.679050 21866 raft_consensus.cc:504] T 6dd9896939284d9cb2c03e9c21a92dfa P dee2d58fd60f4b39a553c007f76ac760 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } }
I20260809 15:44:48.684171 21866 leader_election.cc:288] T 6dd9896939284d9cb2c03e9c21a92dfa P dee2d58fd60f4b39a553c007f76ac760 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers faa9b2a805cd43eca626c68bc7e7168f (127.20.202.65:37685), 6517ff96fe0a42a2bde064dc7f1ecc97 (127.20.202.66:42701)
I20260809 15:44:48.687757 21853 tablet_bootstrap.cc:492] T d18bbcc7ebd14b479ddacb4aad814d47 P 6517ff96fe0a42a2bde064dc7f1ecc97: No bootstrap required, opened a new log
I20260809 15:44:48.688450 21853 ts_tablet_manager.cc:1397] T d18bbcc7ebd14b479ddacb4aad814d47 P 6517ff96fe0a42a2bde064dc7f1ecc97: Time spent bootstrapping tablet: real 0.023s user 0.007s sys 0.009s
I20260809 15:44:48.696430 21856 tablet_bootstrap.cc:654] T ab976a6064754a689b7c88c5fa968b86 P dee2d58fd60f4b39a553c007f76ac760: Neither blocks nor log segments found. Creating new log.
I20260809 15:44:48.697172 21853 raft_consensus.cc:348] T d18bbcc7ebd14b479ddacb4aad814d47 P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } }
I20260809 15:44:48.703068 21853 raft_consensus.cc:374] T d18bbcc7ebd14b479ddacb4aad814d47 P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:44:48.709371 21853 raft_consensus.cc:729] T d18bbcc7ebd14b479ddacb4aad814d47 P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6517ff96fe0a42a2bde064dc7f1ecc97, State: Initialized, Role: FOLLOWER
I20260809 15:44:48.710011 21853 consensus_queue.cc:260] T d18bbcc7ebd14b479ddacb4aad814d47 P 6517ff96fe0a42a2bde064dc7f1ecc97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } }
I20260809 15:44:48.710855 21850 tablet_bootstrap.cc:492] T d18bbcc7ebd14b479ddacb4aad814d47 P faa9b2a805cd43eca626c68bc7e7168f: No bootstrap required, opened a new log
I20260809 15:44:48.712177 21853 ts_tablet_manager.cc:1428] T d18bbcc7ebd14b479ddacb4aad814d47 P 6517ff96fe0a42a2bde064dc7f1ecc97: Time spent starting tablet: real 0.023s user 0.003s sys 0.003s
I20260809 15:44:48.712487 21850 ts_tablet_manager.cc:1397] T d18bbcc7ebd14b479ddacb4aad814d47 P faa9b2a805cd43eca626c68bc7e7168f: Time spent bootstrapping tablet: real 0.058s user 0.013s sys 0.012s
I20260809 15:44:48.712946 21853 tablet_bootstrap.cc:492] T 4fe458f602674472bedadfdd3c7ee1e6 P 6517ff96fe0a42a2bde064dc7f1ecc97: Bootstrap starting.
I20260809 15:44:48.715499 21850 raft_consensus.cc:348] T d18bbcc7ebd14b479ddacb4aad814d47 P faa9b2a805cd43eca626c68bc7e7168f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } }
I20260809 15:44:48.716214 21850 raft_consensus.cc:374] T d18bbcc7ebd14b479ddacb4aad814d47 P faa9b2a805cd43eca626c68bc7e7168f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:44:48.716480 21850 raft_consensus.cc:729] T d18bbcc7ebd14b479ddacb4aad814d47 P faa9b2a805cd43eca626c68bc7e7168f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: faa9b2a805cd43eca626c68bc7e7168f, State: Initialized, Role: FOLLOWER
W20260809 15:44:48.719609 21691 tablet.cc:2369] T ae87ef2626584e6bacd9f1af2c8991d8 P 6517ff96fe0a42a2bde064dc7f1ecc97: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 15:44:48.719913 21850 consensus_queue.cc:260] T d18bbcc7ebd14b479ddacb4aad814d47 P faa9b2a805cd43eca626c68bc7e7168f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } }
I20260809 15:44:48.724444 21850 ts_tablet_manager.cc:1428] T d18bbcc7ebd14b479ddacb4aad814d47 P faa9b2a805cd43eca626c68bc7e7168f: Time spent starting tablet: real 0.012s user 0.007s sys 0.000s
I20260809 15:44:48.725382 21850 tablet_bootstrap.cc:492] T 4fe458f602674472bedadfdd3c7ee1e6 P faa9b2a805cd43eca626c68bc7e7168f: Bootstrap starting.
I20260809 15:44:48.726631 21513 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dd9896939284d9cb2c03e9c21a92dfa" candidate_uuid: "dee2d58fd60f4b39a553c007f76ac760" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "faa9b2a805cd43eca626c68bc7e7168f" is_pre_election: true
W20260809 15:44:48.728601 21709 leader_election.cc:341] T 6dd9896939284d9cb2c03e9c21a92dfa P dee2d58fd60f4b39a553c007f76ac760 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer faa9b2a805cd43eca626c68bc7e7168f (127.20.202.65:37685): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:44:48.732568 21850 tablet_bootstrap.cc:654] T 4fe458f602674472bedadfdd3c7ee1e6 P faa9b2a805cd43eca626c68bc7e7168f: Neither blocks nor log segments found. Creating new log.
I20260809 15:44:48.734268 21853 tablet_bootstrap.cc:654] T 4fe458f602674472bedadfdd3c7ee1e6 P 6517ff96fe0a42a2bde064dc7f1ecc97: Neither blocks nor log segments found. Creating new log.
I20260809 15:44:48.741176 21856 tablet_bootstrap.cc:492] T ab976a6064754a689b7c88c5fa968b86 P dee2d58fd60f4b39a553c007f76ac760: No bootstrap required, opened a new log
I20260809 15:44:48.741618 21856 ts_tablet_manager.cc:1397] T ab976a6064754a689b7c88c5fa968b86 P dee2d58fd60f4b39a553c007f76ac760: Time spent bootstrapping tablet: real 0.068s user 0.004s sys 0.008s
I20260809 15:44:48.744354 21856 raft_consensus.cc:348] T ab976a6064754a689b7c88c5fa968b86 P dee2d58fd60f4b39a553c007f76ac760 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } }
I20260809 15:44:48.745080 21856 raft_consensus.cc:374] T ab976a6064754a689b7c88c5fa968b86 P dee2d58fd60f4b39a553c007f76ac760 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:44:48.745254 21635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dd9896939284d9cb2c03e9c21a92dfa" candidate_uuid: "dee2d58fd60f4b39a553c007f76ac760" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" is_pre_election: true
I20260809 15:44:48.745397 21856 raft_consensus.cc:729] T ab976a6064754a689b7c88c5fa968b86 P dee2d58fd60f4b39a553c007f76ac760 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dee2d58fd60f4b39a553c007f76ac760, State: Initialized, Role: FOLLOWER
I20260809 15:44:48.746198 21856 consensus_queue.cc:260] T ab976a6064754a689b7c88c5fa968b86 P dee2d58fd60f4b39a553c007f76ac760 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } }
W20260809 15:44:48.747013 21710 leader_election.cc:341] T 6dd9896939284d9cb2c03e9c21a92dfa P dee2d58fd60f4b39a553c007f76ac760 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6517ff96fe0a42a2bde064dc7f1ecc97 (127.20.202.66:42701): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:44:48.747613 21710 leader_election.cc:302] T 6dd9896939284d9cb2c03e9c21a92dfa P dee2d58fd60f4b39a553c007f76ac760 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dee2d58fd60f4b39a553c007f76ac760; no voters: 6517ff96fe0a42a2bde064dc7f1ecc97, faa9b2a805cd43eca626c68bc7e7168f
I20260809 15:44:48.749614 21866 raft_consensus.cc:2726] T 6dd9896939284d9cb2c03e9c21a92dfa P dee2d58fd60f4b39a553c007f76ac760 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 15:44:48.751339 21856 ts_tablet_manager.cc:1428] T ab976a6064754a689b7c88c5fa968b86 P dee2d58fd60f4b39a553c007f76ac760: Time spent starting tablet: real 0.009s user 0.004s sys 0.004s
I20260809 15:44:48.752261 21856 tablet_bootstrap.cc:492] T 8d9c4e17e76743439089a2a53c45d01d P dee2d58fd60f4b39a553c007f76ac760: Bootstrap starting.
I20260809 15:44:48.758095 21856 tablet_bootstrap.cc:654] T 8d9c4e17e76743439089a2a53c45d01d P dee2d58fd60f4b39a553c007f76ac760: Neither blocks nor log segments found. Creating new log.
I20260809 15:44:48.760914 21850 tablet_bootstrap.cc:492] T 4fe458f602674472bedadfdd3c7ee1e6 P faa9b2a805cd43eca626c68bc7e7168f: No bootstrap required, opened a new log
I20260809 15:44:48.761334 21850 ts_tablet_manager.cc:1397] T 4fe458f602674472bedadfdd3c7ee1e6 P faa9b2a805cd43eca626c68bc7e7168f: Time spent bootstrapping tablet: real 0.036s user 0.011s sys 0.003s
I20260809 15:44:48.761777 21853 tablet_bootstrap.cc:492] T 4fe458f602674472bedadfdd3c7ee1e6 P 6517ff96fe0a42a2bde064dc7f1ecc97: No bootstrap required, opened a new log
I20260809 15:44:48.762256 21853 ts_tablet_manager.cc:1397] T 4fe458f602674472bedadfdd3c7ee1e6 P 6517ff96fe0a42a2bde064dc7f1ecc97: Time spent bootstrapping tablet: real 0.049s user 0.015s sys 0.017s
I20260809 15:44:48.764284 21850 raft_consensus.cc:348] T 4fe458f602674472bedadfdd3c7ee1e6 P faa9b2a805cd43eca626c68bc7e7168f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } }
I20260809 15:44:48.764953 21850 raft_consensus.cc:374] T 4fe458f602674472bedadfdd3c7ee1e6 P faa9b2a805cd43eca626c68bc7e7168f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:44:48.765317 21850 raft_consensus.cc:729] T 4fe458f602674472bedadfdd3c7ee1e6 P faa9b2a805cd43eca626c68bc7e7168f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: faa9b2a805cd43eca626c68bc7e7168f, State: Initialized, Role: FOLLOWER
I20260809 15:44:48.764848 21853 raft_consensus.cc:348] T 4fe458f602674472bedadfdd3c7ee1e6 P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } }
I20260809 15:44:48.765847 21853 raft_consensus.cc:374] T 4fe458f602674472bedadfdd3c7ee1e6 P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:44:48.766177 21853 raft_consensus.cc:729] T 4fe458f602674472bedadfdd3c7ee1e6 P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6517ff96fe0a42a2bde064dc7f1ecc97, State: Initialized, Role: FOLLOWER
I20260809 15:44:48.766417 21850 consensus_queue.cc:260] T 4fe458f602674472bedadfdd3c7ee1e6 P faa9b2a805cd43eca626c68bc7e7168f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } }
I20260809 15:44:48.766886 21853 consensus_queue.cc:260] T 4fe458f602674472bedadfdd3c7ee1e6 P 6517ff96fe0a42a2bde064dc7f1ecc97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } }
I20260809 15:44:48.768718 21850 ts_tablet_manager.cc:1428] T 4fe458f602674472bedadfdd3c7ee1e6 P faa9b2a805cd43eca626c68bc7e7168f: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260809 15:44:48.769333 21853 ts_tablet_manager.cc:1428] T 4fe458f602674472bedadfdd3c7ee1e6 P 6517ff96fe0a42a2bde064dc7f1ecc97: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260809 15:44:48.769515 21850 tablet_bootstrap.cc:492] T ae87ef2626584e6bacd9f1af2c8991d8 P faa9b2a805cd43eca626c68bc7e7168f: Bootstrap starting.
I20260809 15:44:48.770328 21853 tablet_bootstrap.cc:492] T ab976a6064754a689b7c88c5fa968b86 P 6517ff96fe0a42a2bde064dc7f1ecc97: Bootstrap starting.
I20260809 15:44:48.776098 21853 tablet_bootstrap.cc:654] T ab976a6064754a689b7c88c5fa968b86 P 6517ff96fe0a42a2bde064dc7f1ecc97: Neither blocks nor log segments found. Creating new log.
I20260809 15:44:48.777448 21850 tablet_bootstrap.cc:654] T ae87ef2626584e6bacd9f1af2c8991d8 P faa9b2a805cd43eca626c68bc7e7168f: Neither blocks nor log segments found. Creating new log.
I20260809 15:44:48.783233 21856 tablet_bootstrap.cc:492] T 8d9c4e17e76743439089a2a53c45d01d P dee2d58fd60f4b39a553c007f76ac760: No bootstrap required, opened a new log
I20260809 15:44:48.783885 21856 ts_tablet_manager.cc:1397] T 8d9c4e17e76743439089a2a53c45d01d P dee2d58fd60f4b39a553c007f76ac760: Time spent bootstrapping tablet: real 0.032s user 0.006s sys 0.018s
I20260809 15:44:48.786392 21856 raft_consensus.cc:348] T 8d9c4e17e76743439089a2a53c45d01d P dee2d58fd60f4b39a553c007f76ac760 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } }
I20260809 15:44:48.787072 21856 raft_consensus.cc:374] T 8d9c4e17e76743439089a2a53c45d01d P dee2d58fd60f4b39a553c007f76ac760 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:44:48.787256 21853 tablet_bootstrap.cc:492] T ab976a6064754a689b7c88c5fa968b86 P 6517ff96fe0a42a2bde064dc7f1ecc97: No bootstrap required, opened a new log
I20260809 15:44:48.787802 21853 ts_tablet_manager.cc:1397] T ab976a6064754a689b7c88c5fa968b86 P 6517ff96fe0a42a2bde064dc7f1ecc97: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.014s
I20260809 15:44:48.790257 21856 raft_consensus.cc:729] T 8d9c4e17e76743439089a2a53c45d01d P dee2d58fd60f4b39a553c007f76ac760 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dee2d58fd60f4b39a553c007f76ac760, State: Initialized, Role: FOLLOWER
I20260809 15:44:48.790305 21853 raft_consensus.cc:348] T ab976a6064754a689b7c88c5fa968b86 P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } }
I20260809 15:44:48.791026 21853 raft_consensus.cc:374] T ab976a6064754a689b7c88c5fa968b86 P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:44:48.791409 21853 raft_consensus.cc:729] T ab976a6064754a689b7c88c5fa968b86 P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6517ff96fe0a42a2bde064dc7f1ecc97, State: Initialized, Role: FOLLOWER
I20260809 15:44:48.790994 21856 consensus_queue.cc:260] T 8d9c4e17e76743439089a2a53c45d01d P dee2d58fd60f4b39a553c007f76ac760 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } }
I20260809 15:44:48.792052 21853 consensus_queue.cc:260] T ab976a6064754a689b7c88c5fa968b86 P 6517ff96fe0a42a2bde064dc7f1ecc97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } }
I20260809 15:44:48.792902 21850 tablet_bootstrap.cc:492] T ae87ef2626584e6bacd9f1af2c8991d8 P faa9b2a805cd43eca626c68bc7e7168f: No bootstrap required, opened a new log
I20260809 15:44:48.795065 21850 ts_tablet_manager.cc:1397] T ae87ef2626584e6bacd9f1af2c8991d8 P faa9b2a805cd43eca626c68bc7e7168f: Time spent bootstrapping tablet: real 0.026s user 0.011s sys 0.002s
I20260809 15:44:48.793424 21856 ts_tablet_manager.cc:1428] T 8d9c4e17e76743439089a2a53c45d01d P dee2d58fd60f4b39a553c007f76ac760: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20260809 15:44:48.794206 21853 ts_tablet_manager.cc:1428] T ab976a6064754a689b7c88c5fa968b86 P 6517ff96fe0a42a2bde064dc7f1ecc97: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260809 15:44:48.797607 21850 raft_consensus.cc:348] T ae87ef2626584e6bacd9f1af2c8991d8 P faa9b2a805cd43eca626c68bc7e7168f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } }
I20260809 15:44:48.798430 21853 tablet_bootstrap.cc:492] T 724982e80d3b455a91f8a564d374a65e P 6517ff96fe0a42a2bde064dc7f1ecc97: Bootstrap starting.
I20260809 15:44:48.799547 21856 tablet_bootstrap.cc:492] T 2f46e8d793294ce384acbf2f5584ecd2 P dee2d58fd60f4b39a553c007f76ac760: Bootstrap starting.
I20260809 15:44:48.798380 21850 raft_consensus.cc:374] T ae87ef2626584e6bacd9f1af2c8991d8 P faa9b2a805cd43eca626c68bc7e7168f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:44:48.803162 21850 raft_consensus.cc:729] T ae87ef2626584e6bacd9f1af2c8991d8 P faa9b2a805cd43eca626c68bc7e7168f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: faa9b2a805cd43eca626c68bc7e7168f, State: Initialized, Role: FOLLOWER
I20260809 15:44:48.803889 21850 consensus_queue.cc:260] T ae87ef2626584e6bacd9f1af2c8991d8 P faa9b2a805cd43eca626c68bc7e7168f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } }
I20260809 15:44:48.806898 21853 tablet_bootstrap.cc:654] T 724982e80d3b455a91f8a564d374a65e P 6517ff96fe0a42a2bde064dc7f1ecc97: Neither blocks nor log segments found. Creating new log.
I20260809 15:44:48.809737 21850 ts_tablet_manager.cc:1428] T ae87ef2626584e6bacd9f1af2c8991d8 P faa9b2a805cd43eca626c68bc7e7168f: Time spent starting tablet: real 0.014s user 0.005s sys 0.000s
I20260809 15:44:48.810602 21850 tablet_bootstrap.cc:492] T 724982e80d3b455a91f8a564d374a65e P faa9b2a805cd43eca626c68bc7e7168f: Bootstrap starting.
I20260809 15:44:48.815824 21858 raft_consensus.cc:482] T 4fe458f602674472bedadfdd3c7ee1e6 P faa9b2a805cd43eca626c68bc7e7168f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:44:48.816272 21858 raft_consensus.cc:504] T 4fe458f602674472bedadfdd3c7ee1e6 P faa9b2a805cd43eca626c68bc7e7168f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } }
I20260809 15:44:48.817669 21853 tablet_bootstrap.cc:492] T 724982e80d3b455a91f8a564d374a65e P 6517ff96fe0a42a2bde064dc7f1ecc97: No bootstrap required, opened a new log
I20260809 15:44:48.818068 21853 ts_tablet_manager.cc:1397] T 724982e80d3b455a91f8a564d374a65e P 6517ff96fe0a42a2bde064dc7f1ecc97: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.011s
I20260809 15:44:48.818565 21858 leader_election.cc:288] T 4fe458f602674472bedadfdd3c7ee1e6 P faa9b2a805cd43eca626c68bc7e7168f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6517ff96fe0a42a2bde064dc7f1ecc97 (127.20.202.66:42701), dee2d58fd60f4b39a553c007f76ac760 (127.20.202.67:36447)
I20260809 15:44:48.821493 21856 tablet_bootstrap.cc:654] T 2f46e8d793294ce384acbf2f5584ecd2 P dee2d58fd60f4b39a553c007f76ac760: Neither blocks nor log segments found. Creating new log.
I20260809 15:44:48.824280 21850 tablet_bootstrap.cc:654] T 724982e80d3b455a91f8a564d374a65e P faa9b2a805cd43eca626c68bc7e7168f: Neither blocks nor log segments found. Creating new log.
I20260809 15:44:48.826746 21853 raft_consensus.cc:348] T 724982e80d3b455a91f8a564d374a65e P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } }
I20260809 15:44:48.827513 21853 raft_consensus.cc:374] T 724982e80d3b455a91f8a564d374a65e P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:44:48.827798 21853 raft_consensus.cc:729] T 724982e80d3b455a91f8a564d374a65e P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6517ff96fe0a42a2bde064dc7f1ecc97, State: Initialized, Role: FOLLOWER
I20260809 15:44:48.829936 21853 consensus_queue.cc:260] T 724982e80d3b455a91f8a564d374a65e P 6517ff96fe0a42a2bde064dc7f1ecc97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } }
I20260809 15:44:48.832050 21858 raft_consensus.cc:482] T d18bbcc7ebd14b479ddacb4aad814d47 P faa9b2a805cd43eca626c68bc7e7168f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:44:48.832482 21858 raft_consensus.cc:504] T d18bbcc7ebd14b479ddacb4aad814d47 P faa9b2a805cd43eca626c68bc7e7168f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } }
I20260809 15:44:48.834259 21858 leader_election.cc:288] T d18bbcc7ebd14b479ddacb4aad814d47 P faa9b2a805cd43eca626c68bc7e7168f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6517ff96fe0a42a2bde064dc7f1ecc97 (127.20.202.66:42701), dee2d58fd60f4b39a553c007f76ac760 (127.20.202.67:36447)
I20260809 15:44:48.835353 21853 ts_tablet_manager.cc:1428] T 724982e80d3b455a91f8a564d374a65e P 6517ff96fe0a42a2bde064dc7f1ecc97: Time spent starting tablet: real 0.017s user 0.006s sys 0.003s
I20260809 15:44:48.842401 21850 tablet_bootstrap.cc:492] T 724982e80d3b455a91f8a564d374a65e P faa9b2a805cd43eca626c68bc7e7168f: No bootstrap required, opened a new log
I20260809 15:44:48.842782 21853 tablet_bootstrap.cc:492] T 2f46e8d793294ce384acbf2f5584ecd2 P 6517ff96fe0a42a2bde064dc7f1ecc97: Bootstrap starting.
I20260809 15:44:48.842871 21850 ts_tablet_manager.cc:1397] T 724982e80d3b455a91f8a564d374a65e P faa9b2a805cd43eca626c68bc7e7168f: Time spent bootstrapping tablet: real 0.032s user 0.003s sys 0.012s
I20260809 15:44:48.845549 21856 tablet_bootstrap.cc:492] T 2f46e8d793294ce384acbf2f5584ecd2 P dee2d58fd60f4b39a553c007f76ac760: No bootstrap required, opened a new log
I20260809 15:44:48.846076 21856 ts_tablet_manager.cc:1397] T 2f46e8d793294ce384acbf2f5584ecd2 P dee2d58fd60f4b39a553c007f76ac760: Time spent bootstrapping tablet: real 0.047s user 0.004s sys 0.009s
I20260809 15:44:48.848676 21856 raft_consensus.cc:348] T 2f46e8d793294ce384acbf2f5584ecd2 P dee2d58fd60f4b39a553c007f76ac760 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } }
I20260809 15:44:48.849359 21856 raft_consensus.cc:374] T 2f46e8d793294ce384acbf2f5584ecd2 P dee2d58fd60f4b39a553c007f76ac760 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:44:48.849629 21856 raft_consensus.cc:729] T 2f46e8d793294ce384acbf2f5584ecd2 P dee2d58fd60f4b39a553c007f76ac760 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dee2d58fd60f4b39a553c007f76ac760, State: Initialized, Role: FOLLOWER
I20260809 15:44:48.850251 21856 consensus_queue.cc:260] T 2f46e8d793294ce384acbf2f5584ecd2 P dee2d58fd60f4b39a553c007f76ac760 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } }
I20260809 15:44:48.853434 21635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe458f602674472bedadfdd3c7ee1e6" candidate_uuid: "faa9b2a805cd43eca626c68bc7e7168f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" is_pre_election: true
I20260809 15:44:48.854068 21635 raft_consensus.cc:2454] T 4fe458f602674472bedadfdd3c7ee1e6 P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate faa9b2a805cd43eca626c68bc7e7168f in term 0.
I20260809 15:44:48.855114 21634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d18bbcc7ebd14b479ddacb4aad814d47" candidate_uuid: "faa9b2a805cd43eca626c68bc7e7168f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" is_pre_election: true
I20260809 15:44:48.855742 21634 raft_consensus.cc:2454] T d18bbcc7ebd14b479ddacb4aad814d47 P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate faa9b2a805cd43eca626c68bc7e7168f in term 0.
I20260809 15:44:48.857050 21853 tablet_bootstrap.cc:654] T 2f46e8d793294ce384acbf2f5584ecd2 P 6517ff96fe0a42a2bde064dc7f1ecc97: Neither blocks nor log segments found. Creating new log.
I20260809 15:44:48.857208 21850 raft_consensus.cc:348] T 724982e80d3b455a91f8a564d374a65e P faa9b2a805cd43eca626c68bc7e7168f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } }
I20260809 15:44:48.858070 21850 raft_consensus.cc:374] T 724982e80d3b455a91f8a564d374a65e P faa9b2a805cd43eca626c68bc7e7168f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:44:48.858414 21850 raft_consensus.cc:729] T 724982e80d3b455a91f8a564d374a65e P faa9b2a805cd43eca626c68bc7e7168f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: faa9b2a805cd43eca626c68bc7e7168f, State: Initialized, Role: FOLLOWER
I20260809 15:44:48.859122 21850 consensus_queue.cc:260] T 724982e80d3b455a91f8a564d374a65e P faa9b2a805cd43eca626c68bc7e7168f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } }
I20260809 15:44:48.861799 21769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d18bbcc7ebd14b479ddacb4aad814d47" candidate_uuid: "faa9b2a805cd43eca626c68bc7e7168f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dee2d58fd60f4b39a553c007f76ac760" is_pre_election: true
I20260809 15:44:48.861367 21460 leader_election.cc:302] T 4fe458f602674472bedadfdd3c7ee1e6 P faa9b2a805cd43eca626c68bc7e7168f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6517ff96fe0a42a2bde064dc7f1ecc97, faa9b2a805cd43eca626c68bc7e7168f; no voters:
I20260809 15:44:48.862522 21850 ts_tablet_manager.cc:1428] T 724982e80d3b455a91f8a564d374a65e P faa9b2a805cd43eca626c68bc7e7168f: Time spent starting tablet: real 0.019s user 0.006s sys 0.000s
I20260809 15:44:48.863618 21856 ts_tablet_manager.cc:1428] T 2f46e8d793294ce384acbf2f5584ecd2 P dee2d58fd60f4b39a553c007f76ac760: Time spent starting tablet: real 0.017s user 0.006s sys 0.001s
I20260809 15:44:48.864478 21460 leader_election.cc:302] T d18bbcc7ebd14b479ddacb4aad814d47 P faa9b2a805cd43eca626c68bc7e7168f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6517ff96fe0a42a2bde064dc7f1ecc97, faa9b2a805cd43eca626c68bc7e7168f; no voters:
I20260809 15:44:48.863402 21770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe458f602674472bedadfdd3c7ee1e6" candidate_uuid: "faa9b2a805cd43eca626c68bc7e7168f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dee2d58fd60f4b39a553c007f76ac760" is_pre_election: true
I20260809 15:44:48.865748 21858 raft_consensus.cc:2781] T 4fe458f602674472bedadfdd3c7ee1e6 P faa9b2a805cd43eca626c68bc7e7168f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 15:44:48.866250 21858 raft_consensus.cc:482] T 4fe458f602674472bedadfdd3c7ee1e6 P faa9b2a805cd43eca626c68bc7e7168f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:44:48.866886 21858 raft_consensus.cc:3037] T 4fe458f602674472bedadfdd3c7ee1e6 P faa9b2a805cd43eca626c68bc7e7168f [term 0 FOLLOWER]: Advancing to term 1
W20260809 15:44:48.872238 21461 leader_election.cc:341] T d18bbcc7ebd14b479ddacb4aad814d47 P faa9b2a805cd43eca626c68bc7e7168f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dee2d58fd60f4b39a553c007f76ac760 (127.20.202.67:36447): Illegal state: must be running to vote when last-logged opid is not known
W20260809 15:44:48.874506 21461 leader_election.cc:341] T 4fe458f602674472bedadfdd3c7ee1e6 P faa9b2a805cd43eca626c68bc7e7168f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dee2d58fd60f4b39a553c007f76ac760 (127.20.202.67:36447): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:44:48.874691 21856 tablet_bootstrap.cc:492] T ae87ef2626584e6bacd9f1af2c8991d8 P dee2d58fd60f4b39a553c007f76ac760: Bootstrap starting.
I20260809 15:44:48.881902 21856 tablet_bootstrap.cc:654] T ae87ef2626584e6bacd9f1af2c8991d8 P dee2d58fd60f4b39a553c007f76ac760: Neither blocks nor log segments found. Creating new log.
I20260809 15:44:48.882196 21850 tablet_bootstrap.cc:492] T 6dd9896939284d9cb2c03e9c21a92dfa P faa9b2a805cd43eca626c68bc7e7168f: Bootstrap starting.
I20260809 15:44:48.882210 21858 raft_consensus.cc:504] T 4fe458f602674472bedadfdd3c7ee1e6 P faa9b2a805cd43eca626c68bc7e7168f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } }
I20260809 15:44:48.888212 21895 raft_consensus.cc:2781] T d18bbcc7ebd14b479ddacb4aad814d47 P faa9b2a805cd43eca626c68bc7e7168f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 15:44:48.888645 21895 raft_consensus.cc:482] T d18bbcc7ebd14b479ddacb4aad814d47 P faa9b2a805cd43eca626c68bc7e7168f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:44:48.889024 21895 raft_consensus.cc:3037] T d18bbcc7ebd14b479ddacb4aad814d47 P faa9b2a805cd43eca626c68bc7e7168f [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:44:48.889691 21853 tablet_bootstrap.cc:492] T 2f46e8d793294ce384acbf2f5584ecd2 P 6517ff96fe0a42a2bde064dc7f1ecc97: No bootstrap required, opened a new log
I20260809 15:44:48.890111 21853 ts_tablet_manager.cc:1397] T 2f46e8d793294ce384acbf2f5584ecd2 P 6517ff96fe0a42a2bde064dc7f1ecc97: Time spent bootstrapping tablet: real 0.047s user 0.015s sys 0.009s
I20260809 15:44:48.890707 21858 leader_election.cc:288] T 4fe458f602674472bedadfdd3c7ee1e6 P faa9b2a805cd43eca626c68bc7e7168f [CANDIDATE]: Term 1 election: Requested vote from peers 6517ff96fe0a42a2bde064dc7f1ecc97 (127.20.202.66:42701), dee2d58fd60f4b39a553c007f76ac760 (127.20.202.67:36447)
I20260809 15:44:48.892373 21770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe458f602674472bedadfdd3c7ee1e6" candidate_uuid: "faa9b2a805cd43eca626c68bc7e7168f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dee2d58fd60f4b39a553c007f76ac760"
I20260809 15:44:48.892692 21853 raft_consensus.cc:348] T 2f46e8d793294ce384acbf2f5584ecd2 P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } }
I20260809 15:44:48.893401 21853 raft_consensus.cc:374] T 2f46e8d793294ce384acbf2f5584ecd2 P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:44:48.893684 21853 raft_consensus.cc:729] T 2f46e8d793294ce384acbf2f5584ecd2 P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6517ff96fe0a42a2bde064dc7f1ecc97, State: Initialized, Role: FOLLOWER
I20260809 15:44:48.893850 21850 tablet_bootstrap.cc:654] T 6dd9896939284d9cb2c03e9c21a92dfa P faa9b2a805cd43eca626c68bc7e7168f: Neither blocks nor log segments found. Creating new log.
W20260809 15:44:48.893808 21461 leader_election.cc:341] T 4fe458f602674472bedadfdd3c7ee1e6 P faa9b2a805cd43eca626c68bc7e7168f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer dee2d58fd60f4b39a553c007f76ac760 (127.20.202.67:36447): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:44:48.894382 21853 consensus_queue.cc:260] T 2f46e8d793294ce384acbf2f5584ecd2 P 6517ff96fe0a42a2bde064dc7f1ecc97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } }
I20260809 15:44:48.896118 21634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe458f602674472bedadfdd3c7ee1e6" candidate_uuid: "faa9b2a805cd43eca626c68bc7e7168f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97"
I20260809 15:44:48.896641 21634 raft_consensus.cc:3037] T 4fe458f602674472bedadfdd3c7ee1e6 P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:44:48.902282 21853 ts_tablet_manager.cc:1428] T 2f46e8d793294ce384acbf2f5584ecd2 P 6517ff96fe0a42a2bde064dc7f1ecc97: Time spent starting tablet: real 0.012s user 0.006s sys 0.000s
I20260809 15:44:48.903751 21634 raft_consensus.cc:2454] T 4fe458f602674472bedadfdd3c7ee1e6 P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate faa9b2a805cd43eca626c68bc7e7168f in term 1.
I20260809 15:44:48.905887 21460 leader_election.cc:302] T 4fe458f602674472bedadfdd3c7ee1e6 P faa9b2a805cd43eca626c68bc7e7168f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6517ff96fe0a42a2bde064dc7f1ecc97, faa9b2a805cd43eca626c68bc7e7168f; no voters: dee2d58fd60f4b39a553c007f76ac760
I20260809 15:44:48.908004 21858 raft_consensus.cc:2781] T 4fe458f602674472bedadfdd3c7ee1e6 P faa9b2a805cd43eca626c68bc7e7168f [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:44:48.912408 21853 tablet_bootstrap.cc:492] T d203a1017c6b44d0a62bbc654ee21d8c P 6517ff96fe0a42a2bde064dc7f1ecc97: Bootstrap starting.
I20260809 15:44:48.915529 21858 raft_consensus.cc:686] T 4fe458f602674472bedadfdd3c7ee1e6 P faa9b2a805cd43eca626c68bc7e7168f [term 1 LEADER]: Becoming Leader. State: Replica: faa9b2a805cd43eca626c68bc7e7168f, State: Running, Role: LEADER
I20260809 15:44:48.917841 21858 consensus_queue.cc:237] T 4fe458f602674472bedadfdd3c7ee1e6 P faa9b2a805cd43eca626c68bc7e7168f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } }
I20260809 15:44:48.920270 21895 raft_consensus.cc:504] T d18bbcc7ebd14b479ddacb4aad814d47 P faa9b2a805cd43eca626c68bc7e7168f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } }
I20260809 15:44:48.925745 21634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d18bbcc7ebd14b479ddacb4aad814d47" candidate_uuid: "faa9b2a805cd43eca626c68bc7e7168f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97"
I20260809 15:44:48.926362 21770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d18bbcc7ebd14b479ddacb4aad814d47" candidate_uuid: "faa9b2a805cd43eca626c68bc7e7168f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dee2d58fd60f4b39a553c007f76ac760"
I20260809 15:44:48.926682 21634 raft_consensus.cc:3037] T d18bbcc7ebd14b479ddacb4aad814d47 P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:44:48.926849 21850 tablet_bootstrap.cc:492] T 6dd9896939284d9cb2c03e9c21a92dfa P faa9b2a805cd43eca626c68bc7e7168f: No bootstrap required, opened a new log
I20260809 15:44:48.924925 21895 leader_election.cc:288] T d18bbcc7ebd14b479ddacb4aad814d47 P faa9b2a805cd43eca626c68bc7e7168f [CANDIDATE]: Term 1 election: Requested vote from peers 6517ff96fe0a42a2bde064dc7f1ecc97 (127.20.202.66:42701), dee2d58fd60f4b39a553c007f76ac760 (127.20.202.67:36447)
W20260809 15:44:48.928627 21461 leader_election.cc:341] T d18bbcc7ebd14b479ddacb4aad814d47 P faa9b2a805cd43eca626c68bc7e7168f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer dee2d58fd60f4b39a553c007f76ac760 (127.20.202.67:36447): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:44:48.928813 21853 tablet_bootstrap.cc:654] T d203a1017c6b44d0a62bbc654ee21d8c P 6517ff96fe0a42a2bde064dc7f1ecc97: Neither blocks nor log segments found. Creating new log.
I20260809 15:44:48.927851 21850 ts_tablet_manager.cc:1397] T 6dd9896939284d9cb2c03e9c21a92dfa P faa9b2a805cd43eca626c68bc7e7168f: Time spent bootstrapping tablet: real 0.050s user 0.013s sys 0.000s
I20260809 15:44:48.929744 21856 tablet_bootstrap.cc:492] T ae87ef2626584e6bacd9f1af2c8991d8 P dee2d58fd60f4b39a553c007f76ac760: No bootstrap required, opened a new log
I20260809 15:44:48.931602 21856 ts_tablet_manager.cc:1397] T ae87ef2626584e6bacd9f1af2c8991d8 P dee2d58fd60f4b39a553c007f76ac760: Time spent bootstrapping tablet: real 0.057s user 0.037s sys 0.000s
I20260809 15:44:48.935289 21634 raft_consensus.cc:2454] T d18bbcc7ebd14b479ddacb4aad814d47 P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate faa9b2a805cd43eca626c68bc7e7168f in term 1.
I20260809 15:44:48.933254 21850 raft_consensus.cc:348] T 6dd9896939284d9cb2c03e9c21a92dfa P faa9b2a805cd43eca626c68bc7e7168f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } }
I20260809 15:44:48.937201 21850 raft_consensus.cc:374] T 6dd9896939284d9cb2c03e9c21a92dfa P faa9b2a805cd43eca626c68bc7e7168f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:44:48.937531 21850 raft_consensus.cc:729] T 6dd9896939284d9cb2c03e9c21a92dfa P faa9b2a805cd43eca626c68bc7e7168f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: faa9b2a805cd43eca626c68bc7e7168f, State: Initialized, Role: FOLLOWER
I20260809 15:44:48.937606 21460 leader_election.cc:302] T d18bbcc7ebd14b479ddacb4aad814d47 P faa9b2a805cd43eca626c68bc7e7168f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6517ff96fe0a42a2bde064dc7f1ecc97, faa9b2a805cd43eca626c68bc7e7168f; no voters: dee2d58fd60f4b39a553c007f76ac760
I20260809 15:44:48.938344 21850 consensus_queue.cc:260] T 6dd9896939284d9cb2c03e9c21a92dfa P faa9b2a805cd43eca626c68bc7e7168f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } }
I20260809 15:44:48.939622 21858 raft_consensus.cc:2781] T d18bbcc7ebd14b479ddacb4aad814d47 P faa9b2a805cd43eca626c68bc7e7168f [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:44:48.940712 21858 raft_consensus.cc:686] T d18bbcc7ebd14b479ddacb4aad814d47 P faa9b2a805cd43eca626c68bc7e7168f [term 1 LEADER]: Becoming Leader. State: Replica: faa9b2a805cd43eca626c68bc7e7168f, State: Running, Role: LEADER
I20260809 15:44:48.941658 21858 consensus_queue.cc:237] T d18bbcc7ebd14b479ddacb4aad814d47 P faa9b2a805cd43eca626c68bc7e7168f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } }
I20260809 15:44:48.942358 21850 ts_tablet_manager.cc:1428] T 6dd9896939284d9cb2c03e9c21a92dfa P faa9b2a805cd43eca626c68bc7e7168f: Time spent starting tablet: real 0.013s user 0.004s sys 0.004s
I20260809 15:44:48.945047 21850 tablet_bootstrap.cc:492] T 8d9c4e17e76743439089a2a53c45d01d P faa9b2a805cd43eca626c68bc7e7168f: Bootstrap starting.
I20260809 15:44:48.945317 21856 raft_consensus.cc:348] T ae87ef2626584e6bacd9f1af2c8991d8 P dee2d58fd60f4b39a553c007f76ac760 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } }
I20260809 15:44:48.946393 21856 raft_consensus.cc:374] T ae87ef2626584e6bacd9f1af2c8991d8 P dee2d58fd60f4b39a553c007f76ac760 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:44:48.947242 21856 raft_consensus.cc:729] T ae87ef2626584e6bacd9f1af2c8991d8 P dee2d58fd60f4b39a553c007f76ac760 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dee2d58fd60f4b39a553c007f76ac760, State: Initialized, Role: FOLLOWER
I20260809 15:44:48.948380 21853 tablet_bootstrap.cc:492] T d203a1017c6b44d0a62bbc654ee21d8c P 6517ff96fe0a42a2bde064dc7f1ecc97: No bootstrap required, opened a new log
I20260809 15:44:48.948941 21853 ts_tablet_manager.cc:1397] T d203a1017c6b44d0a62bbc654ee21d8c P 6517ff96fe0a42a2bde064dc7f1ecc97: Time spent bootstrapping tablet: real 0.039s user 0.005s sys 0.016s
I20260809 15:44:48.948412 21856 consensus_queue.cc:260] T ae87ef2626584e6bacd9f1af2c8991d8 P dee2d58fd60f4b39a553c007f76ac760 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } }
I20260809 15:44:48.952435 21856 ts_tablet_manager.cc:1428] T ae87ef2626584e6bacd9f1af2c8991d8 P dee2d58fd60f4b39a553c007f76ac760: Time spent starting tablet: real 0.020s user 0.003s sys 0.006s
I20260809 15:44:48.953593 21856 tablet_bootstrap.cc:492] T d203a1017c6b44d0a62bbc654ee21d8c P dee2d58fd60f4b39a553c007f76ac760: Bootstrap starting.
I20260809 15:44:48.955165 21853 raft_consensus.cc:348] T d203a1017c6b44d0a62bbc654ee21d8c P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } }
I20260809 15:44:48.956346 21853 raft_consensus.cc:374] T d203a1017c6b44d0a62bbc654ee21d8c P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:44:48.956930 21853 raft_consensus.cc:729] T d203a1017c6b44d0a62bbc654ee21d8c P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6517ff96fe0a42a2bde064dc7f1ecc97, State: Initialized, Role: FOLLOWER
I20260809 15:44:48.958086 21853 consensus_queue.cc:260] T d203a1017c6b44d0a62bbc654ee21d8c P 6517ff96fe0a42a2bde064dc7f1ecc97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } }
I20260809 15:44:48.960511 21853 ts_tablet_manager.cc:1428] T d203a1017c6b44d0a62bbc654ee21d8c P 6517ff96fe0a42a2bde064dc7f1ecc97: Time spent starting tablet: real 0.011s user 0.005s sys 0.001s
I20260809 15:44:48.961346 21853 tablet_bootstrap.cc:492] T 6dd9896939284d9cb2c03e9c21a92dfa P 6517ff96fe0a42a2bde064dc7f1ecc97: Bootstrap starting.
I20260809 15:44:48.962041 21856 tablet_bootstrap.cc:654] T d203a1017c6b44d0a62bbc654ee21d8c P dee2d58fd60f4b39a553c007f76ac760: Neither blocks nor log segments found. Creating new log.
I20260809 15:44:48.963176 21865 raft_consensus.cc:482] T 724982e80d3b455a91f8a564d374a65e P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:44:48.963376 21850 tablet_bootstrap.cc:654] T 8d9c4e17e76743439089a2a53c45d01d P faa9b2a805cd43eca626c68bc7e7168f: Neither blocks nor log segments found. Creating new log.
I20260809 15:44:48.963647 21865 raft_consensus.cc:504] T 724982e80d3b455a91f8a564d374a65e P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } }
I20260809 15:44:48.966337 21865 leader_election.cc:288] T 724982e80d3b455a91f8a564d374a65e P 6517ff96fe0a42a2bde064dc7f1ecc97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers faa9b2a805cd43eca626c68bc7e7168f (127.20.202.65:37685), dee2d58fd60f4b39a553c007f76ac760 (127.20.202.67:36447)
I20260809 15:44:48.989351 21333 catalog_manager.cc:5495] T 4fe458f602674472bedadfdd3c7ee1e6 P faa9b2a805cd43eca626c68bc7e7168f reported cstate change: term changed from 0 to 1, leader changed from <none> to faa9b2a805cd43eca626c68bc7e7168f (127.20.202.65). New cstate: current_term: 1 leader_uuid: "faa9b2a805cd43eca626c68bc7e7168f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } health_report { overall_health: UNKNOWN } } }
I20260809 15:44:48.992960 21853 tablet_bootstrap.cc:654] T 6dd9896939284d9cb2c03e9c21a92dfa P 6517ff96fe0a42a2bde064dc7f1ecc97: Neither blocks nor log segments found. Creating new log.
I20260809 15:44:49.000116 21858 raft_consensus.cc:482] T ae87ef2626584e6bacd9f1af2c8991d8 P faa9b2a805cd43eca626c68bc7e7168f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:44:49.000885 21858 raft_consensus.cc:504] T ae87ef2626584e6bacd9f1af2c8991d8 P faa9b2a805cd43eca626c68bc7e7168f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } }
I20260809 15:44:49.001996 21895 raft_consensus.cc:482] T 6dd9896939284d9cb2c03e9c21a92dfa P faa9b2a805cd43eca626c68bc7e7168f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:44:49.002485 21895 raft_consensus.cc:504] T 6dd9896939284d9cb2c03e9c21a92dfa P faa9b2a805cd43eca626c68bc7e7168f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } }
I20260809 15:44:49.003154 21858 leader_election.cc:288] T ae87ef2626584e6bacd9f1af2c8991d8 P faa9b2a805cd43eca626c68bc7e7168f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6517ff96fe0a42a2bde064dc7f1ecc97 (127.20.202.66:42701), dee2d58fd60f4b39a553c007f76ac760 (127.20.202.67:36447)
I20260809 15:44:49.004069 21858 raft_consensus.cc:482] T ab976a6064754a689b7c88c5fa968b86 P faa9b2a805cd43eca626c68bc7e7168f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:44:49.004456 21895 leader_election.cc:288] T 6dd9896939284d9cb2c03e9c21a92dfa P faa9b2a805cd43eca626c68bc7e7168f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6517ff96fe0a42a2bde064dc7f1ecc97 (127.20.202.66:42701), dee2d58fd60f4b39a553c007f76ac760 (127.20.202.67:36447)
I20260809 15:44:49.004814 21865 raft_consensus.cc:482] T 2f46e8d793294ce384acbf2f5584ecd2 P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:44:49.004470 21858 raft_consensus.cc:504] T ab976a6064754a689b7c88c5fa968b86 P faa9b2a805cd43eca626c68bc7e7168f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } }
I20260809 15:44:49.005273 21865 raft_consensus.cc:504] T 2f46e8d793294ce384acbf2f5584ecd2 P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } }
I20260809 15:44:49.006924 21865 leader_election.cc:288] T 2f46e8d793294ce384acbf2f5584ecd2 P 6517ff96fe0a42a2bde064dc7f1ecc97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers faa9b2a805cd43eca626c68bc7e7168f (127.20.202.65:37685), dee2d58fd60f4b39a553c007f76ac760 (127.20.202.67:36447)
I20260809 15:44:49.006945 21858 leader_election.cc:288] T ab976a6064754a689b7c88c5fa968b86 P faa9b2a805cd43eca626c68bc7e7168f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6517ff96fe0a42a2bde064dc7f1ecc97 (127.20.202.66:42701), dee2d58fd60f4b39a553c007f76ac760 (127.20.202.67:36447)
I20260809 15:44:49.015144 21634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae87ef2626584e6bacd9f1af2c8991d8" candidate_uuid: "faa9b2a805cd43eca626c68bc7e7168f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" is_pre_election: true
I20260809 15:44:49.007617 21770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae87ef2626584e6bacd9f1af2c8991d8" candidate_uuid: "faa9b2a805cd43eca626c68bc7e7168f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dee2d58fd60f4b39a553c007f76ac760" is_pre_election: true
I20260809 15:44:49.013170 21769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dd9896939284d9cb2c03e9c21a92dfa" candidate_uuid: "faa9b2a805cd43eca626c68bc7e7168f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dee2d58fd60f4b39a553c007f76ac760" is_pre_election: true
I20260809 15:44:49.022563 21770 raft_consensus.cc:2454] T ae87ef2626584e6bacd9f1af2c8991d8 P dee2d58fd60f4b39a553c007f76ac760 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate faa9b2a805cd43eca626c68bc7e7168f in term 0.
I20260809 15:44:49.033834 21866 raft_consensus.cc:482] T 2f46e8d793294ce384acbf2f5584ecd2 P dee2d58fd60f4b39a553c007f76ac760 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:44:49.022025 21634 raft_consensus.cc:2454] T ae87ef2626584e6bacd9f1af2c8991d8 P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate faa9b2a805cd43eca626c68bc7e7168f in term 0.
I20260809 15:44:49.033897 21633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab976a6064754a689b7c88c5fa968b86" candidate_uuid: "faa9b2a805cd43eca626c68bc7e7168f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" is_pre_election: true
I20260809 15:44:49.036243 21856 tablet_bootstrap.cc:492] T d203a1017c6b44d0a62bbc654ee21d8c P dee2d58fd60f4b39a553c007f76ac760: No bootstrap required, opened a new log
I20260809 15:44:49.035607 21866 raft_consensus.cc:504] T 2f46e8d793294ce384acbf2f5584ecd2 P dee2d58fd60f4b39a553c007f76ac760 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } }
I20260809 15:44:49.023375 21865 raft_consensus.cc:482] T ab976a6064754a689b7c88c5fa968b86 P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:44:49.022990 21635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dd9896939284d9cb2c03e9c21a92dfa" candidate_uuid: "faa9b2a805cd43eca626c68bc7e7168f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" is_pre_election: true
I20260809 15:44:49.036339 21768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab976a6064754a689b7c88c5fa968b86" candidate_uuid: "faa9b2a805cd43eca626c68bc7e7168f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dee2d58fd60f4b39a553c007f76ac760" is_pre_election: true
I20260809 15:44:49.039503 21858 raft_consensus.cc:482] T d203a1017c6b44d0a62bbc654ee21d8c P faa9b2a805cd43eca626c68bc7e7168f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:44:49.039965 21461 leader_election.cc:302] T ae87ef2626584e6bacd9f1af2c8991d8 P faa9b2a805cd43eca626c68bc7e7168f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dee2d58fd60f4b39a553c007f76ac760, faa9b2a805cd43eca626c68bc7e7168f; no voters:
I20260809 15:44:49.045308 21631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f46e8d793294ce384acbf2f5584ecd2" candidate_uuid: "dee2d58fd60f4b39a553c007f76ac760" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" is_pre_election: true
I20260809 15:44:49.046659 21895 raft_consensus.cc:2781] T ae87ef2626584e6bacd9f1af2c8991d8 P faa9b2a805cd43eca626c68bc7e7168f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 15:44:49.046600 21631 raft_consensus.cc:2454] T 2f46e8d793294ce384acbf2f5584ecd2 P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dee2d58fd60f4b39a553c007f76ac760 in term 0.
I20260809 15:44:49.043627 21513 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f46e8d793294ce384acbf2f5584ecd2" candidate_uuid: "dee2d58fd60f4b39a553c007f76ac760" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "faa9b2a805cd43eca626c68bc7e7168f" is_pre_election: true
I20260809 15:44:49.047818 21710 leader_election.cc:302] T 2f46e8d793294ce384acbf2f5584ecd2 P dee2d58fd60f4b39a553c007f76ac760 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6517ff96fe0a42a2bde064dc7f1ecc97, dee2d58fd60f4b39a553c007f76ac760; no voters:
I20260809 15:44:49.048862 21858 raft_consensus.cc:504] T d203a1017c6b44d0a62bbc654ee21d8c P faa9b2a805cd43eca626c68bc7e7168f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } }
W20260809 15:44:49.050455 21709 leader_election.cc:341] T 2f46e8d793294ce384acbf2f5584ecd2 P dee2d58fd60f4b39a553c007f76ac760 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer faa9b2a805cd43eca626c68bc7e7168f (127.20.202.65:37685): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:44:49.050699 21858 leader_election.cc:288] T d203a1017c6b44d0a62bbc654ee21d8c P faa9b2a805cd43eca626c68bc7e7168f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6517ff96fe0a42a2bde064dc7f1ecc97 (127.20.202.66:42701), dee2d58fd60f4b39a553c007f76ac760 (127.20.202.67:36447)
I20260809 15:44:49.047032 21895 raft_consensus.cc:482] T ae87ef2626584e6bacd9f1af2c8991d8 P faa9b2a805cd43eca626c68bc7e7168f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:44:49.051092 21895 raft_consensus.cc:3037] T ae87ef2626584e6bacd9f1af2c8991d8 P faa9b2a805cd43eca626c68bc7e7168f [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:44:49.043772 21865 raft_consensus.cc:504] T ab976a6064754a689b7c88c5fa968b86 P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } }
I20260809 15:44:49.054572 21865 leader_election.cc:288] T ab976a6064754a689b7c88c5fa968b86 P 6517ff96fe0a42a2bde064dc7f1ecc97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers faa9b2a805cd43eca626c68bc7e7168f (127.20.202.65:37685), dee2d58fd60f4b39a553c007f76ac760 (127.20.202.67:36447)
I20260809 15:44:49.040338 21866 leader_election.cc:288] T 2f46e8d793294ce384acbf2f5584ecd2 P dee2d58fd60f4b39a553c007f76ac760 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers faa9b2a805cd43eca626c68bc7e7168f (127.20.202.65:37685), 6517ff96fe0a42a2bde064dc7f1ecc97 (127.20.202.66:42701)
I20260809 15:44:49.037497 21856 ts_tablet_manager.cc:1397] T d203a1017c6b44d0a62bbc654ee21d8c P dee2d58fd60f4b39a553c007f76ac760: Time spent bootstrapping tablet: real 0.084s user 0.025s sys 0.019s
I20260809 15:44:49.035159 21769 raft_consensus.cc:2454] T 6dd9896939284d9cb2c03e9c21a92dfa P dee2d58fd60f4b39a553c007f76ac760 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate faa9b2a805cd43eca626c68bc7e7168f in term 0.
I20260809 15:44:49.056350 21768 raft_consensus.cc:2454] T ab976a6064754a689b7c88c5fa968b86 P dee2d58fd60f4b39a553c007f76ac760 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate faa9b2a805cd43eca626c68bc7e7168f in term 0.
W20260809 15:44:49.058197 21460 leader_election.cc:341] T 6dd9896939284d9cb2c03e9c21a92dfa P faa9b2a805cd43eca626c68bc7e7168f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6517ff96fe0a42a2bde064dc7f1ecc97 (127.20.202.66:42701): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:44:49.057657 21866 raft_consensus.cc:2781] T 2f46e8d793294ce384acbf2f5584ecd2 P dee2d58fd60f4b39a553c007f76ac760 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 15:44:49.062568 21866 raft_consensus.cc:482] T 2f46e8d793294ce384acbf2f5584ecd2 P dee2d58fd60f4b39a553c007f76ac760 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:44:49.063144 21461 leader_election.cc:302] T ab976a6064754a689b7c88c5fa968b86 P faa9b2a805cd43eca626c68bc7e7168f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dee2d58fd60f4b39a553c007f76ac760, faa9b2a805cd43eca626c68bc7e7168f; no voters:
I20260809 15:44:49.063212 21856 raft_consensus.cc:348] T d203a1017c6b44d0a62bbc654ee21d8c P dee2d58fd60f4b39a553c007f76ac760 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } }
I20260809 15:44:49.063906 21631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d203a1017c6b44d0a62bbc654ee21d8c" candidate_uuid: "faa9b2a805cd43eca626c68bc7e7168f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" is_pre_election: true
I20260809 15:44:49.065337 21866 raft_consensus.cc:3037] T 2f46e8d793294ce384acbf2f5584ecd2 P dee2d58fd60f4b39a553c007f76ac760 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:44:49.067050 21858 raft_consensus.cc:2781] T ab976a6064754a689b7c88c5fa968b86 P faa9b2a805cd43eca626c68bc7e7168f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 15:44:49.057142 21633 raft_consensus.cc:2454] T ab976a6064754a689b7c88c5fa968b86 P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate faa9b2a805cd43eca626c68bc7e7168f in term 0.
I20260809 15:44:49.067464 21858 raft_consensus.cc:482] T ab976a6064754a689b7c88c5fa968b86 P faa9b2a805cd43eca626c68bc7e7168f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:44:49.068626 21631 raft_consensus.cc:2454] T d203a1017c6b44d0a62bbc654ee21d8c P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate faa9b2a805cd43eca626c68bc7e7168f in term 0.
I20260809 15:44:49.065812 21856 raft_consensus.cc:374] T d203a1017c6b44d0a62bbc654ee21d8c P dee2d58fd60f4b39a553c007f76ac760 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:44:49.065220 21461 leader_election.cc:302] T 6dd9896939284d9cb2c03e9c21a92dfa P faa9b2a805cd43eca626c68bc7e7168f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dee2d58fd60f4b39a553c007f76ac760, faa9b2a805cd43eca626c68bc7e7168f; no voters: 6517ff96fe0a42a2bde064dc7f1ecc97
I20260809 15:44:49.069564 21858 raft_consensus.cc:3037] T ab976a6064754a689b7c88c5fa968b86 P faa9b2a805cd43eca626c68bc7e7168f [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:44:49.070266 21853 tablet_bootstrap.cc:492] T 6dd9896939284d9cb2c03e9c21a92dfa P 6517ff96fe0a42a2bde064dc7f1ecc97: No bootstrap required, opened a new log
I20260809 15:44:49.069882 21856 raft_consensus.cc:729] T d203a1017c6b44d0a62bbc654ee21d8c P dee2d58fd60f4b39a553c007f76ac760 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dee2d58fd60f4b39a553c007f76ac760, State: Initialized, Role: FOLLOWER
I20260809 15:44:49.100033 21853 ts_tablet_manager.cc:1397] T 6dd9896939284d9cb2c03e9c21a92dfa P 6517ff96fe0a42a2bde064dc7f1ecc97: Time spent bootstrapping tablet: real 0.139s user 0.008s sys 0.039s
I20260809 15:44:49.079411 21460 leader_election.cc:302] T d203a1017c6b44d0a62bbc654ee21d8c P faa9b2a805cd43eca626c68bc7e7168f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6517ff96fe0a42a2bde064dc7f1ecc97, faa9b2a805cd43eca626c68bc7e7168f; no voters:
I20260809 15:44:49.100217 21856 consensus_queue.cc:260] T d203a1017c6b44d0a62bbc654ee21d8c P dee2d58fd60f4b39a553c007f76ac760 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } }
I20260809 15:44:49.101969 21902 raft_consensus.cc:2781] T d203a1017c6b44d0a62bbc654ee21d8c P faa9b2a805cd43eca626c68bc7e7168f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 15:44:49.070632 21768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d203a1017c6b44d0a62bbc654ee21d8c" candidate_uuid: "faa9b2a805cd43eca626c68bc7e7168f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dee2d58fd60f4b39a553c007f76ac760" is_pre_election: true
I20260809 15:44:49.102476 21513 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "724982e80d3b455a91f8a564d374a65e" candidate_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "faa9b2a805cd43eca626c68bc7e7168f" is_pre_election: true
I20260809 15:44:49.107890 21513 raft_consensus.cc:2454] T 724982e80d3b455a91f8a564d374a65e P faa9b2a805cd43eca626c68bc7e7168f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6517ff96fe0a42a2bde064dc7f1ecc97 in term 0.
I20260809 15:44:49.103247 21902 raft_consensus.cc:482] T d203a1017c6b44d0a62bbc654ee21d8c P faa9b2a805cd43eca626c68bc7e7168f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:44:49.109902 21902 raft_consensus.cc:3037] T d203a1017c6b44d0a62bbc654ee21d8c P faa9b2a805cd43eca626c68bc7e7168f [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:44:49.098098 21866 raft_consensus.cc:504] T 2f46e8d793294ce384acbf2f5584ecd2 P dee2d58fd60f4b39a553c007f76ac760 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } }
I20260809 15:44:49.104702 21895 raft_consensus.cc:504] T ae87ef2626584e6bacd9f1af2c8991d8 P faa9b2a805cd43eca626c68bc7e7168f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } }
I20260809 15:44:49.108013 21768 raft_consensus.cc:2454] T d203a1017c6b44d0a62bbc654ee21d8c P dee2d58fd60f4b39a553c007f76ac760 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate faa9b2a805cd43eca626c68bc7e7168f in term 0.
I20260809 15:44:49.111061 21574 leader_election.cc:302] T 724982e80d3b455a91f8a564d374a65e P 6517ff96fe0a42a2bde064dc7f1ecc97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6517ff96fe0a42a2bde064dc7f1ecc97, faa9b2a805cd43eca626c68bc7e7168f; no voters:
I20260809 15:44:49.112295 21511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f46e8d793294ce384acbf2f5584ecd2" candidate_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "faa9b2a805cd43eca626c68bc7e7168f" is_pre_election: true
I20260809 15:44:49.115065 21865 raft_consensus.cc:2781] T 724982e80d3b455a91f8a564d374a65e P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 15:44:49.109962 21510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab976a6064754a689b7c88c5fa968b86" candidate_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "faa9b2a805cd43eca626c68bc7e7168f" is_pre_election: true
I20260809 15:44:49.115686 21865 raft_consensus.cc:482] T 724982e80d3b455a91f8a564d374a65e P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:44:49.116037 21865 raft_consensus.cc:3037] T 724982e80d3b455a91f8a564d374a65e P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:44:49.116545 21770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab976a6064754a689b7c88c5fa968b86" candidate_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dee2d58fd60f4b39a553c007f76ac760" is_pre_election: true
I20260809 15:44:49.117686 21770 raft_consensus.cc:2454] T ab976a6064754a689b7c88c5fa968b86 P dee2d58fd60f4b39a553c007f76ac760 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6517ff96fe0a42a2bde064dc7f1ecc97 in term 0.
W20260809 15:44:49.118104 21574 leader_election.cc:341] T 2f46e8d793294ce384acbf2f5584ecd2 P 6517ff96fe0a42a2bde064dc7f1ecc97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer faa9b2a805cd43eca626c68bc7e7168f (127.20.202.65:37685): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:44:49.119872 21769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f46e8d793294ce384acbf2f5584ecd2" candidate_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dee2d58fd60f4b39a553c007f76ac760" is_pre_election: true
I20260809 15:44:49.120844 21577 leader_election.cc:302] T ab976a6064754a689b7c88c5fa968b86 P 6517ff96fe0a42a2bde064dc7f1ecc97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6517ff96fe0a42a2bde064dc7f1ecc97, dee2d58fd60f4b39a553c007f76ac760; no voters:
I20260809 15:44:49.121495 21769 raft_consensus.cc:2379] T 2f46e8d793294ce384acbf2f5584ecd2 P dee2d58fd60f4b39a553c007f76ac760 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6517ff96fe0a42a2bde064dc7f1ecc97 in current term 1: Already voted for candidate dee2d58fd60f4b39a553c007f76ac760 in this term.
I20260809 15:44:49.119762 21633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae87ef2626584e6bacd9f1af2c8991d8" candidate_uuid: "faa9b2a805cd43eca626c68bc7e7168f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97"
I20260809 15:44:49.124114 21850 tablet_bootstrap.cc:492] T 8d9c4e17e76743439089a2a53c45d01d P faa9b2a805cd43eca626c68bc7e7168f: No bootstrap required, opened a new log
I20260809 15:44:49.124552 21850 ts_tablet_manager.cc:1397] T 8d9c4e17e76743439089a2a53c45d01d P faa9b2a805cd43eca626c68bc7e7168f: Time spent bootstrapping tablet: real 0.180s user 0.013s sys 0.008s
I20260809 15:44:49.124349 21768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "724982e80d3b455a91f8a564d374a65e" candidate_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dee2d58fd60f4b39a553c007f76ac760" is_pre_election: true
I20260809 15:44:49.124423 21633 raft_consensus.cc:3037] T ae87ef2626584e6bacd9f1af2c8991d8 P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:44:49.123553 21858 raft_consensus.cc:504] T ab976a6064754a689b7c88c5fa968b86 P faa9b2a805cd43eca626c68bc7e7168f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } }
I20260809 15:44:49.125409 21853 raft_consensus.cc:348] T 6dd9896939284d9cb2c03e9c21a92dfa P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } }
I20260809 15:44:49.126065 21853 raft_consensus.cc:374] T 6dd9896939284d9cb2c03e9c21a92dfa P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:44:49.126319 21853 raft_consensus.cc:729] T 6dd9896939284d9cb2c03e9c21a92dfa P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6517ff96fe0a42a2bde064dc7f1ecc97, State: Initialized, Role: FOLLOWER
I20260809 15:44:49.126864 21853 consensus_queue.cc:260] T 6dd9896939284d9cb2c03e9c21a92dfa P 6517ff96fe0a42a2bde064dc7f1ecc97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } }
I20260809 15:44:49.127496 21866 leader_election.cc:288] T 2f46e8d793294ce384acbf2f5584ecd2 P dee2d58fd60f4b39a553c007f76ac760 [CANDIDATE]: Term 1 election: Requested vote from peers faa9b2a805cd43eca626c68bc7e7168f (127.20.202.65:37685), 6517ff96fe0a42a2bde064dc7f1ecc97 (127.20.202.66:42701)
I20260809 15:44:49.127841 21511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f46e8d793294ce384acbf2f5584ecd2" candidate_uuid: "dee2d58fd60f4b39a553c007f76ac760" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "faa9b2a805cd43eca626c68bc7e7168f"
I20260809 15:44:49.128211 21631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f46e8d793294ce384acbf2f5584ecd2" candidate_uuid: "dee2d58fd60f4b39a553c007f76ac760" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97"
I20260809 15:44:49.128767 21631 raft_consensus.cc:3037] T 2f46e8d793294ce384acbf2f5584ecd2 P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 0 FOLLOWER]: Advancing to term 1
W20260809 15:44:49.129101 21709 leader_election.cc:341] T 2f46e8d793294ce384acbf2f5584ecd2 P dee2d58fd60f4b39a553c007f76ac760 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer faa9b2a805cd43eca626c68bc7e7168f (127.20.202.65:37685): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:44:49.134876 21631 raft_consensus.cc:2454] T 2f46e8d793294ce384acbf2f5584ecd2 P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dee2d58fd60f4b39a553c007f76ac760 in term 1.
I20260809 15:44:49.140460 21710 leader_election.cc:302] T 2f46e8d793294ce384acbf2f5584ecd2 P dee2d58fd60f4b39a553c007f76ac760 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6517ff96fe0a42a2bde064dc7f1ecc97, dee2d58fd60f4b39a553c007f76ac760; no voters: faa9b2a805cd43eca626c68bc7e7168f
I20260809 15:44:49.141219 21866 raft_consensus.cc:2781] T 2f46e8d793294ce384acbf2f5584ecd2 P dee2d58fd60f4b39a553c007f76ac760 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:44:49.143208 21510 raft_consensus.cc:2379] T ab976a6064754a689b7c88c5fa968b86 P faa9b2a805cd43eca626c68bc7e7168f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6517ff96fe0a42a2bde064dc7f1ecc97 in current term 1: Already voted for candidate faa9b2a805cd43eca626c68bc7e7168f in this term.
I20260809 15:44:49.143412 21858 leader_election.cc:288] T ab976a6064754a689b7c88c5fa968b86 P faa9b2a805cd43eca626c68bc7e7168f [CANDIDATE]: Term 1 election: Requested vote from peers 6517ff96fe0a42a2bde064dc7f1ecc97 (127.20.202.66:42701), dee2d58fd60f4b39a553c007f76ac760 (127.20.202.67:36447)
I20260809 15:44:49.143944 21858 raft_consensus.cc:2781] T 6dd9896939284d9cb2c03e9c21a92dfa P faa9b2a805cd43eca626c68bc7e7168f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 15:44:49.144261 21858 raft_consensus.cc:482] T 6dd9896939284d9cb2c03e9c21a92dfa P faa9b2a805cd43eca626c68bc7e7168f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:44:49.144573 21858 raft_consensus.cc:3037] T 6dd9896939284d9cb2c03e9c21a92dfa P faa9b2a805cd43eca626c68bc7e7168f [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:44:49.143527 21856 ts_tablet_manager.cc:1428] T d203a1017c6b44d0a62bbc654ee21d8c P dee2d58fd60f4b39a553c007f76ac760: Time spent starting tablet: real 0.086s user 0.018s sys 0.007s
I20260809 15:44:49.143064 21895 leader_election.cc:288] T ae87ef2626584e6bacd9f1af2c8991d8 P faa9b2a805cd43eca626c68bc7e7168f [CANDIDATE]: Term 1 election: Requested vote from peers 6517ff96fe0a42a2bde064dc7f1ecc97 (127.20.202.66:42701), dee2d58fd60f4b39a553c007f76ac760 (127.20.202.67:36447)
I20260809 15:44:49.142153 21866 raft_consensus.cc:686] T 2f46e8d793294ce384acbf2f5584ecd2 P dee2d58fd60f4b39a553c007f76ac760 [term 1 LEADER]: Becoming Leader. State: Replica: dee2d58fd60f4b39a553c007f76ac760, State: Running, Role: LEADER
I20260809 15:44:49.147578 21856 tablet_bootstrap.cc:492] T d18bbcc7ebd14b479ddacb4aad814d47 P dee2d58fd60f4b39a553c007f76ac760: Bootstrap starting.
I20260809 15:44:49.148561 21902 raft_consensus.cc:504] T d203a1017c6b44d0a62bbc654ee21d8c P faa9b2a805cd43eca626c68bc7e7168f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } }
I20260809 15:44:49.148891 21633 raft_consensus.cc:2454] T ae87ef2626584e6bacd9f1af2c8991d8 P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate faa9b2a805cd43eca626c68bc7e7168f in term 1.
I20260809 15:44:49.150233 21866 consensus_queue.cc:237] T 2f46e8d793294ce384acbf2f5584ecd2 P dee2d58fd60f4b39a553c007f76ac760 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } }
I20260809 15:44:49.151765 21460 leader_election.cc:302] T ae87ef2626584e6bacd9f1af2c8991d8 P faa9b2a805cd43eca626c68bc7e7168f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6517ff96fe0a42a2bde064dc7f1ecc97, faa9b2a805cd43eca626c68bc7e7168f; no voters:
I20260809 15:44:49.152248 21902 leader_election.cc:288] T d203a1017c6b44d0a62bbc654ee21d8c P faa9b2a805cd43eca626c68bc7e7168f [CANDIDATE]: Term 1 election: Requested vote from peers 6517ff96fe0a42a2bde064dc7f1ecc97 (127.20.202.66:42701), dee2d58fd60f4b39a553c007f76ac760 (127.20.202.67:36447)
I20260809 15:44:49.152720 21902 raft_consensus.cc:2781] T ae87ef2626584e6bacd9f1af2c8991d8 P faa9b2a805cd43eca626c68bc7e7168f [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:44:49.153113 21902 raft_consensus.cc:686] T ae87ef2626584e6bacd9f1af2c8991d8 P faa9b2a805cd43eca626c68bc7e7168f [term 1 LEADER]: Becoming Leader. State: Replica: faa9b2a805cd43eca626c68bc7e7168f, State: Running, Role: LEADER
I20260809 15:44:49.153409 21635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d203a1017c6b44d0a62bbc654ee21d8c" candidate_uuid: "faa9b2a805cd43eca626c68bc7e7168f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97"
I20260809 15:44:49.153929 21635 raft_consensus.cc:3037] T d203a1017c6b44d0a62bbc654ee21d8c P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:44:49.150233 21850 raft_consensus.cc:348] T 8d9c4e17e76743439089a2a53c45d01d P faa9b2a805cd43eca626c68bc7e7168f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } }
I20260809 15:44:49.155133 21850 raft_consensus.cc:374] T 8d9c4e17e76743439089a2a53c45d01d P faa9b2a805cd43eca626c68bc7e7168f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:44:49.155411 21850 raft_consensus.cc:729] T 8d9c4e17e76743439089a2a53c45d01d P faa9b2a805cd43eca626c68bc7e7168f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: faa9b2a805cd43eca626c68bc7e7168f, State: Initialized, Role: FOLLOWER
I20260809 15:44:49.155974 21850 consensus_queue.cc:260] T 8d9c4e17e76743439089a2a53c45d01d P faa9b2a805cd43eca626c68bc7e7168f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } }
I20260809 15:44:49.153842 21902 consensus_queue.cc:237] T ae87ef2626584e6bacd9f1af2c8991d8 P faa9b2a805cd43eca626c68bc7e7168f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } }
I20260809 15:44:49.158313 21631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab976a6064754a689b7c88c5fa968b86" candidate_uuid: "faa9b2a805cd43eca626c68bc7e7168f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97"
I20260809 15:44:49.158859 21631 raft_consensus.cc:3037] T ab976a6064754a689b7c88c5fa968b86 P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:44:49.161948 21858 raft_consensus.cc:504] T 6dd9896939284d9cb2c03e9c21a92dfa P faa9b2a805cd43eca626c68bc7e7168f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } }
I20260809 15:44:49.163990 21915 raft_consensus.cc:482] T 8d9c4e17e76743439089a2a53c45d01d P dee2d58fd60f4b39a553c007f76ac760 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:44:49.165938 21850 ts_tablet_manager.cc:1428] T 8d9c4e17e76743439089a2a53c45d01d P faa9b2a805cd43eca626c68bc7e7168f: Time spent starting tablet: real 0.041s user 0.007s sys 0.000s
I20260809 15:44:49.166792 21850 tablet_bootstrap.cc:492] T 2f46e8d793294ce384acbf2f5584ecd2 P faa9b2a805cd43eca626c68bc7e7168f: Bootstrap starting.
I20260809 15:44:49.164450 21915 raft_consensus.cc:504] T 8d9c4e17e76743439089a2a53c45d01d P dee2d58fd60f4b39a553c007f76ac760 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } }
W20260809 15:44:49.181656 21577 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.67:36447, user_credentials={real_user=slave}} blocked reactor thread for 61255.8us
I20260809 15:44:49.185684 21853 ts_tablet_manager.cc:1428] T 6dd9896939284d9cb2c03e9c21a92dfa P 6517ff96fe0a42a2bde064dc7f1ecc97: Time spent starting tablet: real 0.085s user 0.020s sys 0.011s
I20260809 15:44:49.185432 21633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dd9896939284d9cb2c03e9c21a92dfa" candidate_uuid: "faa9b2a805cd43eca626c68bc7e7168f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97"
I20260809 15:44:49.171919 21865 raft_consensus.cc:504] T 724982e80d3b455a91f8a564d374a65e P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } }
I20260809 15:44:49.188668 21631 raft_consensus.cc:2454] T ab976a6064754a689b7c88c5fa968b86 P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate faa9b2a805cd43eca626c68bc7e7168f in term 1.
I20260809 15:44:49.186522 21633 raft_consensus.cc:3037] T 6dd9896939284d9cb2c03e9c21a92dfa P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:44:49.191023 21920 raft_consensus.cc:2741] T ab976a6064754a689b7c88c5fa968b86 P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260809 15:44:49.192291 21865 leader_election.cc:288] T 724982e80d3b455a91f8a564d374a65e P 6517ff96fe0a42a2bde064dc7f1ecc97 [CANDIDATE]: Term 1 election: Requested vote from peers faa9b2a805cd43eca626c68bc7e7168f (127.20.202.65:37685), dee2d58fd60f4b39a553c007f76ac760 (127.20.202.67:36447)
I20260809 15:44:49.192692 21856 tablet_bootstrap.cc:654] T d18bbcc7ebd14b479ddacb4aad814d47 P dee2d58fd60f4b39a553c007f76ac760: Neither blocks nor log segments found. Creating new log.
I20260809 15:44:49.192569 21510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "724982e80d3b455a91f8a564d374a65e" candidate_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "faa9b2a805cd43eca626c68bc7e7168f"
I20260809 15:44:49.190863 21635 raft_consensus.cc:2454] T d203a1017c6b44d0a62bbc654ee21d8c P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate faa9b2a805cd43eca626c68bc7e7168f in term 1.
I20260809 15:44:49.193125 21510 raft_consensus.cc:3037] T 724982e80d3b455a91f8a564d374a65e P faa9b2a805cd43eca626c68bc7e7168f [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:44:49.201232 21460 leader_election.cc:302] T ab976a6064754a689b7c88c5fa968b86 P faa9b2a805cd43eca626c68bc7e7168f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6517ff96fe0a42a2bde064dc7f1ecc97, faa9b2a805cd43eca626c68bc7e7168f; no voters:
I20260809 15:44:49.223479 21850 tablet_bootstrap.cc:654] T 2f46e8d793294ce384acbf2f5584ecd2 P faa9b2a805cd43eca626c68bc7e7168f: Neither blocks nor log segments found. Creating new log.
I20260809 15:44:49.190146 21853 tablet_bootstrap.cc:492] T 8d9c4e17e76743439089a2a53c45d01d P 6517ff96fe0a42a2bde064dc7f1ecc97: Bootstrap starting.
I20260809 15:44:49.193274 21915 leader_election.cc:288] T 8d9c4e17e76743439089a2a53c45d01d P dee2d58fd60f4b39a553c007f76ac760 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers faa9b2a805cd43eca626c68bc7e7168f (127.20.202.65:37685), 6517ff96fe0a42a2bde064dc7f1ecc97 (127.20.202.66:42701)
I20260809 15:44:49.224144 21919 raft_consensus.cc:2781] T ab976a6064754a689b7c88c5fa968b86 P faa9b2a805cd43eca626c68bc7e7168f [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:44:49.189801 21858 leader_election.cc:288] T 6dd9896939284d9cb2c03e9c21a92dfa P faa9b2a805cd43eca626c68bc7e7168f [CANDIDATE]: Term 1 election: Requested vote from peers 6517ff96fe0a42a2bde064dc7f1ecc97 (127.20.202.66:42701), dee2d58fd60f4b39a553c007f76ac760 (127.20.202.67:36447)
I20260809 15:44:49.212800 21633 raft_consensus.cc:2454] T 6dd9896939284d9cb2c03e9c21a92dfa P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate faa9b2a805cd43eca626c68bc7e7168f in term 1.
I20260809 15:44:49.194972 21634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d9c4e17e76743439089a2a53c45d01d" candidate_uuid: "dee2d58fd60f4b39a553c007f76ac760" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" is_pre_election: true
I20260809 15:44:49.226718 21919 raft_consensus.cc:686] T ab976a6064754a689b7c88c5fa968b86 P faa9b2a805cd43eca626c68bc7e7168f [term 1 LEADER]: Becoming Leader. State: Replica: faa9b2a805cd43eca626c68bc7e7168f, State: Running, Role: LEADER
I20260809 15:44:49.187634 21577 leader_election.cc:302] T 2f46e8d793294ce384acbf2f5584ecd2 P 6517ff96fe0a42a2bde064dc7f1ecc97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6517ff96fe0a42a2bde064dc7f1ecc97; no voters: dee2d58fd60f4b39a553c007f76ac760, faa9b2a805cd43eca626c68bc7e7168f
I20260809 15:44:49.194746 21511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d9c4e17e76743439089a2a53c45d01d" candidate_uuid: "dee2d58fd60f4b39a553c007f76ac760" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "faa9b2a805cd43eca626c68bc7e7168f" is_pre_election: true
W20260809 15:44:49.231870 21577 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.67:36447, user_credentials={real_user=slave}} blocked reactor thread for 48733.8us
I20260809 15:44:49.235538 21923 raft_consensus.cc:2726] T 2f46e8d793294ce384acbf2f5584ecd2 P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260809 15:44:49.233465 21710 leader_election.cc:341] T 8d9c4e17e76743439089a2a53c45d01d P dee2d58fd60f4b39a553c007f76ac760 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6517ff96fe0a42a2bde064dc7f1ecc97 (127.20.202.66:42701): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:44:49.232676 21511 raft_consensus.cc:2454] T 8d9c4e17e76743439089a2a53c45d01d P faa9b2a805cd43eca626c68bc7e7168f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dee2d58fd60f4b39a553c007f76ac760 in term 0.
I20260809 15:44:49.230261 21919 consensus_queue.cc:237] T ab976a6064754a689b7c88c5fa968b86 P faa9b2a805cd43eca626c68bc7e7168f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } }
I20260809 15:44:49.238787 21709 leader_election.cc:302] T 8d9c4e17e76743439089a2a53c45d01d P dee2d58fd60f4b39a553c007f76ac760 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dee2d58fd60f4b39a553c007f76ac760, faa9b2a805cd43eca626c68bc7e7168f; no voters: 6517ff96fe0a42a2bde064dc7f1ecc97
I20260809 15:44:49.240304 21915 raft_consensus.cc:2781] T 8d9c4e17e76743439089a2a53c45d01d P dee2d58fd60f4b39a553c007f76ac760 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 15:44:49.240657 21915 raft_consensus.cc:482] T 8d9c4e17e76743439089a2a53c45d01d P dee2d58fd60f4b39a553c007f76ac760 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:44:49.240972 21915 raft_consensus.cc:3037] T 8d9c4e17e76743439089a2a53c45d01d P dee2d58fd60f4b39a553c007f76ac760 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:44:49.232842 21460 leader_election.cc:302] T d203a1017c6b44d0a62bbc654ee21d8c P faa9b2a805cd43eca626c68bc7e7168f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6517ff96fe0a42a2bde064dc7f1ecc97, faa9b2a805cd43eca626c68bc7e7168f; no voters:
I20260809 15:44:49.248966 21853 tablet_bootstrap.cc:654] T 8d9c4e17e76743439089a2a53c45d01d P 6517ff96fe0a42a2bde064dc7f1ecc97: Neither blocks nor log segments found. Creating new log.
I20260809 15:44:49.250483 21460 leader_election.cc:302] T 6dd9896939284d9cb2c03e9c21a92dfa P faa9b2a805cd43eca626c68bc7e7168f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6517ff96fe0a42a2bde064dc7f1ecc97, faa9b2a805cd43eca626c68bc7e7168f; no voters:
I20260809 15:44:49.251077 21856 tablet_bootstrap.cc:492] T d18bbcc7ebd14b479ddacb4aad814d47 P dee2d58fd60f4b39a553c007f76ac760: No bootstrap required, opened a new log
I20260809 15:44:49.251415 21858 raft_consensus.cc:2781] T 6dd9896939284d9cb2c03e9c21a92dfa P faa9b2a805cd43eca626c68bc7e7168f [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:44:49.251538 21856 ts_tablet_manager.cc:1397] T d18bbcc7ebd14b479ddacb4aad814d47 P dee2d58fd60f4b39a553c007f76ac760: Time spent bootstrapping tablet: real 0.104s user 0.009s sys 0.010s
I20260809 15:44:49.251906 21858 raft_consensus.cc:686] T 6dd9896939284d9cb2c03e9c21a92dfa P faa9b2a805cd43eca626c68bc7e7168f [term 1 LEADER]: Becoming Leader. State: Replica: faa9b2a805cd43eca626c68bc7e7168f, State: Running, Role: LEADER
I20260809 15:44:49.253149 21858 consensus_queue.cc:237] T 6dd9896939284d9cb2c03e9c21a92dfa P faa9b2a805cd43eca626c68bc7e7168f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } }
W20260809 15:44:49.244234 21577 leader_election.cc:341] T 724982e80d3b455a91f8a564d374a65e P 6517ff96fe0a42a2bde064dc7f1ecc97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dee2d58fd60f4b39a553c007f76ac760 (127.20.202.67:36447): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:44:49.251554 21510 raft_consensus.cc:2454] T 724982e80d3b455a91f8a564d374a65e P faa9b2a805cd43eca626c68bc7e7168f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6517ff96fe0a42a2bde064dc7f1ecc97 in term 1.
I20260809 15:44:49.245992 21902 raft_consensus.cc:2781] T d203a1017c6b44d0a62bbc654ee21d8c P faa9b2a805cd43eca626c68bc7e7168f [term 1 FOLLOWER]: Leader election won for term 1
W20260809 15:44:49.256556 21461 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.67:36447, user_credentials={real_user=slave}} blocked reactor thread for 191413us
I20260809 15:44:49.259503 21915 raft_consensus.cc:504] T 8d9c4e17e76743439089a2a53c45d01d P dee2d58fd60f4b39a553c007f76ac760 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } }
I20260809 15:44:49.260138 21768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae87ef2626584e6bacd9f1af2c8991d8" candidate_uuid: "faa9b2a805cd43eca626c68bc7e7168f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dee2d58fd60f4b39a553c007f76ac760"
I20260809 15:44:49.269721 21574 leader_election.cc:302] T 724982e80d3b455a91f8a564d374a65e P 6517ff96fe0a42a2bde064dc7f1ecc97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6517ff96fe0a42a2bde064dc7f1ecc97, faa9b2a805cd43eca626c68bc7e7168f; no voters:
W20260809 15:44:49.334304 21577 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.67:36447, user_credentials={real_user=slave}} blocked reactor thread for 91171.7us
I20260809 15:44:49.334933 21634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d9c4e17e76743439089a2a53c45d01d" candidate_uuid: "dee2d58fd60f4b39a553c007f76ac760" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97"
I20260809 15:44:49.270738 21923 raft_consensus.cc:2781] T 724982e80d3b455a91f8a564d374a65e P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:44:49.263649 21902 raft_consensus.cc:686] T d203a1017c6b44d0a62bbc654ee21d8c P faa9b2a805cd43eca626c68bc7e7168f [term 1 LEADER]: Becoming Leader. State: Replica: faa9b2a805cd43eca626c68bc7e7168f, State: Running, Role: LEADER
I20260809 15:44:49.335960 21769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "724982e80d3b455a91f8a564d374a65e" candidate_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dee2d58fd60f4b39a553c007f76ac760"
I20260809 15:44:49.268775 21768 raft_consensus.cc:3037] T ae87ef2626584e6bacd9f1af2c8991d8 P dee2d58fd60f4b39a553c007f76ac760 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:44:49.337838 21850 tablet_bootstrap.cc:492] T 2f46e8d793294ce384acbf2f5584ecd2 P faa9b2a805cd43eca626c68bc7e7168f: No bootstrap required, opened a new log
I20260809 15:44:49.338272 21850 ts_tablet_manager.cc:1397] T 2f46e8d793294ce384acbf2f5584ecd2 P faa9b2a805cd43eca626c68bc7e7168f: Time spent bootstrapping tablet: real 0.172s user 0.005s sys 0.016s
I20260809 15:44:49.337324 21902 consensus_queue.cc:237] T d203a1017c6b44d0a62bbc654ee21d8c P faa9b2a805cd43eca626c68bc7e7168f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } }
I20260809 15:44:49.338538 21510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d9c4e17e76743439089a2a53c45d01d" candidate_uuid: "dee2d58fd60f4b39a553c007f76ac760" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "faa9b2a805cd43eca626c68bc7e7168f"
I20260809 15:44:49.339095 21510 raft_consensus.cc:3037] T 8d9c4e17e76743439089a2a53c45d01d P faa9b2a805cd43eca626c68bc7e7168f [term 0 FOLLOWER]: Advancing to term 1
W20260809 15:44:49.341015 21577 leader_election.cc:341] T 724982e80d3b455a91f8a564d374a65e P 6517ff96fe0a42a2bde064dc7f1ecc97 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer dee2d58fd60f4b39a553c007f76ac760 (127.20.202.67:36447): Illegal state: must be running to vote when last-logged opid is not known
W20260809 15:44:49.341641 21710 leader_election.cc:341] T 8d9c4e17e76743439089a2a53c45d01d P dee2d58fd60f4b39a553c007f76ac760 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 6517ff96fe0a42a2bde064dc7f1ecc97 (127.20.202.66:42701): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:44:49.344820 21768 raft_consensus.cc:2454] T ae87ef2626584e6bacd9f1af2c8991d8 P dee2d58fd60f4b39a553c007f76ac760 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate faa9b2a805cd43eca626c68bc7e7168f in term 1.
I20260809 15:44:49.344498 21339 catalog_manager.cc:5495] T 6dd9896939284d9cb2c03e9c21a92dfa P faa9b2a805cd43eca626c68bc7e7168f reported cstate change: term changed from 0 to 1, leader changed from <none> to faa9b2a805cd43eca626c68bc7e7168f (127.20.202.65). New cstate: current_term: 1 leader_uuid: "faa9b2a805cd43eca626c68bc7e7168f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } health_report { overall_health: UNKNOWN } } }
I20260809 15:44:49.348227 21769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d203a1017c6b44d0a62bbc654ee21d8c" candidate_uuid: "faa9b2a805cd43eca626c68bc7e7168f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dee2d58fd60f4b39a553c007f76ac760"
I20260809 15:44:49.318987 21915 leader_election.cc:288] T 8d9c4e17e76743439089a2a53c45d01d P dee2d58fd60f4b39a553c007f76ac760 [CANDIDATE]: Term 1 election: Requested vote from peers faa9b2a805cd43eca626c68bc7e7168f (127.20.202.65:37685), 6517ff96fe0a42a2bde064dc7f1ecc97 (127.20.202.66:42701)
I20260809 15:44:49.348789 21769 raft_consensus.cc:3037] T d203a1017c6b44d0a62bbc654ee21d8c P dee2d58fd60f4b39a553c007f76ac760 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:44:49.270363 21333 catalog_manager.cc:5495] T 2f46e8d793294ce384acbf2f5584ecd2 P dee2d58fd60f4b39a553c007f76ac760 reported cstate change: term changed from 0 to 1, leader changed from <none> to dee2d58fd60f4b39a553c007f76ac760 (127.20.202.67). New cstate: current_term: 1 leader_uuid: "dee2d58fd60f4b39a553c007f76ac760" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } health_report { overall_health: HEALTHY } } }
I20260809 15:44:49.346045 21923 raft_consensus.cc:686] T 724982e80d3b455a91f8a564d374a65e P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 1 LEADER]: Becoming Leader. State: Replica: 6517ff96fe0a42a2bde064dc7f1ecc97, State: Running, Role: LEADER
I20260809 15:44:49.350176 21770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dd9896939284d9cb2c03e9c21a92dfa" candidate_uuid: "faa9b2a805cd43eca626c68bc7e7168f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dee2d58fd60f4b39a553c007f76ac760"
I20260809 15:44:49.351119 21770 raft_consensus.cc:3037] T 6dd9896939284d9cb2c03e9c21a92dfa P dee2d58fd60f4b39a553c007f76ac760 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:44:49.350919 21923 consensus_queue.cc:237] T 724982e80d3b455a91f8a564d374a65e P 6517ff96fe0a42a2bde064dc7f1ecc97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } }
I20260809 15:44:49.351912 21850 raft_consensus.cc:348] T 2f46e8d793294ce384acbf2f5584ecd2 P faa9b2a805cd43eca626c68bc7e7168f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } }
I20260809 15:44:49.352532 21850 raft_consensus.cc:374] T 2f46e8d793294ce384acbf2f5584ecd2 P faa9b2a805cd43eca626c68bc7e7168f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:44:49.352806 21850 raft_consensus.cc:729] T 2f46e8d793294ce384acbf2f5584ecd2 P faa9b2a805cd43eca626c68bc7e7168f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: faa9b2a805cd43eca626c68bc7e7168f, State: Initialized, Role: FOLLOWER
I20260809 15:44:49.346807 21510 raft_consensus.cc:2454] T 8d9c4e17e76743439089a2a53c45d01d P faa9b2a805cd43eca626c68bc7e7168f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dee2d58fd60f4b39a553c007f76ac760 in term 1.
I20260809 15:44:49.353962 21850 consensus_queue.cc:260] T 2f46e8d793294ce384acbf2f5584ecd2 P faa9b2a805cd43eca626c68bc7e7168f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } }
I20260809 15:44:49.355612 21850 ts_tablet_manager.cc:1428] T 2f46e8d793294ce384acbf2f5584ecd2 P faa9b2a805cd43eca626c68bc7e7168f: Time spent starting tablet: real 0.017s user 0.003s sys 0.003s
I20260809 15:44:49.354269 21856 raft_consensus.cc:348] T d18bbcc7ebd14b479ddacb4aad814d47 P dee2d58fd60f4b39a553c007f76ac760 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } }
I20260809 15:44:49.356005 21856 raft_consensus.cc:374] T d18bbcc7ebd14b479ddacb4aad814d47 P dee2d58fd60f4b39a553c007f76ac760 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:44:49.356292 21856 raft_consensus.cc:729] T d18bbcc7ebd14b479ddacb4aad814d47 P dee2d58fd60f4b39a553c007f76ac760 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dee2d58fd60f4b39a553c007f76ac760, State: Initialized, Role: FOLLOWER
I20260809 15:44:49.356787 21856 consensus_queue.cc:260] T d18bbcc7ebd14b479ddacb4aad814d47 P dee2d58fd60f4b39a553c007f76ac760 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } }
I20260809 15:44:49.358325 21856 ts_tablet_manager.cc:1428] T d18bbcc7ebd14b479ddacb4aad814d47 P dee2d58fd60f4b39a553c007f76ac760: Time spent starting tablet: real 0.106s user 0.004s sys 0.000s
I20260809 15:44:49.358951 21856 tablet_bootstrap.cc:492] T 724982e80d3b455a91f8a564d374a65e P dee2d58fd60f4b39a553c007f76ac760: Bootstrap starting.
I20260809 15:44:49.359522 21339 catalog_manager.cc:5495] T ae87ef2626584e6bacd9f1af2c8991d8 P faa9b2a805cd43eca626c68bc7e7168f reported cstate change: term changed from 0 to 1, leader changed from <none> to faa9b2a805cd43eca626c68bc7e7168f (127.20.202.65). New cstate: current_term: 1 leader_uuid: "faa9b2a805cd43eca626c68bc7e7168f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } health_report { overall_health: UNKNOWN } } }
I20260809 15:44:49.352933 21768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab976a6064754a689b7c88c5fa968b86" candidate_uuid: "faa9b2a805cd43eca626c68bc7e7168f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dee2d58fd60f4b39a553c007f76ac760"
I20260809 15:44:49.365917 21709 leader_election.cc:302] T 8d9c4e17e76743439089a2a53c45d01d P dee2d58fd60f4b39a553c007f76ac760 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dee2d58fd60f4b39a553c007f76ac760, faa9b2a805cd43eca626c68bc7e7168f; no voters: 6517ff96fe0a42a2bde064dc7f1ecc97
I20260809 15:44:49.365543 21768 raft_consensus.cc:3037] T ab976a6064754a689b7c88c5fa968b86 P dee2d58fd60f4b39a553c007f76ac760 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:44:49.371510 21915 raft_consensus.cc:2781] T 8d9c4e17e76743439089a2a53c45d01d P dee2d58fd60f4b39a553c007f76ac760 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:44:49.361264 21339 catalog_manager.cc:5495] T d18bbcc7ebd14b479ddacb4aad814d47 P faa9b2a805cd43eca626c68bc7e7168f reported cstate change: term changed from 0 to 1, leader changed from <none> to faa9b2a805cd43eca626c68bc7e7168f (127.20.202.65). New cstate: current_term: 1 leader_uuid: "faa9b2a805cd43eca626c68bc7e7168f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } health_report { overall_health: UNKNOWN } } }
I20260809 15:44:49.376629 21339 catalog_manager.cc:5495] T ab976a6064754a689b7c88c5fa968b86 P faa9b2a805cd43eca626c68bc7e7168f reported cstate change: term changed from 0 to 1, leader changed from <none> to faa9b2a805cd43eca626c68bc7e7168f (127.20.202.65). New cstate: current_term: 1 leader_uuid: "faa9b2a805cd43eca626c68bc7e7168f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } health_report { overall_health: UNKNOWN } } }
I20260809 15:44:49.380190 21768 raft_consensus.cc:2454] T ab976a6064754a689b7c88c5fa968b86 P dee2d58fd60f4b39a553c007f76ac760 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate faa9b2a805cd43eca626c68bc7e7168f in term 1.
I20260809 15:44:49.383096 21853 tablet_bootstrap.cc:492] T 8d9c4e17e76743439089a2a53c45d01d P 6517ff96fe0a42a2bde064dc7f1ecc97: No bootstrap required, opened a new log
I20260809 15:44:49.383527 21853 ts_tablet_manager.cc:1397] T 8d9c4e17e76743439089a2a53c45d01d P 6517ff96fe0a42a2bde064dc7f1ecc97: Time spent bootstrapping tablet: real 0.194s user 0.010s sys 0.017s
I20260809 15:44:49.381415 21915 raft_consensus.cc:686] T 8d9c4e17e76743439089a2a53c45d01d P dee2d58fd60f4b39a553c007f76ac760 [term 1 LEADER]: Becoming Leader. State: Replica: dee2d58fd60f4b39a553c007f76ac760, State: Running, Role: LEADER
I20260809 15:44:49.386416 21853 raft_consensus.cc:348] T 8d9c4e17e76743439089a2a53c45d01d P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } }
I20260809 15:44:49.387060 21853 raft_consensus.cc:374] T 8d9c4e17e76743439089a2a53c45d01d P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:44:49.387341 21853 raft_consensus.cc:729] T 8d9c4e17e76743439089a2a53c45d01d P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6517ff96fe0a42a2bde064dc7f1ecc97, State: Initialized, Role: FOLLOWER
I20260809 15:44:49.384892 21770 raft_consensus.cc:2454] T 6dd9896939284d9cb2c03e9c21a92dfa P dee2d58fd60f4b39a553c007f76ac760 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate faa9b2a805cd43eca626c68bc7e7168f in term 1.
I20260809 15:44:49.387845 21915 consensus_queue.cc:237] T 8d9c4e17e76743439089a2a53c45d01d P dee2d58fd60f4b39a553c007f76ac760 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } }
I20260809 15:44:49.388520 21853 consensus_queue.cc:260] T 8d9c4e17e76743439089a2a53c45d01d P 6517ff96fe0a42a2bde064dc7f1ecc97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } }
I20260809 15:44:49.389987 21769 raft_consensus.cc:2454] T d203a1017c6b44d0a62bbc654ee21d8c P dee2d58fd60f4b39a553c007f76ac760 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate faa9b2a805cd43eca626c68bc7e7168f in term 1.
I20260809 15:44:49.391462 21853 ts_tablet_manager.cc:1428] T 8d9c4e17e76743439089a2a53c45d01d P 6517ff96fe0a42a2bde064dc7f1ecc97: Time spent starting tablet: real 0.008s user 0.001s sys 0.005s
I20260809 15:44:49.401258 21856 tablet_bootstrap.cc:654] T 724982e80d3b455a91f8a564d374a65e P dee2d58fd60f4b39a553c007f76ac760: Neither blocks nor log segments found. Creating new log.
W20260809 15:44:49.415903 21461 consensus_peers.cc:487] T 4fe458f602674472bedadfdd3c7ee1e6 P faa9b2a805cd43eca626c68bc7e7168f -> Peer dee2d58fd60f4b39a553c007f76ac760 (127.20.202.67:36447): Couldn't send request to peer dee2d58fd60f4b39a553c007f76ac760. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260809 15:44:49.417431 21341 catalog_manager.cc:5495] T 8d9c4e17e76743439089a2a53c45d01d P dee2d58fd60f4b39a553c007f76ac760 reported cstate change: term changed from 0 to 1, leader changed from <none> to dee2d58fd60f4b39a553c007f76ac760 (127.20.202.67). New cstate: current_term: 1 leader_uuid: "dee2d58fd60f4b39a553c007f76ac760" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } health_report { overall_health: HEALTHY } } }
I20260809 15:44:49.420082 21331 catalog_manager.cc:5495] T d203a1017c6b44d0a62bbc654ee21d8c P faa9b2a805cd43eca626c68bc7e7168f reported cstate change: term changed from 0 to 1, leader changed from <none> to faa9b2a805cd43eca626c68bc7e7168f (127.20.202.65). New cstate: current_term: 1 leader_uuid: "faa9b2a805cd43eca626c68bc7e7168f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } health_report { overall_health: UNKNOWN } } }
I20260809 15:44:49.423504 21856 tablet_bootstrap.cc:492] T 724982e80d3b455a91f8a564d374a65e P dee2d58fd60f4b39a553c007f76ac760: No bootstrap required, opened a new log
I20260809 15:44:49.423923 21856 ts_tablet_manager.cc:1397] T 724982e80d3b455a91f8a564d374a65e P dee2d58fd60f4b39a553c007f76ac760: Time spent bootstrapping tablet: real 0.065s user 0.013s sys 0.010s
I20260809 15:44:49.427383 21856 raft_consensus.cc:348] T 724982e80d3b455a91f8a564d374a65e P dee2d58fd60f4b39a553c007f76ac760 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } }
I20260809 15:44:49.428038 21856 raft_consensus.cc:374] T 724982e80d3b455a91f8a564d374a65e P dee2d58fd60f4b39a553c007f76ac760 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:44:49.429198 21856 raft_consensus.cc:729] T 724982e80d3b455a91f8a564d374a65e P dee2d58fd60f4b39a553c007f76ac760 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dee2d58fd60f4b39a553c007f76ac760, State: Initialized, Role: FOLLOWER
I20260809 15:44:49.429827 21856 consensus_queue.cc:260] T 724982e80d3b455a91f8a564d374a65e P dee2d58fd60f4b39a553c007f76ac760 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } }
I20260809 15:44:49.430009 21339 catalog_manager.cc:5495] T 724982e80d3b455a91f8a564d374a65e P 6517ff96fe0a42a2bde064dc7f1ecc97 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6517ff96fe0a42a2bde064dc7f1ecc97 (127.20.202.66). New cstate: current_term: 1 leader_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } health_report { overall_health: UNKNOWN } } }
I20260809 15:44:49.431943 21856 ts_tablet_manager.cc:1428] T 724982e80d3b455a91f8a564d374a65e P dee2d58fd60f4b39a553c007f76ac760: Time spent starting tablet: real 0.008s user 0.004s sys 0.003s
I20260809 15:44:49.432776 21856 tablet_bootstrap.cc:492] T 4fe458f602674472bedadfdd3c7ee1e6 P dee2d58fd60f4b39a553c007f76ac760: Bootstrap starting.
I20260809 15:44:49.437703 21856 tablet_bootstrap.cc:654] T 4fe458f602674472bedadfdd3c7ee1e6 P dee2d58fd60f4b39a553c007f76ac760: Neither blocks nor log segments found. Creating new log.
I20260809 15:44:49.455140 21856 tablet_bootstrap.cc:492] T 4fe458f602674472bedadfdd3c7ee1e6 P dee2d58fd60f4b39a553c007f76ac760: No bootstrap required, opened a new log
I20260809 15:44:49.455530 21856 ts_tablet_manager.cc:1397] T 4fe458f602674472bedadfdd3c7ee1e6 P dee2d58fd60f4b39a553c007f76ac760: Time spent bootstrapping tablet: real 0.023s user 0.019s sys 0.003s
I20260809 15:44:49.458133 21858 consensus_queue.cc:1035] T 4fe458f602674472bedadfdd3c7ee1e6 P faa9b2a805cd43eca626c68bc7e7168f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:44:49.458137 21856 raft_consensus.cc:348] T 4fe458f602674472bedadfdd3c7ee1e6 P dee2d58fd60f4b39a553c007f76ac760 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } }
I20260809 15:44:49.459690 21856 raft_consensus.cc:374] T 4fe458f602674472bedadfdd3c7ee1e6 P dee2d58fd60f4b39a553c007f76ac760 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:44:49.459986 21856 raft_consensus.cc:729] T 4fe458f602674472bedadfdd3c7ee1e6 P dee2d58fd60f4b39a553c007f76ac760 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dee2d58fd60f4b39a553c007f76ac760, State: Initialized, Role: FOLLOWER
I20260809 15:44:49.460305 21769 raft_consensus.cc:3037] T d18bbcc7ebd14b479ddacb4aad814d47 P dee2d58fd60f4b39a553c007f76ac760 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:44:49.460675 21856 consensus_queue.cc:260] T 4fe458f602674472bedadfdd3c7ee1e6 P dee2d58fd60f4b39a553c007f76ac760 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } }
I20260809 15:44:49.462663 21856 ts_tablet_manager.cc:1428] T 4fe458f602674472bedadfdd3c7ee1e6 P dee2d58fd60f4b39a553c007f76ac760: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260809 15:44:49.467728 21858 consensus_queue.cc:1035] T d18bbcc7ebd14b479ddacb4aad814d47 P faa9b2a805cd43eca626c68bc7e7168f [LEADER]: Connected to new peer: Peer: permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:44:49.516688 21919 consensus_queue.cc:1035] T d18bbcc7ebd14b479ddacb4aad814d47 P faa9b2a805cd43eca626c68bc7e7168f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:44:49.590801 21902 consensus_queue.cc:1035] T ae87ef2626584e6bacd9f1af2c8991d8 P faa9b2a805cd43eca626c68bc7e7168f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:44:49.603466 21923 raft_consensus.cc:482] T 8d9c4e17e76743439089a2a53c45d01d P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:44:49.603961 21923 raft_consensus.cc:504] T 8d9c4e17e76743439089a2a53c45d01d P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } }
I20260809 15:44:49.605832 21923 leader_election.cc:288] T 8d9c4e17e76743439089a2a53c45d01d P 6517ff96fe0a42a2bde064dc7f1ecc97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers faa9b2a805cd43eca626c68bc7e7168f (127.20.202.65:37685), dee2d58fd60f4b39a553c007f76ac760 (127.20.202.67:36447)
I20260809 15:44:49.607980 21510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d9c4e17e76743439089a2a53c45d01d" candidate_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "faa9b2a805cd43eca626c68bc7e7168f" is_pre_election: true
I20260809 15:44:49.608755 21510 raft_consensus.cc:2379] T 8d9c4e17e76743439089a2a53c45d01d P faa9b2a805cd43eca626c68bc7e7168f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6517ff96fe0a42a2bde064dc7f1ecc97 in current term 1: Already voted for candidate dee2d58fd60f4b39a553c007f76ac760 in this term.
I20260809 15:44:49.603359 21895 consensus_queue.cc:1035] T ae87ef2626584e6bacd9f1af2c8991d8 P faa9b2a805cd43eca626c68bc7e7168f [LEADER]: Connected to new peer: Peer: permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:44:49.615061 21769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d9c4e17e76743439089a2a53c45d01d" candidate_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dee2d58fd60f4b39a553c007f76ac760" is_pre_election: true
I20260809 15:44:49.616529 21577 leader_election.cc:302] T 8d9c4e17e76743439089a2a53c45d01d P 6517ff96fe0a42a2bde064dc7f1ecc97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6517ff96fe0a42a2bde064dc7f1ecc97; no voters: dee2d58fd60f4b39a553c007f76ac760, faa9b2a805cd43eca626c68bc7e7168f
I20260809 15:44:49.617635 21923 raft_consensus.cc:3037] T 8d9c4e17e76743439089a2a53c45d01d P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:44:49.626579 21923 raft_consensus.cc:2726] T 8d9c4e17e76743439089a2a53c45d01d P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 15:44:49.633186 21510 raft_consensus.cc:3037] T 2f46e8d793294ce384acbf2f5584ecd2 P faa9b2a805cd43eca626c68bc7e7168f [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:44:49.638183 21915 consensus_queue.cc:1035] T 2f46e8d793294ce384acbf2f5584ecd2 P dee2d58fd60f4b39a553c007f76ac760 [LEADER]: Connected to new peer: Peer: permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:44:49.650504 21866 raft_consensus.cc:482] T 4fe458f602674472bedadfdd3c7ee1e6 P dee2d58fd60f4b39a553c007f76ac760 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:44:49.650931 21866 raft_consensus.cc:504] T 4fe458f602674472bedadfdd3c7ee1e6 P dee2d58fd60f4b39a553c007f76ac760 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } }
I20260809 15:44:49.652593 21866 leader_election.cc:288] T 4fe458f602674472bedadfdd3c7ee1e6 P dee2d58fd60f4b39a553c007f76ac760 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers faa9b2a805cd43eca626c68bc7e7168f (127.20.202.65:37685), 6517ff96fe0a42a2bde064dc7f1ecc97 (127.20.202.66:42701)
I20260809 15:44:49.655297 21895 consensus_queue.cc:1035] T 6dd9896939284d9cb2c03e9c21a92dfa P faa9b2a805cd43eca626c68bc7e7168f [LEADER]: Connected to new peer: Peer: permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:44:49.655798 21634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe458f602674472bedadfdd3c7ee1e6" candidate_uuid: "dee2d58fd60f4b39a553c007f76ac760" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" is_pre_election: true
I20260809 15:44:49.656978 21915 consensus_queue.cc:1035] T 2f46e8d793294ce384acbf2f5584ecd2 P dee2d58fd60f4b39a553c007f76ac760 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:44:49.666553 21915 raft_consensus.cc:482] T 724982e80d3b455a91f8a564d374a65e P dee2d58fd60f4b39a553c007f76ac760 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:44:49.666622 21895 consensus_queue.cc:1035] T ab976a6064754a689b7c88c5fa968b86 P faa9b2a805cd43eca626c68bc7e7168f [LEADER]: Connected to new peer: Peer: permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:44:49.667021 21915 raft_consensus.cc:504] T 724982e80d3b455a91f8a564d374a65e P dee2d58fd60f4b39a553c007f76ac760 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } }
I20260809 15:44:49.669971 21511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "724982e80d3b455a91f8a564d374a65e" candidate_uuid: "dee2d58fd60f4b39a553c007f76ac760" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "faa9b2a805cd43eca626c68bc7e7168f" is_pre_election: true
I20260809 15:44:49.670717 21511 raft_consensus.cc:2379] T 724982e80d3b455a91f8a564d374a65e P faa9b2a805cd43eca626c68bc7e7168f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dee2d58fd60f4b39a553c007f76ac760 in current term 1: Already voted for candidate 6517ff96fe0a42a2bde064dc7f1ecc97 in this term.
I20260809 15:44:49.672998 21915 leader_election.cc:288] T 724982e80d3b455a91f8a564d374a65e P dee2d58fd60f4b39a553c007f76ac760 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers faa9b2a805cd43eca626c68bc7e7168f (127.20.202.65:37685), 6517ff96fe0a42a2bde064dc7f1ecc97 (127.20.202.66:42701)
I20260809 15:44:49.673094 21635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "724982e80d3b455a91f8a564d374a65e" candidate_uuid: "dee2d58fd60f4b39a553c007f76ac760" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" is_pre_election: true
I20260809 15:44:49.673995 21895 consensus_queue.cc:1035] T ab976a6064754a689b7c88c5fa968b86 P faa9b2a805cd43eca626c68bc7e7168f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:44:49.674409 21710 leader_election.cc:302] T 724982e80d3b455a91f8a564d374a65e P dee2d58fd60f4b39a553c007f76ac760 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dee2d58fd60f4b39a553c007f76ac760; no voters: 6517ff96fe0a42a2bde064dc7f1ecc97, faa9b2a805cd43eca626c68bc7e7168f
I20260809 15:44:49.675391 21915 raft_consensus.cc:3037] T 724982e80d3b455a91f8a564d374a65e P dee2d58fd60f4b39a553c007f76ac760 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:44:49.680992 21510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe458f602674472bedadfdd3c7ee1e6" candidate_uuid: "dee2d58fd60f4b39a553c007f76ac760" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "faa9b2a805cd43eca626c68bc7e7168f" is_pre_election: true
I20260809 15:44:49.682379 21709 leader_election.cc:302] T 4fe458f602674472bedadfdd3c7ee1e6 P dee2d58fd60f4b39a553c007f76ac760 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dee2d58fd60f4b39a553c007f76ac760; no voters: 6517ff96fe0a42a2bde064dc7f1ecc97, faa9b2a805cd43eca626c68bc7e7168f
I20260809 15:44:49.683110 21866 raft_consensus.cc:2726] T 4fe458f602674472bedadfdd3c7ee1e6 P dee2d58fd60f4b39a553c007f76ac760 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 15:44:49.690412 21915 raft_consensus.cc:2726] T 724982e80d3b455a91f8a564d374a65e P dee2d58fd60f4b39a553c007f76ac760 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 15:44:49.702657 21902 consensus_queue.cc:1035] T 6dd9896939284d9cb2c03e9c21a92dfa P faa9b2a805cd43eca626c68bc7e7168f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:44:49.765654 21769 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8d9c4e17e76743439089a2a53c45d01d"
dest_uuid: "dee2d58fd60f4b39a553c007f76ac760"
mode: GRACEFUL
new_leader_uuid: "faa9b2a805cd43eca626c68bc7e7168f"
from {username='slave'} at 127.0.0.1:56218
I20260809 15:44:49.766172 21769 raft_consensus.cc:595] T 8d9c4e17e76743439089a2a53c45d01d P dee2d58fd60f4b39a553c007f76ac760 [term 1 LEADER]: Received request to transfer leadership to TS faa9b2a805cd43eca626c68bc7e7168f
I20260809 15:44:49.778469 21923 consensus_queue.cc:1035] T 724982e80d3b455a91f8a564d374a65e P 6517ff96fe0a42a2bde064dc7f1ecc97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:44:49.790114 21923 consensus_queue.cc:1035] T 724982e80d3b455a91f8a564d374a65e P 6517ff96fe0a42a2bde064dc7f1ecc97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:44:49.795871 21511 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ae87ef2626584e6bacd9f1af2c8991d8"
dest_uuid: "faa9b2a805cd43eca626c68bc7e7168f"
mode: GRACEFUL
new_leader_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97"
from {username='slave'} at 127.0.0.1:59382
I20260809 15:44:49.796437 21511 raft_consensus.cc:595] T ae87ef2626584e6bacd9f1af2c8991d8 P faa9b2a805cd43eca626c68bc7e7168f [term 1 LEADER]: Received request to transfer leadership to TS 6517ff96fe0a42a2bde064dc7f1ecc97
I20260809 15:44:49.805298 21511 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ab976a6064754a689b7c88c5fa968b86"
dest_uuid: "faa9b2a805cd43eca626c68bc7e7168f"
mode: GRACEFUL
new_leader_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97"
from {username='slave'} at 127.0.0.1:59382
I20260809 15:44:49.805856 21511 raft_consensus.cc:595] T ab976a6064754a689b7c88c5fa968b86 P faa9b2a805cd43eca626c68bc7e7168f [term 1 LEADER]: Received request to transfer leadership to TS 6517ff96fe0a42a2bde064dc7f1ecc97
I20260809 15:44:49.811182 21769 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2f46e8d793294ce384acbf2f5584ecd2"
dest_uuid: "dee2d58fd60f4b39a553c007f76ac760"
mode: GRACEFUL
new_leader_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97"
from {username='slave'} at 127.0.0.1:56218
I20260809 15:44:49.811769 21769 raft_consensus.cc:595] T 2f46e8d793294ce384acbf2f5584ecd2 P dee2d58fd60f4b39a553c007f76ac760 [term 1 LEADER]: Received request to transfer leadership to TS 6517ff96fe0a42a2bde064dc7f1ecc97
I20260809 15:44:49.817025 21510 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6dd9896939284d9cb2c03e9c21a92dfa"
dest_uuid: "faa9b2a805cd43eca626c68bc7e7168f"
mode: GRACEFUL
new_leader_uuid: "dee2d58fd60f4b39a553c007f76ac760"
from {username='slave'} at 127.0.0.1:59382
I20260809 15:44:49.817636 21510 raft_consensus.cc:595] T 6dd9896939284d9cb2c03e9c21a92dfa P faa9b2a805cd43eca626c68bc7e7168f [term 1 LEADER]: Received request to transfer leadership to TS dee2d58fd60f4b39a553c007f76ac760
I20260809 15:44:49.822806 21902 consensus_queue.cc:1035] T d203a1017c6b44d0a62bbc654ee21d8c P faa9b2a805cd43eca626c68bc7e7168f [LEADER]: Connected to new peer: Peer: permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:44:49.835363 21858 consensus_queue.cc:1035] T d203a1017c6b44d0a62bbc654ee21d8c P faa9b2a805cd43eca626c68bc7e7168f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:44:49.836874 21915 consensus_queue.cc:1035] T 8d9c4e17e76743439089a2a53c45d01d P dee2d58fd60f4b39a553c007f76ac760 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:44:49.848485 21769 raft_consensus.cc:3037] T 4fe458f602674472bedadfdd3c7ee1e6 P dee2d58fd60f4b39a553c007f76ac760 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:44:49.864486 21866 consensus_queue.cc:1035] T 8d9c4e17e76743439089a2a53c45d01d P dee2d58fd60f4b39a553c007f76ac760 [LEADER]: Connected to new peer: Peer: permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:44:49.894614 21915 raft_consensus.cc:979] T 8d9c4e17e76743439089a2a53c45d01d P dee2d58fd60f4b39a553c007f76ac760: : Instructing follower faa9b2a805cd43eca626c68bc7e7168f to start an election
I20260809 15:44:49.895073 22020 raft_consensus.cc:1067] T 8d9c4e17e76743439089a2a53c45d01d P dee2d58fd60f4b39a553c007f76ac760 [term 1 LEADER]: Signalling peer faa9b2a805cd43eca626c68bc7e7168f to start an election
I20260809 15:44:49.896730 21510 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8d9c4e17e76743439089a2a53c45d01d"
dest_uuid: "faa9b2a805cd43eca626c68bc7e7168f"
from {username='slave'} at 127.0.0.1:50142
I20260809 15:44:49.897274 21510 raft_consensus.cc:482] T 8d9c4e17e76743439089a2a53c45d01d P faa9b2a805cd43eca626c68bc7e7168f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 15:44:49.897634 21510 raft_consensus.cc:3037] T 8d9c4e17e76743439089a2a53c45d01d P faa9b2a805cd43eca626c68bc7e7168f [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:44:49.903836 21510 raft_consensus.cc:504] T 8d9c4e17e76743439089a2a53c45d01d P faa9b2a805cd43eca626c68bc7e7168f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } }
I20260809 15:44:49.905967 21510 leader_election.cc:288] T 8d9c4e17e76743439089a2a53c45d01d P faa9b2a805cd43eca626c68bc7e7168f [CANDIDATE]: Term 2 election: Requested vote from peers 6517ff96fe0a42a2bde064dc7f1ecc97 (127.20.202.66:42701), dee2d58fd60f4b39a553c007f76ac760 (127.20.202.67:36447)
I20260809 15:44:49.906180 21635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d9c4e17e76743439089a2a53c45d01d" candidate_uuid: "faa9b2a805cd43eca626c68bc7e7168f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97"
I20260809 15:44:49.906540 21769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d9c4e17e76743439089a2a53c45d01d" candidate_uuid: "faa9b2a805cd43eca626c68bc7e7168f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "dee2d58fd60f4b39a553c007f76ac760"
I20260809 15:44:49.906730 21635 raft_consensus.cc:3037] T 8d9c4e17e76743439089a2a53c45d01d P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:44:49.907078 21769 raft_consensus.cc:3032] T 8d9c4e17e76743439089a2a53c45d01d P dee2d58fd60f4b39a553c007f76ac760 [term 1 LEADER]: Stepping down as leader of term 1
I20260809 15:44:49.907408 21769 raft_consensus.cc:729] T 8d9c4e17e76743439089a2a53c45d01d P dee2d58fd60f4b39a553c007f76ac760 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: dee2d58fd60f4b39a553c007f76ac760, State: Running, Role: LEADER
I20260809 15:44:49.908129 21769 consensus_queue.cc:260] T 8d9c4e17e76743439089a2a53c45d01d P dee2d58fd60f4b39a553c007f76ac760 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } }
I20260809 15:44:49.909420 21769 raft_consensus.cc:3037] T 8d9c4e17e76743439089a2a53c45d01d P dee2d58fd60f4b39a553c007f76ac760 [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:44:49.912434 21635 raft_consensus.cc:2454] T 8d9c4e17e76743439089a2a53c45d01d P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate faa9b2a805cd43eca626c68bc7e7168f in term 2.
I20260809 15:44:49.913437 21460 leader_election.cc:302] T 8d9c4e17e76743439089a2a53c45d01d P faa9b2a805cd43eca626c68bc7e7168f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6517ff96fe0a42a2bde064dc7f1ecc97, faa9b2a805cd43eca626c68bc7e7168f; no voters:
I20260809 15:44:49.914146 21858 raft_consensus.cc:2781] T 8d9c4e17e76743439089a2a53c45d01d P faa9b2a805cd43eca626c68bc7e7168f [term 2 FOLLOWER]: Leader election won for term 2
I20260809 15:44:49.914628 21858 raft_consensus.cc:686] T 8d9c4e17e76743439089a2a53c45d01d P faa9b2a805cd43eca626c68bc7e7168f [term 2 LEADER]: Becoming Leader. State: Replica: faa9b2a805cd43eca626c68bc7e7168f, State: Running, Role: LEADER
I20260809 15:44:49.914826 21769 raft_consensus.cc:2454] T 8d9c4e17e76743439089a2a53c45d01d P dee2d58fd60f4b39a553c007f76ac760 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate faa9b2a805cd43eca626c68bc7e7168f in term 2.
I20260809 15:44:49.915475 21858 consensus_queue.cc:237] T 8d9c4e17e76743439089a2a53c45d01d P faa9b2a805cd43eca626c68bc7e7168f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } }
I20260809 15:44:49.921809 21333 catalog_manager.cc:5495] T 8d9c4e17e76743439089a2a53c45d01d P faa9b2a805cd43eca626c68bc7e7168f reported cstate change: term changed from 1 to 2, leader changed from dee2d58fd60f4b39a553c007f76ac760 (127.20.202.67) to faa9b2a805cd43eca626c68bc7e7168f (127.20.202.65). New cstate: current_term: 2 leader_uuid: "faa9b2a805cd43eca626c68bc7e7168f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } health_report { overall_health: UNKNOWN } } }
I20260809 15:44:50.177028 22017 raft_consensus.cc:979] T 2f46e8d793294ce384acbf2f5584ecd2 P dee2d58fd60f4b39a553c007f76ac760: : Instructing follower 6517ff96fe0a42a2bde064dc7f1ecc97 to start an election
I20260809 15:44:50.177510 22017 raft_consensus.cc:1067] T 2f46e8d793294ce384acbf2f5584ecd2 P dee2d58fd60f4b39a553c007f76ac760 [term 1 LEADER]: Signalling peer 6517ff96fe0a42a2bde064dc7f1ecc97 to start an election
I20260809 15:44:50.182485 21895 raft_consensus.cc:979] T 6dd9896939284d9cb2c03e9c21a92dfa P faa9b2a805cd43eca626c68bc7e7168f: : Instructing follower dee2d58fd60f4b39a553c007f76ac760 to start an election
I20260809 15:44:50.182726 21635 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2f46e8d793294ce384acbf2f5584ecd2"
dest_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97"
from {username='slave'} at 127.0.0.1:39640
I20260809 15:44:50.182927 21895 raft_consensus.cc:1067] T 6dd9896939284d9cb2c03e9c21a92dfa P faa9b2a805cd43eca626c68bc7e7168f [term 1 LEADER]: Signalling peer dee2d58fd60f4b39a553c007f76ac760 to start an election
I20260809 15:44:50.183332 21635 raft_consensus.cc:482] T 2f46e8d793294ce384acbf2f5584ecd2 P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 15:44:50.183781 21635 raft_consensus.cc:3037] T 2f46e8d793294ce384acbf2f5584ecd2 P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:44:50.185948 21769 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6dd9896939284d9cb2c03e9c21a92dfa"
dest_uuid: "dee2d58fd60f4b39a553c007f76ac760"
from {username='slave'} at 127.0.0.1:34924
I20260809 15:44:50.186468 21769 raft_consensus.cc:482] T 6dd9896939284d9cb2c03e9c21a92dfa P dee2d58fd60f4b39a553c007f76ac760 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 15:44:50.186828 21769 raft_consensus.cc:3037] T 6dd9896939284d9cb2c03e9c21a92dfa P dee2d58fd60f4b39a553c007f76ac760 [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:44:50.190853 21769 raft_consensus.cc:504] T 6dd9896939284d9cb2c03e9c21a92dfa P dee2d58fd60f4b39a553c007f76ac760 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } }
I20260809 15:44:50.190819 21635 raft_consensus.cc:504] T 2f46e8d793294ce384acbf2f5584ecd2 P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } }
I20260809 15:44:50.192577 21769 leader_election.cc:288] T 6dd9896939284d9cb2c03e9c21a92dfa P dee2d58fd60f4b39a553c007f76ac760 [CANDIDATE]: Term 2 election: Requested vote from peers faa9b2a805cd43eca626c68bc7e7168f (127.20.202.65:37685), 6517ff96fe0a42a2bde064dc7f1ecc97 (127.20.202.66:42701)
I20260809 15:44:50.193527 21635 leader_election.cc:288] T 2f46e8d793294ce384acbf2f5584ecd2 P 6517ff96fe0a42a2bde064dc7f1ecc97 [CANDIDATE]: Term 2 election: Requested vote from peers faa9b2a805cd43eca626c68bc7e7168f (127.20.202.65:37685), dee2d58fd60f4b39a553c007f76ac760 (127.20.202.67:36447)
I20260809 15:44:50.193753 21510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dd9896939284d9cb2c03e9c21a92dfa" candidate_uuid: "dee2d58fd60f4b39a553c007f76ac760" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "faa9b2a805cd43eca626c68bc7e7168f"
I20260809 15:44:50.194335 21510 raft_consensus.cc:3032] T 6dd9896939284d9cb2c03e9c21a92dfa P faa9b2a805cd43eca626c68bc7e7168f [term 1 LEADER]: Stepping down as leader of term 1
I20260809 15:44:50.193673 21511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f46e8d793294ce384acbf2f5584ecd2" candidate_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "faa9b2a805cd43eca626c68bc7e7168f"
I20260809 15:44:50.194656 21510 raft_consensus.cc:729] T 6dd9896939284d9cb2c03e9c21a92dfa P faa9b2a805cd43eca626c68bc7e7168f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: faa9b2a805cd43eca626c68bc7e7168f, State: Running, Role: LEADER
I20260809 15:44:50.195072 21511 raft_consensus.cc:3037] T 2f46e8d793294ce384acbf2f5584ecd2 P faa9b2a805cd43eca626c68bc7e7168f [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:44:50.195706 21635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dd9896939284d9cb2c03e9c21a92dfa" candidate_uuid: "dee2d58fd60f4b39a553c007f76ac760" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97"
I20260809 15:44:50.195432 21510 consensus_queue.cc:260] T 6dd9896939284d9cb2c03e9c21a92dfa P faa9b2a805cd43eca626c68bc7e7168f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } }
I20260809 15:44:50.196321 21635 raft_consensus.cc:3037] T 6dd9896939284d9cb2c03e9c21a92dfa P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:44:50.196764 21510 raft_consensus.cc:3037] T 6dd9896939284d9cb2c03e9c21a92dfa P faa9b2a805cd43eca626c68bc7e7168f [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:44:50.201426 21511 raft_consensus.cc:2454] T 2f46e8d793294ce384acbf2f5584ecd2 P faa9b2a805cd43eca626c68bc7e7168f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6517ff96fe0a42a2bde064dc7f1ecc97 in term 2.
I20260809 15:44:50.202229 21635 raft_consensus.cc:2454] T 6dd9896939284d9cb2c03e9c21a92dfa P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dee2d58fd60f4b39a553c007f76ac760 in term 2.
I20260809 15:44:50.202589 21574 leader_election.cc:302] T 2f46e8d793294ce384acbf2f5584ecd2 P 6517ff96fe0a42a2bde064dc7f1ecc97 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6517ff96fe0a42a2bde064dc7f1ecc97, faa9b2a805cd43eca626c68bc7e7168f; no voters:
I20260809 15:44:50.203625 21710 leader_election.cc:302] T 6dd9896939284d9cb2c03e9c21a92dfa P dee2d58fd60f4b39a553c007f76ac760 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6517ff96fe0a42a2bde064dc7f1ecc97, dee2d58fd60f4b39a553c007f76ac760; no voters:
I20260809 15:44:50.204349 22020 raft_consensus.cc:2781] T 6dd9896939284d9cb2c03e9c21a92dfa P dee2d58fd60f4b39a553c007f76ac760 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 15:44:50.204602 21513 raft_consensus.cc:1226] T 2f46e8d793294ce384acbf2f5584ecd2 P faa9b2a805cd43eca626c68bc7e7168f [term 2 FOLLOWER]: Rejecting Update request from peer dee2d58fd60f4b39a553c007f76ac760 for earlier term 1. Current term is 2. Ops: []
I20260809 15:44:50.203972 21510 raft_consensus.cc:2454] T 6dd9896939284d9cb2c03e9c21a92dfa P faa9b2a805cd43eca626c68bc7e7168f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dee2d58fd60f4b39a553c007f76ac760 in term 2.
I20260809 15:44:50.204789 21769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f46e8d793294ce384acbf2f5584ecd2" candidate_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "dee2d58fd60f4b39a553c007f76ac760"
I20260809 15:44:50.203624 21923 raft_consensus.cc:2781] T 2f46e8d793294ce384acbf2f5584ecd2 P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 15:44:50.205335 21769 raft_consensus.cc:3032] T 2f46e8d793294ce384acbf2f5584ecd2 P dee2d58fd60f4b39a553c007f76ac760 [term 1 LEADER]: Stepping down as leader of term 1
I20260809 15:44:50.205451 21923 raft_consensus.cc:686] T 2f46e8d793294ce384acbf2f5584ecd2 P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 2 LEADER]: Becoming Leader. State: Replica: 6517ff96fe0a42a2bde064dc7f1ecc97, State: Running, Role: LEADER
I20260809 15:44:50.205663 21769 raft_consensus.cc:729] T 2f46e8d793294ce384acbf2f5584ecd2 P dee2d58fd60f4b39a553c007f76ac760 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: dee2d58fd60f4b39a553c007f76ac760, State: Running, Role: LEADER
I20260809 15:44:50.206291 21923 consensus_queue.cc:237] T 2f46e8d793294ce384acbf2f5584ecd2 P 6517ff96fe0a42a2bde064dc7f1ecc97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } }
I20260809 15:44:50.204813 22020 raft_consensus.cc:686] T 6dd9896939284d9cb2c03e9c21a92dfa P dee2d58fd60f4b39a553c007f76ac760 [term 2 LEADER]: Becoming Leader. State: Replica: dee2d58fd60f4b39a553c007f76ac760, State: Running, Role: LEADER
I20260809 15:44:50.206945 21769 consensus_queue.cc:260] T 2f46e8d793294ce384acbf2f5584ecd2 P dee2d58fd60f4b39a553c007f76ac760 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } }
I20260809 15:44:50.208384 21769 raft_consensus.cc:3037] T 2f46e8d793294ce384acbf2f5584ecd2 P dee2d58fd60f4b39a553c007f76ac760 [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:44:50.207885 22020 consensus_queue.cc:237] T 6dd9896939284d9cb2c03e9c21a92dfa P dee2d58fd60f4b39a553c007f76ac760 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } }
I20260809 15:44:50.217347 21769 raft_consensus.cc:2454] T 2f46e8d793294ce384acbf2f5584ecd2 P dee2d58fd60f4b39a553c007f76ac760 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6517ff96fe0a42a2bde064dc7f1ecc97 in term 2.
I20260809 15:44:50.217114 21339 catalog_manager.cc:5495] T 2f46e8d793294ce384acbf2f5584ecd2 P 6517ff96fe0a42a2bde064dc7f1ecc97 reported cstate change: term changed from 1 to 2, leader changed from dee2d58fd60f4b39a553c007f76ac760 (127.20.202.67) to 6517ff96fe0a42a2bde064dc7f1ecc97 (127.20.202.66). New cstate: current_term: 2 leader_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } health_report { overall_health: UNKNOWN } } }
I20260809 15:44:50.219727 21902 raft_consensus.cc:979] T ab976a6064754a689b7c88c5fa968b86 P faa9b2a805cd43eca626c68bc7e7168f: : Instructing follower 6517ff96fe0a42a2bde064dc7f1ecc97 to start an election
I20260809 15:44:50.220108 21902 raft_consensus.cc:979] T ae87ef2626584e6bacd9f1af2c8991d8 P faa9b2a805cd43eca626c68bc7e7168f: : Instructing follower 6517ff96fe0a42a2bde064dc7f1ecc97 to start an election
I20260809 15:44:50.220288 21858 raft_consensus.cc:1067] T ab976a6064754a689b7c88c5fa968b86 P faa9b2a805cd43eca626c68bc7e7168f [term 1 LEADER]: Signalling peer 6517ff96fe0a42a2bde064dc7f1ecc97 to start an election
I20260809 15:44:50.220439 21902 raft_consensus.cc:1067] T ae87ef2626584e6bacd9f1af2c8991d8 P faa9b2a805cd43eca626c68bc7e7168f [term 1 LEADER]: Signalling peer 6517ff96fe0a42a2bde064dc7f1ecc97 to start an election
I20260809 15:44:50.221766 21333 catalog_manager.cc:5495] T 6dd9896939284d9cb2c03e9c21a92dfa P dee2d58fd60f4b39a553c007f76ac760 reported cstate change: term changed from 1 to 2, leader changed from faa9b2a805cd43eca626c68bc7e7168f (127.20.202.65) to dee2d58fd60f4b39a553c007f76ac760 (127.20.202.67). New cstate: current_term: 2 leader_uuid: "dee2d58fd60f4b39a553c007f76ac760" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } health_report { overall_health: HEALTHY } } }
I20260809 15:44:50.222443 21633 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ae87ef2626584e6bacd9f1af2c8991d8"
dest_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97"
from {username='slave'} at 127.0.0.1:39642
I20260809 15:44:50.222976 21633 raft_consensus.cc:482] T ae87ef2626584e6bacd9f1af2c8991d8 P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 15:44:50.223455 21633 raft_consensus.cc:3037] T ae87ef2626584e6bacd9f1af2c8991d8 P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:44:50.224422 21635 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ab976a6064754a689b7c88c5fa968b86"
dest_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97"
from {username='slave'} at 127.0.0.1:39642
I20260809 15:44:50.224969 21635 raft_consensus.cc:482] T ab976a6064754a689b7c88c5fa968b86 P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 15:44:50.225404 21635 raft_consensus.cc:3037] T ab976a6064754a689b7c88c5fa968b86 P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:44:50.231185 21635 raft_consensus.cc:504] T ab976a6064754a689b7c88c5fa968b86 P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } }
I20260809 15:44:50.233119 21635 leader_election.cc:288] T ab976a6064754a689b7c88c5fa968b86 P 6517ff96fe0a42a2bde064dc7f1ecc97 [CANDIDATE]: Term 2 election: Requested vote from peers faa9b2a805cd43eca626c68bc7e7168f (127.20.202.65:37685), dee2d58fd60f4b39a553c007f76ac760 (127.20.202.67:36447)
I20260809 15:44:50.233826 21513 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab976a6064754a689b7c88c5fa968b86" candidate_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "faa9b2a805cd43eca626c68bc7e7168f"
I20260809 15:44:50.234247 21769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab976a6064754a689b7c88c5fa968b86" candidate_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "dee2d58fd60f4b39a553c007f76ac760"
I20260809 15:44:50.234397 21513 raft_consensus.cc:3032] T ab976a6064754a689b7c88c5fa968b86 P faa9b2a805cd43eca626c68bc7e7168f [term 1 LEADER]: Stepping down as leader of term 1
I20260809 15:44:50.234755 21513 raft_consensus.cc:729] T ab976a6064754a689b7c88c5fa968b86 P faa9b2a805cd43eca626c68bc7e7168f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: faa9b2a805cd43eca626c68bc7e7168f, State: Running, Role: LEADER
I20260809 15:44:50.234798 21769 raft_consensus.cc:3037] T ab976a6064754a689b7c88c5fa968b86 P dee2d58fd60f4b39a553c007f76ac760 [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:44:50.235589 21513 consensus_queue.cc:260] T ab976a6064754a689b7c88c5fa968b86 P faa9b2a805cd43eca626c68bc7e7168f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } }
I20260809 15:44:50.236716 21513 raft_consensus.cc:3037] T ab976a6064754a689b7c88c5fa968b86 P faa9b2a805cd43eca626c68bc7e7168f [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:44:50.239482 21633 raft_consensus.cc:504] T ae87ef2626584e6bacd9f1af2c8991d8 P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } }
I20260809 15:44:50.240777 21769 raft_consensus.cc:2454] T ab976a6064754a689b7c88c5fa968b86 P dee2d58fd60f4b39a553c007f76ac760 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6517ff96fe0a42a2bde064dc7f1ecc97 in term 2.
I20260809 15:44:50.241690 21577 leader_election.cc:302] T ab976a6064754a689b7c88c5fa968b86 P 6517ff96fe0a42a2bde064dc7f1ecc97 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6517ff96fe0a42a2bde064dc7f1ecc97, dee2d58fd60f4b39a553c007f76ac760; no voters:
I20260809 15:44:50.241806 21633 leader_election.cc:288] T ae87ef2626584e6bacd9f1af2c8991d8 P 6517ff96fe0a42a2bde064dc7f1ecc97 [CANDIDATE]: Term 2 election: Requested vote from peers faa9b2a805cd43eca626c68bc7e7168f (127.20.202.65:37685), dee2d58fd60f4b39a553c007f76ac760 (127.20.202.67:36447)
I20260809 15:44:50.242431 21923 raft_consensus.cc:2781] T ab976a6064754a689b7c88c5fa968b86 P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 15:44:50.242926 21923 raft_consensus.cc:686] T ab976a6064754a689b7c88c5fa968b86 P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 2 LEADER]: Becoming Leader. State: Replica: 6517ff96fe0a42a2bde064dc7f1ecc97, State: Running, Role: LEADER
I20260809 15:44:50.243227 21769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae87ef2626584e6bacd9f1af2c8991d8" candidate_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "dee2d58fd60f4b39a553c007f76ac760"
I20260809 15:44:50.243786 21769 raft_consensus.cc:3037] T ae87ef2626584e6bacd9f1af2c8991d8 P dee2d58fd60f4b39a553c007f76ac760 [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:44:50.243891 21923 consensus_queue.cc:237] T ab976a6064754a689b7c88c5fa968b86 P 6517ff96fe0a42a2bde064dc7f1ecc97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } }
I20260809 15:44:50.243927 21510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae87ef2626584e6bacd9f1af2c8991d8" candidate_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "faa9b2a805cd43eca626c68bc7e7168f"
I20260809 15:44:50.245306 21510 raft_consensus.cc:3032] T ae87ef2626584e6bacd9f1af2c8991d8 P faa9b2a805cd43eca626c68bc7e7168f [term 1 LEADER]: Stepping down as leader of term 1
I20260809 15:44:50.245620 21510 raft_consensus.cc:729] T ae87ef2626584e6bacd9f1af2c8991d8 P faa9b2a805cd43eca626c68bc7e7168f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: faa9b2a805cd43eca626c68bc7e7168f, State: Running, Role: LEADER
I20260809 15:44:50.246340 21510 consensus_queue.cc:260] T ae87ef2626584e6bacd9f1af2c8991d8 P faa9b2a805cd43eca626c68bc7e7168f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } }
I20260809 15:44:50.247491 21510 raft_consensus.cc:3037] T ae87ef2626584e6bacd9f1af2c8991d8 P faa9b2a805cd43eca626c68bc7e7168f [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:44:50.249421 21769 raft_consensus.cc:2454] T ae87ef2626584e6bacd9f1af2c8991d8 P dee2d58fd60f4b39a553c007f76ac760 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6517ff96fe0a42a2bde064dc7f1ecc97 in term 2.
I20260809 15:44:50.250459 21577 leader_election.cc:302] T ae87ef2626584e6bacd9f1af2c8991d8 P 6517ff96fe0a42a2bde064dc7f1ecc97 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6517ff96fe0a42a2bde064dc7f1ecc97, dee2d58fd60f4b39a553c007f76ac760; no voters:
I20260809 15:44:50.251228 21923 raft_consensus.cc:2781] T ae87ef2626584e6bacd9f1af2c8991d8 P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 15:44:50.251778 21923 raft_consensus.cc:686] T ae87ef2626584e6bacd9f1af2c8991d8 P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 2 LEADER]: Becoming Leader. State: Replica: 6517ff96fe0a42a2bde064dc7f1ecc97, State: Running, Role: LEADER
I20260809 15:44:50.253134 21923 consensus_queue.cc:237] T ae87ef2626584e6bacd9f1af2c8991d8 P 6517ff96fe0a42a2bde064dc7f1ecc97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } }
I20260809 15:44:50.244794 21513 raft_consensus.cc:2454] T ab976a6064754a689b7c88c5fa968b86 P faa9b2a805cd43eca626c68bc7e7168f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6517ff96fe0a42a2bde064dc7f1ecc97 in term 2.
I20260809 15:44:50.256915 21510 raft_consensus.cc:2454] T ae87ef2626584e6bacd9f1af2c8991d8 P faa9b2a805cd43eca626c68bc7e7168f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6517ff96fe0a42a2bde064dc7f1ecc97 in term 2.
I20260809 15:44:50.260782 21333 catalog_manager.cc:5495] T ab976a6064754a689b7c88c5fa968b86 P 6517ff96fe0a42a2bde064dc7f1ecc97 reported cstate change: term changed from 1 to 2, leader changed from faa9b2a805cd43eca626c68bc7e7168f (127.20.202.65) to 6517ff96fe0a42a2bde064dc7f1ecc97 (127.20.202.66). New cstate: current_term: 2 leader_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } health_report { overall_health: UNKNOWN } } }
I20260809 15:44:50.271150 21333 catalog_manager.cc:5495] T ae87ef2626584e6bacd9f1af2c8991d8 P 6517ff96fe0a42a2bde064dc7f1ecc97 reported cstate change: term changed from 1 to 2, leader changed from faa9b2a805cd43eca626c68bc7e7168f (127.20.202.65) to 6517ff96fe0a42a2bde064dc7f1ecc97 (127.20.202.66). New cstate: current_term: 2 leader_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } health_report { overall_health: UNKNOWN } } }
I20260809 15:44:50.295539 21633 raft_consensus.cc:1261] T 8d9c4e17e76743439089a2a53c45d01d P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 2 FOLLOWER]: Refusing update from remote peer faa9b2a805cd43eca626c68bc7e7168f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 15:44:50.296561 21919 consensus_queue.cc:1035] T 8d9c4e17e76743439089a2a53c45d01d P faa9b2a805cd43eca626c68bc7e7168f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 15:44:50.303982 21769 raft_consensus.cc:1261] T 8d9c4e17e76743439089a2a53c45d01d P dee2d58fd60f4b39a553c007f76ac760 [term 2 FOLLOWER]: Refusing update from remote peer faa9b2a805cd43eca626c68bc7e7168f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 15:44:50.305670 21858 consensus_queue.cc:1035] T 8d9c4e17e76743439089a2a53c45d01d P faa9b2a805cd43eca626c68bc7e7168f [LEADER]: Connected to new peer: Peer: permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 15:44:50.597211 21769 raft_consensus.cc:1261] T 2f46e8d793294ce384acbf2f5584ecd2 P dee2d58fd60f4b39a553c007f76ac760 [term 2 FOLLOWER]: Refusing update from remote peer 6517ff96fe0a42a2bde064dc7f1ecc97: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 15:44:50.598305 21923 consensus_queue.cc:1035] T 2f46e8d793294ce384acbf2f5584ecd2 P 6517ff96fe0a42a2bde064dc7f1ecc97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 15:44:50.604786 21510 raft_consensus.cc:1261] T 2f46e8d793294ce384acbf2f5584ecd2 P faa9b2a805cd43eca626c68bc7e7168f [term 2 FOLLOWER]: Refusing update from remote peer 6517ff96fe0a42a2bde064dc7f1ecc97: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 15:44:50.606192 21923 consensus_queue.cc:1035] T 2f46e8d793294ce384acbf2f5584ecd2 P 6517ff96fe0a42a2bde064dc7f1ecc97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 15:44:50.673427 21769 raft_consensus.cc:1261] T ab976a6064754a689b7c88c5fa968b86 P dee2d58fd60f4b39a553c007f76ac760 [term 2 FOLLOWER]: Refusing update from remote peer 6517ff96fe0a42a2bde064dc7f1ecc97: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 15:44:50.674361 22021 consensus_queue.cc:1035] T ab976a6064754a689b7c88c5fa968b86 P 6517ff96fe0a42a2bde064dc7f1ecc97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 15:44:50.680977 21510 raft_consensus.cc:1261] T ab976a6064754a689b7c88c5fa968b86 P faa9b2a805cd43eca626c68bc7e7168f [term 2 FOLLOWER]: Refusing update from remote peer 6517ff96fe0a42a2bde064dc7f1ecc97: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 15:44:50.682315 21923 consensus_queue.cc:1035] T ab976a6064754a689b7c88c5fa968b86 P 6517ff96fe0a42a2bde064dc7f1ecc97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 15:44:50.738343 21510 raft_consensus.cc:1261] T 6dd9896939284d9cb2c03e9c21a92dfa P faa9b2a805cd43eca626c68bc7e7168f [term 2 FOLLOWER]: Refusing update from remote peer dee2d58fd60f4b39a553c007f76ac760: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 15:44:50.739251 22020 consensus_queue.cc:1035] T 6dd9896939284d9cb2c03e9c21a92dfa P dee2d58fd60f4b39a553c007f76ac760 [LEADER]: Connected to new peer: Peer: permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 15:44:50.746930 21633 raft_consensus.cc:1261] T 6dd9896939284d9cb2c03e9c21a92dfa P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 2 FOLLOWER]: Refusing update from remote peer dee2d58fd60f4b39a553c007f76ac760: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 15:44:50.748181 22020 consensus_queue.cc:1035] T 6dd9896939284d9cb2c03e9c21a92dfa P dee2d58fd60f4b39a553c007f76ac760 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 15:44:50.807576 21769 raft_consensus.cc:1261] T ae87ef2626584e6bacd9f1af2c8991d8 P dee2d58fd60f4b39a553c007f76ac760 [term 2 FOLLOWER]: Refusing update from remote peer 6517ff96fe0a42a2bde064dc7f1ecc97: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 15:44:50.808476 21923 consensus_queue.cc:1035] T ae87ef2626584e6bacd9f1af2c8991d8 P 6517ff96fe0a42a2bde064dc7f1ecc97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 15:44:50.815672 21510 raft_consensus.cc:1261] T ae87ef2626584e6bacd9f1af2c8991d8 P faa9b2a805cd43eca626c68bc7e7168f [term 2 FOLLOWER]: Refusing update from remote peer 6517ff96fe0a42a2bde064dc7f1ecc97: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 15:44:50.817106 21920 consensus_queue.cc:1035] T ae87ef2626584e6bacd9f1af2c8991d8 P 6517ff96fe0a42a2bde064dc7f1ecc97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 15:44:52.843035 21289 auto_leader_rebalancer-test.cc:269] The leader distribution is
I20260809 15:44:52.844038 21289 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 15:44:52.863157 21633 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ae87ef2626584e6bacd9f1af2c8991d8"
dest_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97"
mode: GRACEFUL
new_leader_uuid: "dee2d58fd60f4b39a553c007f76ac760"
from {username='slave'} at 127.0.0.1:40824
I20260809 15:44:52.863722 21633 raft_consensus.cc:595] T ae87ef2626584e6bacd9f1af2c8991d8 P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 2 LEADER]: Received request to transfer leadership to TS dee2d58fd60f4b39a553c007f76ac760
I20260809 15:44:52.873999 21510 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d203a1017c6b44d0a62bbc654ee21d8c"
dest_uuid: "faa9b2a805cd43eca626c68bc7e7168f"
mode: GRACEFUL
new_leader_uuid: "dee2d58fd60f4b39a553c007f76ac760"
from {username='slave'} at 127.0.0.1:59392
I20260809 15:44:52.874521 21510 raft_consensus.cc:595] T d203a1017c6b44d0a62bbc654ee21d8c P faa9b2a805cd43eca626c68bc7e7168f [term 1 LEADER]: Received request to transfer leadership to TS dee2d58fd60f4b39a553c007f76ac760
I20260809 15:44:52.875466 21289 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260809 15:44:52.875823 21289 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 15:44:53.094889 21923 raft_consensus.cc:979] T ae87ef2626584e6bacd9f1af2c8991d8 P 6517ff96fe0a42a2bde064dc7f1ecc97: : Instructing follower dee2d58fd60f4b39a553c007f76ac760 to start an election
I20260809 15:44:53.095324 22039 raft_consensus.cc:1067] T ae87ef2626584e6bacd9f1af2c8991d8 P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 2 LEADER]: Signalling peer dee2d58fd60f4b39a553c007f76ac760 to start an election
I20260809 15:44:53.096573 21769 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ae87ef2626584e6bacd9f1af2c8991d8"
dest_uuid: "dee2d58fd60f4b39a553c007f76ac760"
from {username='slave'} at 127.0.0.1:34936
I20260809 15:44:53.096988 21769 raft_consensus.cc:482] T ae87ef2626584e6bacd9f1af2c8991d8 P dee2d58fd60f4b39a553c007f76ac760 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 15:44:53.097252 21769 raft_consensus.cc:3037] T ae87ef2626584e6bacd9f1af2c8991d8 P dee2d58fd60f4b39a553c007f76ac760 [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:44:53.100986 21769 raft_consensus.cc:504] T ae87ef2626584e6bacd9f1af2c8991d8 P dee2d58fd60f4b39a553c007f76ac760 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } }
I20260809 15:44:53.102208 21769 leader_election.cc:288] T ae87ef2626584e6bacd9f1af2c8991d8 P dee2d58fd60f4b39a553c007f76ac760 [CANDIDATE]: Term 3 election: Requested vote from peers faa9b2a805cd43eca626c68bc7e7168f (127.20.202.65:37685), 6517ff96fe0a42a2bde064dc7f1ecc97 (127.20.202.66:42701)
I20260809 15:44:53.103030 21510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae87ef2626584e6bacd9f1af2c8991d8" candidate_uuid: "dee2d58fd60f4b39a553c007f76ac760" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "faa9b2a805cd43eca626c68bc7e7168f"
I20260809 15:44:53.103181 21633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae87ef2626584e6bacd9f1af2c8991d8" candidate_uuid: "dee2d58fd60f4b39a553c007f76ac760" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97"
I20260809 15:44:53.103633 21510 raft_consensus.cc:3037] T ae87ef2626584e6bacd9f1af2c8991d8 P faa9b2a805cd43eca626c68bc7e7168f [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:44:53.103689 21633 raft_consensus.cc:3032] T ae87ef2626584e6bacd9f1af2c8991d8 P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 2 LEADER]: Stepping down as leader of term 2
I20260809 15:44:53.104040 21633 raft_consensus.cc:729] T ae87ef2626584e6bacd9f1af2c8991d8 P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 6517ff96fe0a42a2bde064dc7f1ecc97, State: Running, Role: LEADER
I20260809 15:44:53.104759 21633 consensus_queue.cc:260] T ae87ef2626584e6bacd9f1af2c8991d8 P 6517ff96fe0a42a2bde064dc7f1ecc97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } }
I20260809 15:44:53.105908 21633 raft_consensus.cc:3037] T ae87ef2626584e6bacd9f1af2c8991d8 P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:44:53.109423 21510 raft_consensus.cc:2454] T ae87ef2626584e6bacd9f1af2c8991d8 P faa9b2a805cd43eca626c68bc7e7168f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dee2d58fd60f4b39a553c007f76ac760 in term 3.
I20260809 15:44:53.110405 21709 leader_election.cc:302] T ae87ef2626584e6bacd9f1af2c8991d8 P dee2d58fd60f4b39a553c007f76ac760 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dee2d58fd60f4b39a553c007f76ac760, faa9b2a805cd43eca626c68bc7e7168f; no voters:
I20260809 15:44:53.110984 22033 raft_consensus.cc:2781] T ae87ef2626584e6bacd9f1af2c8991d8 P dee2d58fd60f4b39a553c007f76ac760 [term 3 FOLLOWER]: Leader election won for term 3
I20260809 15:44:53.111203 21633 raft_consensus.cc:2454] T ae87ef2626584e6bacd9f1af2c8991d8 P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dee2d58fd60f4b39a553c007f76ac760 in term 3.
I20260809 15:44:53.112150 22033 raft_consensus.cc:686] T ae87ef2626584e6bacd9f1af2c8991d8 P dee2d58fd60f4b39a553c007f76ac760 [term 3 LEADER]: Becoming Leader. State: Replica: dee2d58fd60f4b39a553c007f76ac760, State: Running, Role: LEADER
I20260809 15:44:53.112859 22033 consensus_queue.cc:237] T ae87ef2626584e6bacd9f1af2c8991d8 P dee2d58fd60f4b39a553c007f76ac760 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } }
I20260809 15:44:53.118937 21333 catalog_manager.cc:5495] T ae87ef2626584e6bacd9f1af2c8991d8 P dee2d58fd60f4b39a553c007f76ac760 reported cstate change: term changed from 2 to 3, leader changed from 6517ff96fe0a42a2bde064dc7f1ecc97 (127.20.202.66) to dee2d58fd60f4b39a553c007f76ac760 (127.20.202.67). New cstate: current_term: 3 leader_uuid: "dee2d58fd60f4b39a553c007f76ac760" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } health_report { overall_health: HEALTHY } } }
I20260809 15:44:53.246611 22034 raft_consensus.cc:979] T d203a1017c6b44d0a62bbc654ee21d8c P faa9b2a805cd43eca626c68bc7e7168f: : Instructing follower dee2d58fd60f4b39a553c007f76ac760 to start an election
I20260809 15:44:53.247028 22051 raft_consensus.cc:1067] T d203a1017c6b44d0a62bbc654ee21d8c P faa9b2a805cd43eca626c68bc7e7168f [term 1 LEADER]: Signalling peer dee2d58fd60f4b39a553c007f76ac760 to start an election
I20260809 15:44:53.248520 21769 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d203a1017c6b44d0a62bbc654ee21d8c"
dest_uuid: "dee2d58fd60f4b39a553c007f76ac760"
from {username='slave'} at 127.0.0.1:34924
I20260809 15:44:53.249006 21769 raft_consensus.cc:482] T d203a1017c6b44d0a62bbc654ee21d8c P dee2d58fd60f4b39a553c007f76ac760 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 15:44:53.249289 21769 raft_consensus.cc:3037] T d203a1017c6b44d0a62bbc654ee21d8c P dee2d58fd60f4b39a553c007f76ac760 [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:44:53.252960 21769 raft_consensus.cc:504] T d203a1017c6b44d0a62bbc654ee21d8c P dee2d58fd60f4b39a553c007f76ac760 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } }
I20260809 15:44:53.254284 21769 leader_election.cc:288] T d203a1017c6b44d0a62bbc654ee21d8c P dee2d58fd60f4b39a553c007f76ac760 [CANDIDATE]: Term 2 election: Requested vote from peers faa9b2a805cd43eca626c68bc7e7168f (127.20.202.65:37685), 6517ff96fe0a42a2bde064dc7f1ecc97 (127.20.202.66:42701)
I20260809 15:44:53.254987 21510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d203a1017c6b44d0a62bbc654ee21d8c" candidate_uuid: "dee2d58fd60f4b39a553c007f76ac760" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "faa9b2a805cd43eca626c68bc7e7168f"
I20260809 15:44:53.255290 21633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d203a1017c6b44d0a62bbc654ee21d8c" candidate_uuid: "dee2d58fd60f4b39a553c007f76ac760" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97"
I20260809 15:44:53.255492 21510 raft_consensus.cc:3032] T d203a1017c6b44d0a62bbc654ee21d8c P faa9b2a805cd43eca626c68bc7e7168f [term 1 LEADER]: Stepping down as leader of term 1
I20260809 15:44:53.255756 21633 raft_consensus.cc:3037] T d203a1017c6b44d0a62bbc654ee21d8c P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:44:53.255770 21510 raft_consensus.cc:729] T d203a1017c6b44d0a62bbc654ee21d8c P faa9b2a805cd43eca626c68bc7e7168f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: faa9b2a805cd43eca626c68bc7e7168f, State: Running, Role: LEADER
I20260809 15:44:53.256471 21510 consensus_queue.cc:260] T d203a1017c6b44d0a62bbc654ee21d8c P faa9b2a805cd43eca626c68bc7e7168f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } }
I20260809 15:44:53.257292 21510 raft_consensus.cc:3037] T d203a1017c6b44d0a62bbc654ee21d8c P faa9b2a805cd43eca626c68bc7e7168f [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:44:53.259759 21633 raft_consensus.cc:2454] T d203a1017c6b44d0a62bbc654ee21d8c P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dee2d58fd60f4b39a553c007f76ac760 in term 2.
I20260809 15:44:53.260540 21710 leader_election.cc:302] T d203a1017c6b44d0a62bbc654ee21d8c P dee2d58fd60f4b39a553c007f76ac760 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6517ff96fe0a42a2bde064dc7f1ecc97, dee2d58fd60f4b39a553c007f76ac760; no voters:
I20260809 15:44:53.261123 22033 raft_consensus.cc:2781] T d203a1017c6b44d0a62bbc654ee21d8c P dee2d58fd60f4b39a553c007f76ac760 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 15:44:53.261332 21510 raft_consensus.cc:2454] T d203a1017c6b44d0a62bbc654ee21d8c P faa9b2a805cd43eca626c68bc7e7168f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dee2d58fd60f4b39a553c007f76ac760 in term 2.
I20260809 15:44:53.262660 22033 raft_consensus.cc:686] T d203a1017c6b44d0a62bbc654ee21d8c P dee2d58fd60f4b39a553c007f76ac760 [term 2 LEADER]: Becoming Leader. State: Replica: dee2d58fd60f4b39a553c007f76ac760, State: Running, Role: LEADER
I20260809 15:44:53.263419 22033 consensus_queue.cc:237] T d203a1017c6b44d0a62bbc654ee21d8c P dee2d58fd60f4b39a553c007f76ac760 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } }
I20260809 15:44:53.271734 21333 catalog_manager.cc:5495] T d203a1017c6b44d0a62bbc654ee21d8c P dee2d58fd60f4b39a553c007f76ac760 reported cstate change: term changed from 1 to 2, leader changed from faa9b2a805cd43eca626c68bc7e7168f (127.20.202.65) to dee2d58fd60f4b39a553c007f76ac760 (127.20.202.67). New cstate: current_term: 2 leader_uuid: "dee2d58fd60f4b39a553c007f76ac760" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } health_report { overall_health: HEALTHY } } }
I20260809 15:44:53.656754 21633 raft_consensus.cc:1261] T ae87ef2626584e6bacd9f1af2c8991d8 P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 3 FOLLOWER]: Refusing update from remote peer dee2d58fd60f4b39a553c007f76ac760: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260809 15:44:53.657730 22033 consensus_queue.cc:1035] T ae87ef2626584e6bacd9f1af2c8991d8 P dee2d58fd60f4b39a553c007f76ac760 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 15:44:53.666275 21510 raft_consensus.cc:1261] T ae87ef2626584e6bacd9f1af2c8991d8 P faa9b2a805cd43eca626c68bc7e7168f [term 3 FOLLOWER]: Refusing update from remote peer dee2d58fd60f4b39a553c007f76ac760: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260809 15:44:53.667690 22053 consensus_queue.cc:1035] T ae87ef2626584e6bacd9f1af2c8991d8 P dee2d58fd60f4b39a553c007f76ac760 [LEADER]: Connected to new peer: Peer: permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 15:44:53.725009 21510 raft_consensus.cc:1261] T d203a1017c6b44d0a62bbc654ee21d8c P faa9b2a805cd43eca626c68bc7e7168f [term 2 FOLLOWER]: Refusing update from remote peer dee2d58fd60f4b39a553c007f76ac760: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 15:44:53.725986 22053 consensus_queue.cc:1035] T d203a1017c6b44d0a62bbc654ee21d8c P dee2d58fd60f4b39a553c007f76ac760 [LEADER]: Connected to new peer: Peer: permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 15:44:53.732805 21633 raft_consensus.cc:1261] T d203a1017c6b44d0a62bbc654ee21d8c P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 2 FOLLOWER]: Refusing update from remote peer dee2d58fd60f4b39a553c007f76ac760: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 15:44:53.734225 22033 consensus_queue.cc:1035] T d203a1017c6b44d0a62bbc654ee21d8c P dee2d58fd60f4b39a553c007f76ac760 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 15:44:53.876458 21289 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 15:44:53.880942 21289 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260809 15:44:53.881273 21289 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 15:44:54.881964 21289 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 15:44:54.886442 21289 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260809 15:44:54.886795 21289 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 15:44:55.887545 21289 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 15:44:55.892028 21289 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260809 15:44:55.892390 21289 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 15:44:56.893321 21289 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 15:44:56.898833 21289 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260809 15:44:56.899304 21289 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 15:44:57.900029 21289 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 15:44:57.904408 21289 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260809 15:44:57.904711 21289 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 15:44:58.905328 21289 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 15:44:58.909641 21289 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260809 15:44:58.909945 21289 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 15:44:59.910595 21289 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 15:44:59.914980 21289 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260809 15:44:59.915343 21289 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 15:45:00.915984 21289 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 15:45:00.920267 21289 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260809 15:45:00.920569 21289 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 15:45:01.921594 21289 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 15:45:01.927647 21289 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260809 15:45:01.928040 21289 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 15:45:02.951741 21289 auto_leader_rebalancer-test.cc:288] The leader distribution is
I20260809 15:45:02.955506 21770 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d203a1017c6b44d0a62bbc654ee21d8c"
dest_uuid: "dee2d58fd60f4b39a553c007f76ac760"
mode: GRACEFUL
new_leader_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97"
from {username='slave'} at 127.0.0.1:56218
I20260809 15:45:02.956110 21770 raft_consensus.cc:595] T d203a1017c6b44d0a62bbc654ee21d8c P dee2d58fd60f4b39a553c007f76ac760 [term 2 LEADER]: Received request to transfer leadership to TS 6517ff96fe0a42a2bde064dc7f1ecc97
I20260809 15:45:02.960183 21510 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8d9c4e17e76743439089a2a53c45d01d"
dest_uuid: "faa9b2a805cd43eca626c68bc7e7168f"
mode: GRACEFUL
new_leader_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97"
from {username='slave'} at 127.0.0.1:59382
I20260809 15:45:02.960570 21510 raft_consensus.cc:595] T 8d9c4e17e76743439089a2a53c45d01d P faa9b2a805cd43eca626c68bc7e7168f [term 2 LEADER]: Received request to transfer leadership to TS 6517ff96fe0a42a2bde064dc7f1ecc97
I20260809 15:45:02.964386 21510 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d18bbcc7ebd14b479ddacb4aad814d47"
dest_uuid: "faa9b2a805cd43eca626c68bc7e7168f"
mode: GRACEFUL
new_leader_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97"
from {username='slave'} at 127.0.0.1:59382
I20260809 15:45:02.964758 21510 raft_consensus.cc:595] T d18bbcc7ebd14b479ddacb4aad814d47 P faa9b2a805cd43eca626c68bc7e7168f [term 1 LEADER]: Received request to transfer leadership to TS 6517ff96fe0a42a2bde064dc7f1ecc97
I20260809 15:45:02.969036 21510 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4fe458f602674472bedadfdd3c7ee1e6"
dest_uuid: "faa9b2a805cd43eca626c68bc7e7168f"
mode: GRACEFUL
new_leader_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97"
from {username='slave'} at 127.0.0.1:59382
I20260809 15:45:02.969461 21510 raft_consensus.cc:595] T 4fe458f602674472bedadfdd3c7ee1e6 P faa9b2a805cd43eca626c68bc7e7168f [term 1 LEADER]: Received request to transfer leadership to TS 6517ff96fe0a42a2bde064dc7f1ecc97
I20260809 15:45:02.973237 21770 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ae87ef2626584e6bacd9f1af2c8991d8"
dest_uuid: "dee2d58fd60f4b39a553c007f76ac760"
mode: GRACEFUL
new_leader_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97"
from {username='slave'} at 127.0.0.1:56218
I20260809 15:45:02.973645 21770 raft_consensus.cc:595] T ae87ef2626584e6bacd9f1af2c8991d8 P dee2d58fd60f4b39a553c007f76ac760 [term 3 LEADER]: Received request to transfer leadership to TS 6517ff96fe0a42a2bde064dc7f1ecc97
I20260809 15:45:03.102308 22033 raft_consensus.cc:979] T d203a1017c6b44d0a62bbc654ee21d8c P dee2d58fd60f4b39a553c007f76ac760: : Instructing follower 6517ff96fe0a42a2bde064dc7f1ecc97 to start an election
I20260809 15:45:03.102800 22085 raft_consensus.cc:1067] T d203a1017c6b44d0a62bbc654ee21d8c P dee2d58fd60f4b39a553c007f76ac760 [term 2 LEADER]: Signalling peer 6517ff96fe0a42a2bde064dc7f1ecc97 to start an election
I20260809 15:45:03.104171 21635 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d203a1017c6b44d0a62bbc654ee21d8c"
dest_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97"
from {username='slave'} at 127.0.0.1:39640
I20260809 15:45:03.104602 21635 raft_consensus.cc:482] T d203a1017c6b44d0a62bbc654ee21d8c P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 15:45:03.104884 21635 raft_consensus.cc:3037] T d203a1017c6b44d0a62bbc654ee21d8c P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:45:03.110255 21635 raft_consensus.cc:504] T d203a1017c6b44d0a62bbc654ee21d8c P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } }
I20260809 15:45:03.111660 21635 leader_election.cc:288] T d203a1017c6b44d0a62bbc654ee21d8c P 6517ff96fe0a42a2bde064dc7f1ecc97 [CANDIDATE]: Term 3 election: Requested vote from peers faa9b2a805cd43eca626c68bc7e7168f (127.20.202.65:37685), dee2d58fd60f4b39a553c007f76ac760 (127.20.202.67:36447)
I20260809 15:45:03.112246 21510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d203a1017c6b44d0a62bbc654ee21d8c" candidate_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "faa9b2a805cd43eca626c68bc7e7168f"
I20260809 15:45:03.112867 21510 raft_consensus.cc:3037] T d203a1017c6b44d0a62bbc654ee21d8c P faa9b2a805cd43eca626c68bc7e7168f [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:45:03.112679 21770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d203a1017c6b44d0a62bbc654ee21d8c" candidate_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "dee2d58fd60f4b39a553c007f76ac760"
I20260809 15:45:03.113276 21770 raft_consensus.cc:3032] T d203a1017c6b44d0a62bbc654ee21d8c P dee2d58fd60f4b39a553c007f76ac760 [term 2 LEADER]: Stepping down as leader of term 2
I20260809 15:45:03.113507 21770 raft_consensus.cc:729] T d203a1017c6b44d0a62bbc654ee21d8c P dee2d58fd60f4b39a553c007f76ac760 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: dee2d58fd60f4b39a553c007f76ac760, State: Running, Role: LEADER
I20260809 15:45:03.114096 21770 consensus_queue.cc:260] T d203a1017c6b44d0a62bbc654ee21d8c P dee2d58fd60f4b39a553c007f76ac760 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } }
I20260809 15:45:03.115144 21770 raft_consensus.cc:3037] T d203a1017c6b44d0a62bbc654ee21d8c P dee2d58fd60f4b39a553c007f76ac760 [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:45:03.117000 21510 raft_consensus.cc:2454] T d203a1017c6b44d0a62bbc654ee21d8c P faa9b2a805cd43eca626c68bc7e7168f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6517ff96fe0a42a2bde064dc7f1ecc97 in term 3.
I20260809 15:45:03.117980 21574 leader_election.cc:302] T d203a1017c6b44d0a62bbc654ee21d8c P 6517ff96fe0a42a2bde064dc7f1ecc97 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6517ff96fe0a42a2bde064dc7f1ecc97, faa9b2a805cd43eca626c68bc7e7168f; no voters:
I20260809 15:45:03.118865 22083 raft_consensus.cc:2781] T d203a1017c6b44d0a62bbc654ee21d8c P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 3 FOLLOWER]: Leader election won for term 3
I20260809 15:45:03.119982 22083 raft_consensus.cc:686] T d203a1017c6b44d0a62bbc654ee21d8c P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 3 LEADER]: Becoming Leader. State: Replica: 6517ff96fe0a42a2bde064dc7f1ecc97, State: Running, Role: LEADER
I20260809 15:45:03.120119 21770 raft_consensus.cc:2454] T d203a1017c6b44d0a62bbc654ee21d8c P dee2d58fd60f4b39a553c007f76ac760 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6517ff96fe0a42a2bde064dc7f1ecc97 in term 3.
I20260809 15:45:03.121268 22033 raft_consensus.cc:979] T ae87ef2626584e6bacd9f1af2c8991d8 P dee2d58fd60f4b39a553c007f76ac760: : Instructing follower 6517ff96fe0a42a2bde064dc7f1ecc97 to start an election
I20260809 15:45:03.120932 22083 consensus_queue.cc:237] T d203a1017c6b44d0a62bbc654ee21d8c P 6517ff96fe0a42a2bde064dc7f1ecc97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } }
I20260809 15:45:03.121632 22033 raft_consensus.cc:1067] T ae87ef2626584e6bacd9f1af2c8991d8 P dee2d58fd60f4b39a553c007f76ac760 [term 3 LEADER]: Signalling peer 6517ff96fe0a42a2bde064dc7f1ecc97 to start an election
I20260809 15:45:03.122851 21635 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ae87ef2626584e6bacd9f1af2c8991d8"
dest_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97"
from {username='slave'} at 127.0.0.1:39640
I20260809 15:45:03.123293 21635 raft_consensus.cc:482] T ae87ef2626584e6bacd9f1af2c8991d8 P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 15:45:03.123548 21635 raft_consensus.cc:3037] T ae87ef2626584e6bacd9f1af2c8991d8 P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:45:03.127955 21635 raft_consensus.cc:504] T ae87ef2626584e6bacd9f1af2c8991d8 P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } }
I20260809 15:45:03.128038 21333 catalog_manager.cc:5495] T d203a1017c6b44d0a62bbc654ee21d8c P 6517ff96fe0a42a2bde064dc7f1ecc97 reported cstate change: term changed from 2 to 3, leader changed from dee2d58fd60f4b39a553c007f76ac760 (127.20.202.67) to 6517ff96fe0a42a2bde064dc7f1ecc97 (127.20.202.66). New cstate: current_term: 3 leader_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } health_report { overall_health: UNKNOWN } } }
I20260809 15:45:03.129582 21635 leader_election.cc:288] T ae87ef2626584e6bacd9f1af2c8991d8 P 6517ff96fe0a42a2bde064dc7f1ecc97 [CANDIDATE]: Term 4 election: Requested vote from peers faa9b2a805cd43eca626c68bc7e7168f (127.20.202.65:37685), dee2d58fd60f4b39a553c007f76ac760 (127.20.202.67:36447)
I20260809 15:45:03.129849 21510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae87ef2626584e6bacd9f1af2c8991d8" candidate_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "faa9b2a805cd43eca626c68bc7e7168f"
I20260809 15:45:03.130234 21510 raft_consensus.cc:3037] T ae87ef2626584e6bacd9f1af2c8991d8 P faa9b2a805cd43eca626c68bc7e7168f [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:45:03.130146 21770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae87ef2626584e6bacd9f1af2c8991d8" candidate_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "dee2d58fd60f4b39a553c007f76ac760"
I20260809 15:45:03.130715 21770 raft_consensus.cc:3032] T ae87ef2626584e6bacd9f1af2c8991d8 P dee2d58fd60f4b39a553c007f76ac760 [term 3 LEADER]: Stepping down as leader of term 3
I20260809 15:45:03.130925 21770 raft_consensus.cc:729] T ae87ef2626584e6bacd9f1af2c8991d8 P dee2d58fd60f4b39a553c007f76ac760 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: dee2d58fd60f4b39a553c007f76ac760, State: Running, Role: LEADER
I20260809 15:45:03.131507 21770 consensus_queue.cc:260] T ae87ef2626584e6bacd9f1af2c8991d8 P dee2d58fd60f4b39a553c007f76ac760 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } }
I20260809 15:45:03.132326 21770 raft_consensus.cc:3037] T ae87ef2626584e6bacd9f1af2c8991d8 P dee2d58fd60f4b39a553c007f76ac760 [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:45:03.134943 21510 raft_consensus.cc:2454] T ae87ef2626584e6bacd9f1af2c8991d8 P faa9b2a805cd43eca626c68bc7e7168f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6517ff96fe0a42a2bde064dc7f1ecc97 in term 4.
I20260809 15:45:03.135830 21574 leader_election.cc:302] T ae87ef2626584e6bacd9f1af2c8991d8 P 6517ff96fe0a42a2bde064dc7f1ecc97 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6517ff96fe0a42a2bde064dc7f1ecc97, faa9b2a805cd43eca626c68bc7e7168f; no voters:
I20260809 15:45:03.136603 22083 raft_consensus.cc:2781] T ae87ef2626584e6bacd9f1af2c8991d8 P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 4 FOLLOWER]: Leader election won for term 4
I20260809 15:45:03.136674 21770 raft_consensus.cc:2454] T ae87ef2626584e6bacd9f1af2c8991d8 P dee2d58fd60f4b39a553c007f76ac760 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6517ff96fe0a42a2bde064dc7f1ecc97 in term 4.
I20260809 15:45:03.137130 22083 raft_consensus.cc:686] T ae87ef2626584e6bacd9f1af2c8991d8 P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 4 LEADER]: Becoming Leader. State: Replica: 6517ff96fe0a42a2bde064dc7f1ecc97, State: Running, Role: LEADER
I20260809 15:45:03.137967 22083 consensus_queue.cc:237] T ae87ef2626584e6bacd9f1af2c8991d8 P 6517ff96fe0a42a2bde064dc7f1ecc97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } }
I20260809 15:45:03.152513 21333 catalog_manager.cc:5495] T ae87ef2626584e6bacd9f1af2c8991d8 P 6517ff96fe0a42a2bde064dc7f1ecc97 reported cstate change: term changed from 3 to 4, leader changed from dee2d58fd60f4b39a553c007f76ac760 (127.20.202.67) to 6517ff96fe0a42a2bde064dc7f1ecc97 (127.20.202.66). New cstate: current_term: 4 leader_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } health_report { overall_health: UNKNOWN } } }
I20260809 15:45:03.282186 22078 raft_consensus.cc:979] T 4fe458f602674472bedadfdd3c7ee1e6 P faa9b2a805cd43eca626c68bc7e7168f: : Instructing follower 6517ff96fe0a42a2bde064dc7f1ecc97 to start an election
I20260809 15:45:03.282629 22095 raft_consensus.cc:1067] T 4fe458f602674472bedadfdd3c7ee1e6 P faa9b2a805cd43eca626c68bc7e7168f [term 1 LEADER]: Signalling peer 6517ff96fe0a42a2bde064dc7f1ecc97 to start an election
I20260809 15:45:03.283874 21635 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4fe458f602674472bedadfdd3c7ee1e6"
dest_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97"
from {username='slave'} at 127.0.0.1:39642
I20260809 15:45:03.284250 21635 raft_consensus.cc:482] T 4fe458f602674472bedadfdd3c7ee1e6 P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 15:45:03.284466 21635 raft_consensus.cc:3037] T 4fe458f602674472bedadfdd3c7ee1e6 P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:45:03.288306 21635 raft_consensus.cc:504] T 4fe458f602674472bedadfdd3c7ee1e6 P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } }
I20260809 15:45:03.289558 21635 leader_election.cc:288] T 4fe458f602674472bedadfdd3c7ee1e6 P 6517ff96fe0a42a2bde064dc7f1ecc97 [CANDIDATE]: Term 2 election: Requested vote from peers faa9b2a805cd43eca626c68bc7e7168f (127.20.202.65:37685), dee2d58fd60f4b39a553c007f76ac760 (127.20.202.67:36447)
I20260809 15:45:03.290194 21510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe458f602674472bedadfdd3c7ee1e6" candidate_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "faa9b2a805cd43eca626c68bc7e7168f"
I20260809 15:45:03.290511 21770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe458f602674472bedadfdd3c7ee1e6" candidate_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "dee2d58fd60f4b39a553c007f76ac760"
I20260809 15:45:03.290726 21510 raft_consensus.cc:3032] T 4fe458f602674472bedadfdd3c7ee1e6 P faa9b2a805cd43eca626c68bc7e7168f [term 1 LEADER]: Stepping down as leader of term 1
I20260809 15:45:03.291005 21510 raft_consensus.cc:729] T 4fe458f602674472bedadfdd3c7ee1e6 P faa9b2a805cd43eca626c68bc7e7168f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: faa9b2a805cd43eca626c68bc7e7168f, State: Running, Role: LEADER
I20260809 15:45:03.291033 21770 raft_consensus.cc:3037] T 4fe458f602674472bedadfdd3c7ee1e6 P dee2d58fd60f4b39a553c007f76ac760 [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:45:03.291787 21510 consensus_queue.cc:260] T 4fe458f602674472bedadfdd3c7ee1e6 P faa9b2a805cd43eca626c68bc7e7168f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } }
I20260809 15:45:03.292949 21510 raft_consensus.cc:3037] T 4fe458f602674472bedadfdd3c7ee1e6 P faa9b2a805cd43eca626c68bc7e7168f [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:45:03.297043 21770 raft_consensus.cc:2454] T 4fe458f602674472bedadfdd3c7ee1e6 P dee2d58fd60f4b39a553c007f76ac760 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6517ff96fe0a42a2bde064dc7f1ecc97 in term 2.
I20260809 15:45:03.298223 21577 leader_election.cc:302] T 4fe458f602674472bedadfdd3c7ee1e6 P 6517ff96fe0a42a2bde064dc7f1ecc97 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6517ff96fe0a42a2bde064dc7f1ecc97, dee2d58fd60f4b39a553c007f76ac760; no voters:
I20260809 15:45:03.298576 21510 raft_consensus.cc:2454] T 4fe458f602674472bedadfdd3c7ee1e6 P faa9b2a805cd43eca626c68bc7e7168f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6517ff96fe0a42a2bde064dc7f1ecc97 in term 2.
I20260809 15:45:03.299161 22086 raft_consensus.cc:2781] T 4fe458f602674472bedadfdd3c7ee1e6 P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 15:45:03.299631 22086 raft_consensus.cc:686] T 4fe458f602674472bedadfdd3c7ee1e6 P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 2 LEADER]: Becoming Leader. State: Replica: 6517ff96fe0a42a2bde064dc7f1ecc97, State: Running, Role: LEADER
I20260809 15:45:03.300455 22086 consensus_queue.cc:237] T 4fe458f602674472bedadfdd3c7ee1e6 P 6517ff96fe0a42a2bde064dc7f1ecc97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } }
I20260809 15:45:03.300927 22095 raft_consensus.cc:979] T d18bbcc7ebd14b479ddacb4aad814d47 P faa9b2a805cd43eca626c68bc7e7168f: : Instructing follower 6517ff96fe0a42a2bde064dc7f1ecc97 to start an election
I20260809 15:45:03.301357 22078 raft_consensus.cc:1067] T d18bbcc7ebd14b479ddacb4aad814d47 P faa9b2a805cd43eca626c68bc7e7168f [term 1 LEADER]: Signalling peer 6517ff96fe0a42a2bde064dc7f1ecc97 to start an election
I20260809 15:45:03.302641 21635 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d18bbcc7ebd14b479ddacb4aad814d47"
dest_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97"
from {username='slave'} at 127.0.0.1:39642
I20260809 15:45:03.303100 21635 raft_consensus.cc:482] T d18bbcc7ebd14b479ddacb4aad814d47 P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 15:45:03.303452 21635 raft_consensus.cc:3037] T d18bbcc7ebd14b479ddacb4aad814d47 P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:45:03.308007 21635 raft_consensus.cc:504] T d18bbcc7ebd14b479ddacb4aad814d47 P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } }
I20260809 15:45:03.308081 21333 catalog_manager.cc:5495] T 4fe458f602674472bedadfdd3c7ee1e6 P 6517ff96fe0a42a2bde064dc7f1ecc97 reported cstate change: term changed from 1 to 2, leader changed from faa9b2a805cd43eca626c68bc7e7168f (127.20.202.65) to 6517ff96fe0a42a2bde064dc7f1ecc97 (127.20.202.66). New cstate: current_term: 2 leader_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } health_report { overall_health: UNKNOWN } } }
I20260809 15:45:03.309953 21635 leader_election.cc:288] T d18bbcc7ebd14b479ddacb4aad814d47 P 6517ff96fe0a42a2bde064dc7f1ecc97 [CANDIDATE]: Term 2 election: Requested vote from peers faa9b2a805cd43eca626c68bc7e7168f (127.20.202.65:37685), dee2d58fd60f4b39a553c007f76ac760 (127.20.202.67:36447)
I20260809 15:45:03.310544 21769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d18bbcc7ebd14b479ddacb4aad814d47" candidate_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "dee2d58fd60f4b39a553c007f76ac760"
I20260809 15:45:03.310765 21510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d18bbcc7ebd14b479ddacb4aad814d47" candidate_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "faa9b2a805cd43eca626c68bc7e7168f"
I20260809 15:45:03.311038 21769 raft_consensus.cc:3037] T d18bbcc7ebd14b479ddacb4aad814d47 P dee2d58fd60f4b39a553c007f76ac760 [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:45:03.311211 21510 raft_consensus.cc:3032] T d18bbcc7ebd14b479ddacb4aad814d47 P faa9b2a805cd43eca626c68bc7e7168f [term 1 LEADER]: Stepping down as leader of term 1
I20260809 15:45:03.311509 21510 raft_consensus.cc:729] T d18bbcc7ebd14b479ddacb4aad814d47 P faa9b2a805cd43eca626c68bc7e7168f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: faa9b2a805cd43eca626c68bc7e7168f, State: Running, Role: LEADER
I20260809 15:45:03.312139 21510 consensus_queue.cc:260] T d18bbcc7ebd14b479ddacb4aad814d47 P faa9b2a805cd43eca626c68bc7e7168f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } }
I20260809 15:45:03.313262 21510 raft_consensus.cc:3037] T d18bbcc7ebd14b479ddacb4aad814d47 P faa9b2a805cd43eca626c68bc7e7168f [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:45:03.317231 21769 raft_consensus.cc:2454] T d18bbcc7ebd14b479ddacb4aad814d47 P dee2d58fd60f4b39a553c007f76ac760 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6517ff96fe0a42a2bde064dc7f1ecc97 in term 2.
I20260809 15:45:03.318068 21577 leader_election.cc:302] T d18bbcc7ebd14b479ddacb4aad814d47 P 6517ff96fe0a42a2bde064dc7f1ecc97 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6517ff96fe0a42a2bde064dc7f1ecc97, dee2d58fd60f4b39a553c007f76ac760; no voters:
I20260809 15:45:03.318274 21510 raft_consensus.cc:2454] T d18bbcc7ebd14b479ddacb4aad814d47 P faa9b2a805cd43eca626c68bc7e7168f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6517ff96fe0a42a2bde064dc7f1ecc97 in term 2.
I20260809 15:45:03.318655 22086 raft_consensus.cc:2781] T d18bbcc7ebd14b479ddacb4aad814d47 P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 15:45:03.319078 22086 raft_consensus.cc:686] T d18bbcc7ebd14b479ddacb4aad814d47 P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 2 LEADER]: Becoming Leader. State: Replica: 6517ff96fe0a42a2bde064dc7f1ecc97, State: Running, Role: LEADER
I20260809 15:45:03.319747 22086 consensus_queue.cc:237] T d18bbcc7ebd14b479ddacb4aad814d47 P 6517ff96fe0a42a2bde064dc7f1ecc97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } }
I20260809 15:45:03.325624 21333 catalog_manager.cc:5495] T d18bbcc7ebd14b479ddacb4aad814d47 P 6517ff96fe0a42a2bde064dc7f1ecc97 reported cstate change: term changed from 1 to 2, leader changed from faa9b2a805cd43eca626c68bc7e7168f (127.20.202.65) to 6517ff96fe0a42a2bde064dc7f1ecc97 (127.20.202.66). New cstate: current_term: 2 leader_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } health_report { overall_health: UNKNOWN } } }
I20260809 15:45:03.334228 22095 raft_consensus.cc:979] T 8d9c4e17e76743439089a2a53c45d01d P faa9b2a805cd43eca626c68bc7e7168f: : Instructing follower 6517ff96fe0a42a2bde064dc7f1ecc97 to start an election
I20260809 15:45:03.334542 22078 raft_consensus.cc:1067] T 8d9c4e17e76743439089a2a53c45d01d P faa9b2a805cd43eca626c68bc7e7168f [term 2 LEADER]: Signalling peer 6517ff96fe0a42a2bde064dc7f1ecc97 to start an election
I20260809 15:45:03.335651 21635 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8d9c4e17e76743439089a2a53c45d01d"
dest_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97"
from {username='slave'} at 127.0.0.1:39642
I20260809 15:45:03.336022 21635 raft_consensus.cc:482] T 8d9c4e17e76743439089a2a53c45d01d P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 15:45:03.336249 21635 raft_consensus.cc:3037] T 8d9c4e17e76743439089a2a53c45d01d P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:45:03.339818 21635 raft_consensus.cc:504] T 8d9c4e17e76743439089a2a53c45d01d P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } }
I20260809 15:45:03.340915 21635 leader_election.cc:288] T 8d9c4e17e76743439089a2a53c45d01d P 6517ff96fe0a42a2bde064dc7f1ecc97 [CANDIDATE]: Term 3 election: Requested vote from peers faa9b2a805cd43eca626c68bc7e7168f (127.20.202.65:37685), dee2d58fd60f4b39a553c007f76ac760 (127.20.202.67:36447)
I20260809 15:45:03.341521 21510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d9c4e17e76743439089a2a53c45d01d" candidate_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "faa9b2a805cd43eca626c68bc7e7168f"
I20260809 15:45:03.341802 21769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d9c4e17e76743439089a2a53c45d01d" candidate_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "dee2d58fd60f4b39a553c007f76ac760"
I20260809 15:45:03.342024 21510 raft_consensus.cc:3032] T 8d9c4e17e76743439089a2a53c45d01d P faa9b2a805cd43eca626c68bc7e7168f [term 2 LEADER]: Stepping down as leader of term 2
I20260809 15:45:03.342237 21769 raft_consensus.cc:3037] T 8d9c4e17e76743439089a2a53c45d01d P dee2d58fd60f4b39a553c007f76ac760 [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:45:03.342311 21510 raft_consensus.cc:729] T 8d9c4e17e76743439089a2a53c45d01d P faa9b2a805cd43eca626c68bc7e7168f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: faa9b2a805cd43eca626c68bc7e7168f, State: Running, Role: LEADER
I20260809 15:45:03.342928 21510 consensus_queue.cc:260] T 8d9c4e17e76743439089a2a53c45d01d P faa9b2a805cd43eca626c68bc7e7168f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } }
I20260809 15:45:03.343803 21510 raft_consensus.cc:3037] T 8d9c4e17e76743439089a2a53c45d01d P faa9b2a805cd43eca626c68bc7e7168f [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:45:03.345875 21769 raft_consensus.cc:2454] T 8d9c4e17e76743439089a2a53c45d01d P dee2d58fd60f4b39a553c007f76ac760 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6517ff96fe0a42a2bde064dc7f1ecc97 in term 3.
I20260809 15:45:03.346617 21577 leader_election.cc:302] T 8d9c4e17e76743439089a2a53c45d01d P 6517ff96fe0a42a2bde064dc7f1ecc97 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6517ff96fe0a42a2bde064dc7f1ecc97, dee2d58fd60f4b39a553c007f76ac760; no voters:
I20260809 15:45:03.347325 22086 raft_consensus.cc:2781] T 8d9c4e17e76743439089a2a53c45d01d P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 3 FOLLOWER]: Leader election won for term 3
I20260809 15:45:03.347407 21510 raft_consensus.cc:2454] T 8d9c4e17e76743439089a2a53c45d01d P faa9b2a805cd43eca626c68bc7e7168f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6517ff96fe0a42a2bde064dc7f1ecc97 in term 3.
I20260809 15:45:03.347699 22086 raft_consensus.cc:686] T 8d9c4e17e76743439089a2a53c45d01d P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 3 LEADER]: Becoming Leader. State: Replica: 6517ff96fe0a42a2bde064dc7f1ecc97, State: Running, Role: LEADER
I20260809 15:45:03.348402 22086 consensus_queue.cc:237] T 8d9c4e17e76743439089a2a53c45d01d P 6517ff96fe0a42a2bde064dc7f1ecc97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } }
I20260809 15:45:03.354147 21333 catalog_manager.cc:5495] T 8d9c4e17e76743439089a2a53c45d01d P 6517ff96fe0a42a2bde064dc7f1ecc97 reported cstate change: term changed from 2 to 3, leader changed from faa9b2a805cd43eca626c68bc7e7168f (127.20.202.65) to 6517ff96fe0a42a2bde064dc7f1ecc97 (127.20.202.66). New cstate: current_term: 3 leader_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } health_report { overall_health: UNKNOWN } } }
I20260809 15:45:03.657464 21769 raft_consensus.cc:1261] T ae87ef2626584e6bacd9f1af2c8991d8 P dee2d58fd60f4b39a553c007f76ac760 [term 4 FOLLOWER]: Refusing update from remote peer 6517ff96fe0a42a2bde064dc7f1ecc97: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260809 15:45:03.658434 22086 consensus_queue.cc:1035] T ae87ef2626584e6bacd9f1af2c8991d8 P 6517ff96fe0a42a2bde064dc7f1ecc97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260809 15:45:03.670843 21510 raft_consensus.cc:1261] T ae87ef2626584e6bacd9f1af2c8991d8 P faa9b2a805cd43eca626c68bc7e7168f [term 4 FOLLOWER]: Refusing update from remote peer 6517ff96fe0a42a2bde064dc7f1ecc97: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260809 15:45:03.671959 22083 consensus_queue.cc:1035] T ae87ef2626584e6bacd9f1af2c8991d8 P 6517ff96fe0a42a2bde064dc7f1ecc97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260809 15:45:03.678999 21769 raft_consensus.cc:1261] T d203a1017c6b44d0a62bbc654ee21d8c P dee2d58fd60f4b39a553c007f76ac760 [term 3 FOLLOWER]: Refusing update from remote peer 6517ff96fe0a42a2bde064dc7f1ecc97: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260809 15:45:03.680286 21769 raft_consensus.cc:1261] T 4fe458f602674472bedadfdd3c7ee1e6 P dee2d58fd60f4b39a553c007f76ac760 [term 2 FOLLOWER]: Refusing update from remote peer 6517ff96fe0a42a2bde064dc7f1ecc97: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 15:45:03.680312 22086 consensus_queue.cc:1035] T d203a1017c6b44d0a62bbc654ee21d8c P 6517ff96fe0a42a2bde064dc7f1ecc97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 15:45:03.681452 22083 consensus_queue.cc:1035] T 4fe458f602674472bedadfdd3c7ee1e6 P 6517ff96fe0a42a2bde064dc7f1ecc97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 15:45:03.687954 21510 raft_consensus.cc:1261] T d203a1017c6b44d0a62bbc654ee21d8c P faa9b2a805cd43eca626c68bc7e7168f [term 3 FOLLOWER]: Refusing update from remote peer 6517ff96fe0a42a2bde064dc7f1ecc97: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260809 15:45:03.691111 22083 consensus_queue.cc:1035] T d203a1017c6b44d0a62bbc654ee21d8c P 6517ff96fe0a42a2bde064dc7f1ecc97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 15:45:03.694980 21510 raft_consensus.cc:1261] T 4fe458f602674472bedadfdd3c7ee1e6 P faa9b2a805cd43eca626c68bc7e7168f [term 2 FOLLOWER]: Refusing update from remote peer 6517ff96fe0a42a2bde064dc7f1ecc97: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 15:45:03.696391 22086 consensus_queue.cc:1035] T 4fe458f602674472bedadfdd3c7ee1e6 P 6517ff96fe0a42a2bde064dc7f1ecc97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 15:45:03.792814 21769 raft_consensus.cc:1261] T 8d9c4e17e76743439089a2a53c45d01d P dee2d58fd60f4b39a553c007f76ac760 [term 3 FOLLOWER]: Refusing update from remote peer 6517ff96fe0a42a2bde064dc7f1ecc97: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260809 15:45:03.793787 22083 consensus_queue.cc:1035] T 8d9c4e17e76743439089a2a53c45d01d P 6517ff96fe0a42a2bde064dc7f1ecc97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 15:45:03.802124 21510 raft_consensus.cc:1261] T 8d9c4e17e76743439089a2a53c45d01d P faa9b2a805cd43eca626c68bc7e7168f [term 3 FOLLOWER]: Refusing update from remote peer 6517ff96fe0a42a2bde064dc7f1ecc97: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260809 15:45:03.803758 22113 consensus_queue.cc:1035] T 8d9c4e17e76743439089a2a53c45d01d P 6517ff96fe0a42a2bde064dc7f1ecc97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 15:45:03.830027 21510 raft_consensus.cc:1261] T d18bbcc7ebd14b479ddacb4aad814d47 P faa9b2a805cd43eca626c68bc7e7168f [term 2 FOLLOWER]: Refusing update from remote peer 6517ff96fe0a42a2bde064dc7f1ecc97: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 15:45:03.830906 22113 consensus_queue.cc:1035] T d18bbcc7ebd14b479ddacb4aad814d47 P 6517ff96fe0a42a2bde064dc7f1ecc97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 15:45:03.838474 21769 raft_consensus.cc:1261] T d18bbcc7ebd14b479ddacb4aad814d47 P dee2d58fd60f4b39a553c007f76ac760 [term 2 FOLLOWER]: Refusing update from remote peer 6517ff96fe0a42a2bde064dc7f1ecc97: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 15:45:03.839862 22113 consensus_queue.cc:1035] T d18bbcc7ebd14b479ddacb4aad814d47 P 6517ff96fe0a42a2bde064dc7f1ecc97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 15:45:06.003952 21289 auto_leader_rebalancer-test.cc:303] The leader distribution is
I20260809 15:45:06.004541 21289 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 15:45:06.010385 21635 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2f46e8d793294ce384acbf2f5584ecd2"
dest_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97"
mode: GRACEFUL
new_leader_uuid: "faa9b2a805cd43eca626c68bc7e7168f"
from {username='slave'} at 127.0.0.1:40824
I20260809 15:45:06.010808 21635 raft_consensus.cc:595] T 2f46e8d793294ce384acbf2f5584ecd2 P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 2 LEADER]: Received request to transfer leadership to TS faa9b2a805cd43eca626c68bc7e7168f
I20260809 15:45:06.012689 21635 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "724982e80d3b455a91f8a564d374a65e"
dest_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97"
mode: GRACEFUL
new_leader_uuid: "dee2d58fd60f4b39a553c007f76ac760"
from {username='slave'} at 127.0.0.1:40824
I20260809 15:45:06.013065 21635 raft_consensus.cc:595] T 724982e80d3b455a91f8a564d374a65e P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 1 LEADER]: Received request to transfer leadership to TS dee2d58fd60f4b39a553c007f76ac760
I20260809 15:45:06.014711 21635 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ab976a6064754a689b7c88c5fa968b86"
dest_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97"
mode: GRACEFUL
new_leader_uuid: "dee2d58fd60f4b39a553c007f76ac760"
from {username='slave'} at 127.0.0.1:40824
I20260809 15:45:06.015087 21635 raft_consensus.cc:595] T ab976a6064754a689b7c88c5fa968b86 P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 2 LEADER]: Received request to transfer leadership to TS dee2d58fd60f4b39a553c007f76ac760
I20260809 15:45:06.016728 21635 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d18bbcc7ebd14b479ddacb4aad814d47"
dest_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97"
mode: GRACEFUL
new_leader_uuid: "faa9b2a805cd43eca626c68bc7e7168f"
from {username='slave'} at 127.0.0.1:40824
I20260809 15:45:06.017081 21635 raft_consensus.cc:595] T d18bbcc7ebd14b479ddacb4aad814d47 P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 2 LEADER]: Received request to transfer leadership to TS faa9b2a805cd43eca626c68bc7e7168f
I20260809 15:45:06.018638 21635 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d203a1017c6b44d0a62bbc654ee21d8c"
dest_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97"
mode: GRACEFUL
new_leader_uuid: "faa9b2a805cd43eca626c68bc7e7168f"
from {username='slave'} at 127.0.0.1:40824
I20260809 15:45:06.018987 21635 raft_consensus.cc:595] T d203a1017c6b44d0a62bbc654ee21d8c P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 3 LEADER]: Received request to transfer leadership to TS faa9b2a805cd43eca626c68bc7e7168f
I20260809 15:45:06.019717 21289 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260809 15:45:06.020056 21289 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 15:45:06.075687 22113 raft_consensus.cc:979] T ab976a6064754a689b7c88c5fa968b86 P 6517ff96fe0a42a2bde064dc7f1ecc97: : Instructing follower dee2d58fd60f4b39a553c007f76ac760 to start an election
I20260809 15:45:06.076092 22126 raft_consensus.cc:1067] T ab976a6064754a689b7c88c5fa968b86 P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 2 LEADER]: Signalling peer dee2d58fd60f4b39a553c007f76ac760 to start an election
I20260809 15:45:06.077258 21769 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ab976a6064754a689b7c88c5fa968b86"
dest_uuid: "dee2d58fd60f4b39a553c007f76ac760"
from {username='slave'} at 127.0.0.1:34936
I20260809 15:45:06.077713 21769 raft_consensus.cc:482] T ab976a6064754a689b7c88c5fa968b86 P dee2d58fd60f4b39a553c007f76ac760 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 15:45:06.077993 21769 raft_consensus.cc:3037] T ab976a6064754a689b7c88c5fa968b86 P dee2d58fd60f4b39a553c007f76ac760 [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:45:06.081780 21769 raft_consensus.cc:504] T ab976a6064754a689b7c88c5fa968b86 P dee2d58fd60f4b39a553c007f76ac760 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } }
I20260809 15:45:06.084050 21769 leader_election.cc:288] T ab976a6064754a689b7c88c5fa968b86 P dee2d58fd60f4b39a553c007f76ac760 [CANDIDATE]: Term 3 election: Requested vote from peers faa9b2a805cd43eca626c68bc7e7168f (127.20.202.65:37685), 6517ff96fe0a42a2bde064dc7f1ecc97 (127.20.202.66:42701)
I20260809 15:45:06.084568 21635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab976a6064754a689b7c88c5fa968b86" candidate_uuid: "dee2d58fd60f4b39a553c007f76ac760" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97"
I20260809 15:45:06.084559 21513 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab976a6064754a689b7c88c5fa968b86" candidate_uuid: "dee2d58fd60f4b39a553c007f76ac760" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "faa9b2a805cd43eca626c68bc7e7168f"
I20260809 15:45:06.085135 21635 raft_consensus.cc:3032] T ab976a6064754a689b7c88c5fa968b86 P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 2 LEADER]: Stepping down as leader of term 2
I20260809 15:45:06.085299 21513 raft_consensus.cc:3037] T ab976a6064754a689b7c88c5fa968b86 P faa9b2a805cd43eca626c68bc7e7168f [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:45:06.085410 21635 raft_consensus.cc:729] T ab976a6064754a689b7c88c5fa968b86 P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 6517ff96fe0a42a2bde064dc7f1ecc97, State: Running, Role: LEADER
I20260809 15:45:06.086087 21635 consensus_queue.cc:260] T ab976a6064754a689b7c88c5fa968b86 P 6517ff96fe0a42a2bde064dc7f1ecc97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } }
I20260809 15:45:06.086889 21635 raft_consensus.cc:3037] T ab976a6064754a689b7c88c5fa968b86 P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:45:06.089633 21513 raft_consensus.cc:2454] T ab976a6064754a689b7c88c5fa968b86 P faa9b2a805cd43eca626c68bc7e7168f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dee2d58fd60f4b39a553c007f76ac760 in term 3.
I20260809 15:45:06.090499 21635 raft_consensus.cc:2454] T ab976a6064754a689b7c88c5fa968b86 P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dee2d58fd60f4b39a553c007f76ac760 in term 3.
I20260809 15:45:06.090546 21709 leader_election.cc:302] T ab976a6064754a689b7c88c5fa968b86 P dee2d58fd60f4b39a553c007f76ac760 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dee2d58fd60f4b39a553c007f76ac760, faa9b2a805cd43eca626c68bc7e7168f; no voters:
I20260809 15:45:06.091250 22085 raft_consensus.cc:2781] T ab976a6064754a689b7c88c5fa968b86 P dee2d58fd60f4b39a553c007f76ac760 [term 3 FOLLOWER]: Leader election won for term 3
I20260809 15:45:06.092023 22085 raft_consensus.cc:686] T ab976a6064754a689b7c88c5fa968b86 P dee2d58fd60f4b39a553c007f76ac760 [term 3 LEADER]: Becoming Leader. State: Replica: dee2d58fd60f4b39a553c007f76ac760, State: Running, Role: LEADER
I20260809 15:45:06.092730 22085 consensus_queue.cc:237] T ab976a6064754a689b7c88c5fa968b86 P dee2d58fd60f4b39a553c007f76ac760 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } }
I20260809 15:45:06.098093 21333 catalog_manager.cc:5495] T ab976a6064754a689b7c88c5fa968b86 P dee2d58fd60f4b39a553c007f76ac760 reported cstate change: term changed from 2 to 3, leader changed from 6517ff96fe0a42a2bde064dc7f1ecc97 (127.20.202.66) to dee2d58fd60f4b39a553c007f76ac760 (127.20.202.67). New cstate: current_term: 3 leader_uuid: "dee2d58fd60f4b39a553c007f76ac760" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } health_report { overall_health: HEALTHY } } }
I20260809 15:45:06.236794 22113 raft_consensus.cc:979] T 724982e80d3b455a91f8a564d374a65e P 6517ff96fe0a42a2bde064dc7f1ecc97: : Instructing follower dee2d58fd60f4b39a553c007f76ac760 to start an election
I20260809 15:45:06.237426 22126 raft_consensus.cc:1067] T 724982e80d3b455a91f8a564d374a65e P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 1 LEADER]: Signalling peer dee2d58fd60f4b39a553c007f76ac760 to start an election
I20260809 15:45:06.238819 21769 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "724982e80d3b455a91f8a564d374a65e"
dest_uuid: "dee2d58fd60f4b39a553c007f76ac760"
from {username='slave'} at 127.0.0.1:34936
I20260809 15:45:06.239746 21769 raft_consensus.cc:482] T 724982e80d3b455a91f8a564d374a65e P dee2d58fd60f4b39a553c007f76ac760 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 15:45:06.240265 21769 raft_consensus.cc:3037] T 724982e80d3b455a91f8a564d374a65e P dee2d58fd60f4b39a553c007f76ac760 [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:45:06.248629 21769 raft_consensus.cc:504] T 724982e80d3b455a91f8a564d374a65e P dee2d58fd60f4b39a553c007f76ac760 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } }
I20260809 15:45:06.250123 21769 leader_election.cc:288] T 724982e80d3b455a91f8a564d374a65e P dee2d58fd60f4b39a553c007f76ac760 [CANDIDATE]: Term 2 election: Requested vote from peers faa9b2a805cd43eca626c68bc7e7168f (127.20.202.65:37685), 6517ff96fe0a42a2bde064dc7f1ecc97 (127.20.202.66:42701)
I20260809 15:45:06.251755 21635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "724982e80d3b455a91f8a564d374a65e" candidate_uuid: "dee2d58fd60f4b39a553c007f76ac760" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97"
I20260809 15:45:06.251681 21513 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "724982e80d3b455a91f8a564d374a65e" candidate_uuid: "dee2d58fd60f4b39a553c007f76ac760" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "faa9b2a805cd43eca626c68bc7e7168f"
I20260809 15:45:06.252313 21635 raft_consensus.cc:3032] T 724982e80d3b455a91f8a564d374a65e P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 1 LEADER]: Stepping down as leader of term 1
I20260809 15:45:06.252442 21513 raft_consensus.cc:3037] T 724982e80d3b455a91f8a564d374a65e P faa9b2a805cd43eca626c68bc7e7168f [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:45:06.252588 21635 raft_consensus.cc:729] T 724982e80d3b455a91f8a564d374a65e P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6517ff96fe0a42a2bde064dc7f1ecc97, State: Running, Role: LEADER
I20260809 15:45:06.253252 21635 consensus_queue.cc:260] T 724982e80d3b455a91f8a564d374a65e P 6517ff96fe0a42a2bde064dc7f1ecc97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } }
I20260809 15:45:06.254688 21635 raft_consensus.cc:3037] T 724982e80d3b455a91f8a564d374a65e P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:45:06.258320 21513 raft_consensus.cc:2454] T 724982e80d3b455a91f8a564d374a65e P faa9b2a805cd43eca626c68bc7e7168f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dee2d58fd60f4b39a553c007f76ac760 in term 2.
I20260809 15:45:06.259326 21709 leader_election.cc:302] T 724982e80d3b455a91f8a564d374a65e P dee2d58fd60f4b39a553c007f76ac760 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dee2d58fd60f4b39a553c007f76ac760, faa9b2a805cd43eca626c68bc7e7168f; no voters:
I20260809 15:45:06.259507 21635 raft_consensus.cc:2454] T 724982e80d3b455a91f8a564d374a65e P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dee2d58fd60f4b39a553c007f76ac760 in term 2.
I20260809 15:45:06.260227 22085 raft_consensus.cc:2781] T 724982e80d3b455a91f8a564d374a65e P dee2d58fd60f4b39a553c007f76ac760 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 15:45:06.260588 22085 raft_consensus.cc:686] T 724982e80d3b455a91f8a564d374a65e P dee2d58fd60f4b39a553c007f76ac760 [term 2 LEADER]: Becoming Leader. State: Replica: dee2d58fd60f4b39a553c007f76ac760, State: Running, Role: LEADER
I20260809 15:45:06.261422 22085 consensus_queue.cc:237] T 724982e80d3b455a91f8a564d374a65e P dee2d58fd60f4b39a553c007f76ac760 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } }
I20260809 15:45:06.269068 21333 catalog_manager.cc:5495] T 724982e80d3b455a91f8a564d374a65e P dee2d58fd60f4b39a553c007f76ac760 reported cstate change: term changed from 1 to 2, leader changed from 6517ff96fe0a42a2bde064dc7f1ecc97 (127.20.202.66) to dee2d58fd60f4b39a553c007f76ac760 (127.20.202.67). New cstate: current_term: 2 leader_uuid: "dee2d58fd60f4b39a553c007f76ac760" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } health_report { overall_health: HEALTHY } } }
I20260809 15:45:06.335891 22113 raft_consensus.cc:979] T d203a1017c6b44d0a62bbc654ee21d8c P 6517ff96fe0a42a2bde064dc7f1ecc97: : Instructing follower faa9b2a805cd43eca626c68bc7e7168f to start an election
I20260809 15:45:06.336289 22126 raft_consensus.cc:1067] T d203a1017c6b44d0a62bbc654ee21d8c P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 3 LEADER]: Signalling peer faa9b2a805cd43eca626c68bc7e7168f to start an election
I20260809 15:45:06.337457 21513 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d203a1017c6b44d0a62bbc654ee21d8c"
dest_uuid: "faa9b2a805cd43eca626c68bc7e7168f"
from {username='slave'} at 127.0.0.1:50158
I20260809 15:45:06.337857 21513 raft_consensus.cc:482] T d203a1017c6b44d0a62bbc654ee21d8c P faa9b2a805cd43eca626c68bc7e7168f [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 15:45:06.338117 21513 raft_consensus.cc:3037] T d203a1017c6b44d0a62bbc654ee21d8c P faa9b2a805cd43eca626c68bc7e7168f [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:45:06.342453 21513 raft_consensus.cc:504] T d203a1017c6b44d0a62bbc654ee21d8c P faa9b2a805cd43eca626c68bc7e7168f [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } }
I20260809 15:45:06.344571 21513 leader_election.cc:288] T d203a1017c6b44d0a62bbc654ee21d8c P faa9b2a805cd43eca626c68bc7e7168f [CANDIDATE]: Term 4 election: Requested vote from peers 6517ff96fe0a42a2bde064dc7f1ecc97 (127.20.202.66:42701), dee2d58fd60f4b39a553c007f76ac760 (127.20.202.67:36447)
I20260809 15:45:06.345705 21769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d203a1017c6b44d0a62bbc654ee21d8c" candidate_uuid: "faa9b2a805cd43eca626c68bc7e7168f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "dee2d58fd60f4b39a553c007f76ac760"
I20260809 15:45:06.346045 21635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d203a1017c6b44d0a62bbc654ee21d8c" candidate_uuid: "faa9b2a805cd43eca626c68bc7e7168f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97"
I20260809 15:45:06.346212 21769 raft_consensus.cc:3037] T d203a1017c6b44d0a62bbc654ee21d8c P dee2d58fd60f4b39a553c007f76ac760 [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:45:06.346458 21635 raft_consensus.cc:3032] T d203a1017c6b44d0a62bbc654ee21d8c P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 3 LEADER]: Stepping down as leader of term 3
I20260809 15:45:06.346748 21635 raft_consensus.cc:729] T d203a1017c6b44d0a62bbc654ee21d8c P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 6517ff96fe0a42a2bde064dc7f1ecc97, State: Running, Role: LEADER
I20260809 15:45:06.347334 21635 consensus_queue.cc:260] T d203a1017c6b44d0a62bbc654ee21d8c P 6517ff96fe0a42a2bde064dc7f1ecc97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } }
I20260809 15:45:06.348403 21635 raft_consensus.cc:3037] T d203a1017c6b44d0a62bbc654ee21d8c P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:45:06.351970 21769 raft_consensus.cc:2454] T d203a1017c6b44d0a62bbc654ee21d8c P dee2d58fd60f4b39a553c007f76ac760 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate faa9b2a805cd43eca626c68bc7e7168f in term 4.
I20260809 15:45:06.352411 21635 raft_consensus.cc:2454] T d203a1017c6b44d0a62bbc654ee21d8c P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate faa9b2a805cd43eca626c68bc7e7168f in term 4.
I20260809 15:45:06.353137 21461 leader_election.cc:302] T d203a1017c6b44d0a62bbc654ee21d8c P faa9b2a805cd43eca626c68bc7e7168f [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dee2d58fd60f4b39a553c007f76ac760, faa9b2a805cd43eca626c68bc7e7168f; no voters:
I20260809 15:45:06.354616 22137 raft_consensus.cc:2781] T d203a1017c6b44d0a62bbc654ee21d8c P faa9b2a805cd43eca626c68bc7e7168f [term 4 FOLLOWER]: Leader election won for term 4
I20260809 15:45:06.355571 22137 raft_consensus.cc:686] T d203a1017c6b44d0a62bbc654ee21d8c P faa9b2a805cd43eca626c68bc7e7168f [term 4 LEADER]: Becoming Leader. State: Replica: faa9b2a805cd43eca626c68bc7e7168f, State: Running, Role: LEADER
I20260809 15:45:06.356266 22137 consensus_queue.cc:237] T d203a1017c6b44d0a62bbc654ee21d8c P faa9b2a805cd43eca626c68bc7e7168f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } }
I20260809 15:45:06.362708 22113 raft_consensus.cc:979] T 2f46e8d793294ce384acbf2f5584ecd2 P 6517ff96fe0a42a2bde064dc7f1ecc97: : Instructing follower faa9b2a805cd43eca626c68bc7e7168f to start an election
I20260809 15:45:06.363137 22126 raft_consensus.cc:1067] T 2f46e8d793294ce384acbf2f5584ecd2 P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 2 LEADER]: Signalling peer faa9b2a805cd43eca626c68bc7e7168f to start an election
I20260809 15:45:06.363585 21333 catalog_manager.cc:5495] T d203a1017c6b44d0a62bbc654ee21d8c P faa9b2a805cd43eca626c68bc7e7168f reported cstate change: term changed from 3 to 4, leader changed from 6517ff96fe0a42a2bde064dc7f1ecc97 (127.20.202.66) to faa9b2a805cd43eca626c68bc7e7168f (127.20.202.65). New cstate: current_term: 4 leader_uuid: "faa9b2a805cd43eca626c68bc7e7168f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } health_report { overall_health: UNKNOWN } } }
I20260809 15:45:06.364346 21513 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2f46e8d793294ce384acbf2f5584ecd2"
dest_uuid: "faa9b2a805cd43eca626c68bc7e7168f"
from {username='slave'} at 127.0.0.1:50158
I20260809 15:45:06.364830 21513 raft_consensus.cc:482] T 2f46e8d793294ce384acbf2f5584ecd2 P faa9b2a805cd43eca626c68bc7e7168f [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 15:45:06.365152 21513 raft_consensus.cc:3037] T 2f46e8d793294ce384acbf2f5584ecd2 P faa9b2a805cd43eca626c68bc7e7168f [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:45:06.369647 21513 raft_consensus.cc:504] T 2f46e8d793294ce384acbf2f5584ecd2 P faa9b2a805cd43eca626c68bc7e7168f [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } }
I20260809 15:45:06.371311 21513 leader_election.cc:288] T 2f46e8d793294ce384acbf2f5584ecd2 P faa9b2a805cd43eca626c68bc7e7168f [CANDIDATE]: Term 3 election: Requested vote from peers 6517ff96fe0a42a2bde064dc7f1ecc97 (127.20.202.66:42701), dee2d58fd60f4b39a553c007f76ac760 (127.20.202.67:36447)
I20260809 15:45:06.372016 21635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f46e8d793294ce384acbf2f5584ecd2" candidate_uuid: "faa9b2a805cd43eca626c68bc7e7168f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97"
I20260809 15:45:06.372226 21769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f46e8d793294ce384acbf2f5584ecd2" candidate_uuid: "faa9b2a805cd43eca626c68bc7e7168f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "dee2d58fd60f4b39a553c007f76ac760"
I20260809 15:45:06.372612 21635 raft_consensus.cc:3032] T 2f46e8d793294ce384acbf2f5584ecd2 P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 2 LEADER]: Stepping down as leader of term 2
I20260809 15:45:06.372800 21769 raft_consensus.cc:3037] T 2f46e8d793294ce384acbf2f5584ecd2 P dee2d58fd60f4b39a553c007f76ac760 [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:45:06.372991 21635 raft_consensus.cc:729] T 2f46e8d793294ce384acbf2f5584ecd2 P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 6517ff96fe0a42a2bde064dc7f1ecc97, State: Running, Role: LEADER
I20260809 15:45:06.373733 21635 consensus_queue.cc:260] T 2f46e8d793294ce384acbf2f5584ecd2 P 6517ff96fe0a42a2bde064dc7f1ecc97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } }
I20260809 15:45:06.374915 21635 raft_consensus.cc:3037] T 2f46e8d793294ce384acbf2f5584ecd2 P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:45:06.376899 21769 raft_consensus.cc:2454] T 2f46e8d793294ce384acbf2f5584ecd2 P dee2d58fd60f4b39a553c007f76ac760 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate faa9b2a805cd43eca626c68bc7e7168f in term 3.
I20260809 15:45:06.377797 21461 leader_election.cc:302] T 2f46e8d793294ce384acbf2f5584ecd2 P faa9b2a805cd43eca626c68bc7e7168f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dee2d58fd60f4b39a553c007f76ac760, faa9b2a805cd43eca626c68bc7e7168f; no voters:
I20260809 15:45:06.378382 22137 raft_consensus.cc:2781] T 2f46e8d793294ce384acbf2f5584ecd2 P faa9b2a805cd43eca626c68bc7e7168f [term 3 FOLLOWER]: Leader election won for term 3
I20260809 15:45:06.378809 22137 raft_consensus.cc:686] T 2f46e8d793294ce384acbf2f5584ecd2 P faa9b2a805cd43eca626c68bc7e7168f [term 3 LEADER]: Becoming Leader. State: Replica: faa9b2a805cd43eca626c68bc7e7168f, State: Running, Role: LEADER
I20260809 15:45:06.379940 21635 raft_consensus.cc:2454] T 2f46e8d793294ce384acbf2f5584ecd2 P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate faa9b2a805cd43eca626c68bc7e7168f in term 3.
I20260809 15:45:06.379643 22137 consensus_queue.cc:237] T 2f46e8d793294ce384acbf2f5584ecd2 P faa9b2a805cd43eca626c68bc7e7168f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } }
I20260809 15:45:06.386447 22113 raft_consensus.cc:979] T d18bbcc7ebd14b479ddacb4aad814d47 P 6517ff96fe0a42a2bde064dc7f1ecc97: : Instructing follower faa9b2a805cd43eca626c68bc7e7168f to start an election
I20260809 15:45:06.386864 22126 raft_consensus.cc:1067] T d18bbcc7ebd14b479ddacb4aad814d47 P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 2 LEADER]: Signalling peer faa9b2a805cd43eca626c68bc7e7168f to start an election
I20260809 15:45:06.387077 21333 catalog_manager.cc:5495] T 2f46e8d793294ce384acbf2f5584ecd2 P faa9b2a805cd43eca626c68bc7e7168f reported cstate change: term changed from 2 to 3, leader changed from 6517ff96fe0a42a2bde064dc7f1ecc97 (127.20.202.66) to faa9b2a805cd43eca626c68bc7e7168f (127.20.202.65). New cstate: current_term: 3 leader_uuid: "faa9b2a805cd43eca626c68bc7e7168f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } health_report { overall_health: UNKNOWN } } }
I20260809 15:45:06.388157 21513 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d18bbcc7ebd14b479ddacb4aad814d47"
dest_uuid: "faa9b2a805cd43eca626c68bc7e7168f"
from {username='slave'} at 127.0.0.1:50158
I20260809 15:45:06.388548 21513 raft_consensus.cc:482] T d18bbcc7ebd14b479ddacb4aad814d47 P faa9b2a805cd43eca626c68bc7e7168f [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 15:45:06.388803 21513 raft_consensus.cc:3037] T d18bbcc7ebd14b479ddacb4aad814d47 P faa9b2a805cd43eca626c68bc7e7168f [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:45:06.393785 21513 raft_consensus.cc:504] T d18bbcc7ebd14b479ddacb4aad814d47 P faa9b2a805cd43eca626c68bc7e7168f [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } }
I20260809 15:45:06.395166 21513 leader_election.cc:288] T d18bbcc7ebd14b479ddacb4aad814d47 P faa9b2a805cd43eca626c68bc7e7168f [CANDIDATE]: Term 3 election: Requested vote from peers 6517ff96fe0a42a2bde064dc7f1ecc97 (127.20.202.66:42701), dee2d58fd60f4b39a553c007f76ac760 (127.20.202.67:36447)
I20260809 15:45:06.395857 21635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d18bbcc7ebd14b479ddacb4aad814d47" candidate_uuid: "faa9b2a805cd43eca626c68bc7e7168f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97"
I20260809 15:45:06.396023 21769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d18bbcc7ebd14b479ddacb4aad814d47" candidate_uuid: "faa9b2a805cd43eca626c68bc7e7168f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "dee2d58fd60f4b39a553c007f76ac760"
I20260809 15:45:06.396353 21635 raft_consensus.cc:3032] T d18bbcc7ebd14b479ddacb4aad814d47 P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 2 LEADER]: Stepping down as leader of term 2
I20260809 15:45:06.396464 21769 raft_consensus.cc:3037] T d18bbcc7ebd14b479ddacb4aad814d47 P dee2d58fd60f4b39a553c007f76ac760 [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:45:06.396646 21635 raft_consensus.cc:729] T d18bbcc7ebd14b479ddacb4aad814d47 P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 6517ff96fe0a42a2bde064dc7f1ecc97, State: Running, Role: LEADER
I20260809 15:45:06.397353 21635 consensus_queue.cc:260] T d18bbcc7ebd14b479ddacb4aad814d47 P 6517ff96fe0a42a2bde064dc7f1ecc97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } }
I20260809 15:45:06.398439 21635 raft_consensus.cc:3037] T d18bbcc7ebd14b479ddacb4aad814d47 P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:45:06.401674 21769 raft_consensus.cc:2454] T d18bbcc7ebd14b479ddacb4aad814d47 P dee2d58fd60f4b39a553c007f76ac760 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate faa9b2a805cd43eca626c68bc7e7168f in term 3.
I20260809 15:45:06.402590 21461 leader_election.cc:302] T d18bbcc7ebd14b479ddacb4aad814d47 P faa9b2a805cd43eca626c68bc7e7168f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dee2d58fd60f4b39a553c007f76ac760, faa9b2a805cd43eca626c68bc7e7168f; no voters:
I20260809 15:45:06.403301 22137 raft_consensus.cc:2781] T d18bbcc7ebd14b479ddacb4aad814d47 P faa9b2a805cd43eca626c68bc7e7168f [term 3 FOLLOWER]: Leader election won for term 3
I20260809 15:45:06.403674 22137 raft_consensus.cc:686] T d18bbcc7ebd14b479ddacb4aad814d47 P faa9b2a805cd43eca626c68bc7e7168f [term 3 LEADER]: Becoming Leader. State: Replica: faa9b2a805cd43eca626c68bc7e7168f, State: Running, Role: LEADER
I20260809 15:45:06.403746 21635 raft_consensus.cc:2454] T d18bbcc7ebd14b479ddacb4aad814d47 P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate faa9b2a805cd43eca626c68bc7e7168f in term 3.
I20260809 15:45:06.404497 22137 consensus_queue.cc:237] T d18bbcc7ebd14b479ddacb4aad814d47 P faa9b2a805cd43eca626c68bc7e7168f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } }
I20260809 15:45:06.410684 21333 catalog_manager.cc:5495] T d18bbcc7ebd14b479ddacb4aad814d47 P faa9b2a805cd43eca626c68bc7e7168f reported cstate change: term changed from 2 to 3, leader changed from 6517ff96fe0a42a2bde064dc7f1ecc97 (127.20.202.66) to faa9b2a805cd43eca626c68bc7e7168f (127.20.202.65). New cstate: current_term: 3 leader_uuid: "faa9b2a805cd43eca626c68bc7e7168f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 } health_report { overall_health: UNKNOWN } } }
I20260809 15:45:06.498668 21513 raft_consensus.cc:1261] T ab976a6064754a689b7c88c5fa968b86 P faa9b2a805cd43eca626c68bc7e7168f [term 3 FOLLOWER]: Refusing update from remote peer dee2d58fd60f4b39a553c007f76ac760: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260809 15:45:06.499677 22085 consensus_queue.cc:1035] T ab976a6064754a689b7c88c5fa968b86 P dee2d58fd60f4b39a553c007f76ac760 [LEADER]: Connected to new peer: Peer: permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 15:45:06.509891 21635 raft_consensus.cc:1261] T ab976a6064754a689b7c88c5fa968b86 P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 3 FOLLOWER]: Refusing update from remote peer dee2d58fd60f4b39a553c007f76ac760: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260809 15:45:06.511152 22146 consensus_queue.cc:1035] T ab976a6064754a689b7c88c5fa968b86 P dee2d58fd60f4b39a553c007f76ac760 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260809 15:45:06.733125 21635 raft_consensus.cc:1261] T 724982e80d3b455a91f8a564d374a65e P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 2 FOLLOWER]: Refusing update from remote peer dee2d58fd60f4b39a553c007f76ac760: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 15:45:06.734087 22146 consensus_queue.cc:1035] T 724982e80d3b455a91f8a564d374a65e P dee2d58fd60f4b39a553c007f76ac760 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 15:45:06.741743 21513 raft_consensus.cc:1261] T 724982e80d3b455a91f8a564d374a65e P faa9b2a805cd43eca626c68bc7e7168f [term 2 FOLLOWER]: Refusing update from remote peer dee2d58fd60f4b39a553c007f76ac760: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 15:45:06.743175 22154 consensus_queue.cc:1035] T 724982e80d3b455a91f8a564d374a65e P dee2d58fd60f4b39a553c007f76ac760 [LEADER]: Connected to new peer: Peer: permanent_uuid: "faa9b2a805cd43eca626c68bc7e7168f" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 37685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260809 15:45:06.786621 21769 raft_consensus.cc:1261] T 2f46e8d793294ce384acbf2f5584ecd2 P dee2d58fd60f4b39a553c007f76ac760 [term 3 FOLLOWER]: Refusing update from remote peer faa9b2a805cd43eca626c68bc7e7168f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260809 15:45:06.787627 22137 consensus_queue.cc:1035] T 2f46e8d793294ce384acbf2f5584ecd2 P faa9b2a805cd43eca626c68bc7e7168f [LEADER]: Connected to new peer: Peer: permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 15:45:06.796437 21635 raft_consensus.cc:1261] T 2f46e8d793294ce384acbf2f5584ecd2 P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 3 FOLLOWER]: Refusing update from remote peer faa9b2a805cd43eca626c68bc7e7168f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260809 15:45:06.797528 22137 consensus_queue.cc:1035] T 2f46e8d793294ce384acbf2f5584ecd2 P faa9b2a805cd43eca626c68bc7e7168f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 15:45:06.831836 21635 raft_consensus.cc:1261] T d203a1017c6b44d0a62bbc654ee21d8c P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 4 FOLLOWER]: Refusing update from remote peer faa9b2a805cd43eca626c68bc7e7168f: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260809 15:45:06.832706 22137 consensus_queue.cc:1035] T d203a1017c6b44d0a62bbc654ee21d8c P faa9b2a805cd43eca626c68bc7e7168f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260809 15:45:06.840724 21770 raft_consensus.cc:1261] T d203a1017c6b44d0a62bbc654ee21d8c P dee2d58fd60f4b39a553c007f76ac760 [term 4 FOLLOWER]: Refusing update from remote peer faa9b2a805cd43eca626c68bc7e7168f: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260809 15:45:06.841958 22163 consensus_queue.cc:1035] T d203a1017c6b44d0a62bbc654ee21d8c P faa9b2a805cd43eca626c68bc7e7168f [LEADER]: Connected to new peer: Peer: permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260809 15:45:06.978551 21635 raft_consensus.cc:1261] T d18bbcc7ebd14b479ddacb4aad814d47 P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 3 FOLLOWER]: Refusing update from remote peer faa9b2a805cd43eca626c68bc7e7168f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260809 15:45:06.979615 22163 consensus_queue.cc:1035] T d18bbcc7ebd14b479ddacb4aad814d47 P faa9b2a805cd43eca626c68bc7e7168f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6517ff96fe0a42a2bde064dc7f1ecc97" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 42701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 15:45:06.988200 21770 raft_consensus.cc:1261] T d18bbcc7ebd14b479ddacb4aad814d47 P dee2d58fd60f4b39a553c007f76ac760 [term 3 FOLLOWER]: Refusing update from remote peer faa9b2a805cd43eca626c68bc7e7168f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260809 15:45:06.989796 22137 consensus_queue.cc:1035] T d18bbcc7ebd14b479ddacb4aad814d47 P faa9b2a805cd43eca626c68bc7e7168f [LEADER]: Connected to new peer: Peer: permanent_uuid: "dee2d58fd60f4b39a553c007f76ac760" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 36447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 15:45:07.020823 21289 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 15:45:07.025498 21289 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260809 15:45:07.025837 21289 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 15:45:08.026552 21289 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 15:45:08.031193 21289 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260809 15:45:08.031575 21289 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 15:45:09.032212 21289 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 15:45:09.036667 21289 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260809 15:45:09.037012 21289 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 15:45:10.037678 21289 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 15:45:10.042315 21289 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260809 15:45:10.042685 21289 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 15:45:11.043437 21289 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 15:45:11.048158 21289 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260809 15:45:11.048506 21289 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 15:45:12.049168 21289 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 15:45:12.053714 21289 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260809 15:45:12.054066 21289 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 15:45:13.054730 21289 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 15:45:13.059242 21289 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260809 15:45:13.059605 21289 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 15:45:14.060259 21289 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 15:45:14.065127 21289 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260809 15:45:14.065542 21289 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 15:45:15.066272 21289 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 15:45:15.070679 21289 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260809 15:45:15.071024 21289 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 15:45:16.094915 21289 auto_leader_rebalancer-test.cc:314] The leader distribution is
I20260809 15:45:16.095479 21289 tablet_server.cc:178] TabletServer@127.20.202.65:0 shutting down...
W20260809 15:45:16.102187 21574 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260809 15:45:16.106765 21574 consensus_peers.cc:487] T 8d9c4e17e76743439089a2a53c45d01d P 6517ff96fe0a42a2bde064dc7f1ecc97 -> Peer faa9b2a805cd43eca626c68bc7e7168f (127.20.202.65:37685): Couldn't send request to peer faa9b2a805cd43eca626c68bc7e7168f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260809 15:45:16.130095 21289 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260809 15:45:16.130671 21289 tablet_replica.cc:330] T 2f46e8d793294ce384acbf2f5584ecd2 P faa9b2a805cd43eca626c68bc7e7168f: stopping tablet replica
I20260809 15:45:16.131402 21289 raft_consensus.cc:2229] T 2f46e8d793294ce384acbf2f5584ecd2 P faa9b2a805cd43eca626c68bc7e7168f [term 3 LEADER]: Raft consensus shutting down.
I20260809 15:45:16.132138 21289 raft_consensus.cc:2258] T 2f46e8d793294ce384acbf2f5584ecd2 P faa9b2a805cd43eca626c68bc7e7168f [term 3 FOLLOWER]: Raft consensus is shut down!
I20260809 15:45:16.134121 21289 tablet_replica.cc:330] T 8d9c4e17e76743439089a2a53c45d01d P faa9b2a805cd43eca626c68bc7e7168f: stopping tablet replica
I20260809 15:45:16.134598 21289 raft_consensus.cc:2229] T 8d9c4e17e76743439089a2a53c45d01d P faa9b2a805cd43eca626c68bc7e7168f [term 3 FOLLOWER]: Raft consensus shutting down.
I20260809 15:45:16.134984 21289 raft_consensus.cc:2258] T 8d9c4e17e76743439089a2a53c45d01d P faa9b2a805cd43eca626c68bc7e7168f [term 3 FOLLOWER]: Raft consensus is shut down!
I20260809 15:45:16.136509 21289 tablet_replica.cc:330] T 6dd9896939284d9cb2c03e9c21a92dfa P faa9b2a805cd43eca626c68bc7e7168f: stopping tablet replica
I20260809 15:45:16.136930 21289 raft_consensus.cc:2229] T 6dd9896939284d9cb2c03e9c21a92dfa P faa9b2a805cd43eca626c68bc7e7168f [term 2 FOLLOWER]: Raft consensus shutting down.
I20260809 15:45:16.137319 21289 raft_consensus.cc:2258] T 6dd9896939284d9cb2c03e9c21a92dfa P faa9b2a805cd43eca626c68bc7e7168f [term 2 FOLLOWER]: Raft consensus is shut down!
I20260809 15:45:16.138840 21289 tablet_replica.cc:330] T 724982e80d3b455a91f8a564d374a65e P faa9b2a805cd43eca626c68bc7e7168f: stopping tablet replica
I20260809 15:45:16.139420 21289 raft_consensus.cc:2229] T 724982e80d3b455a91f8a564d374a65e P faa9b2a805cd43eca626c68bc7e7168f [term 2 FOLLOWER]: Raft consensus shutting down.
I20260809 15:45:16.139762 21289 raft_consensus.cc:2258] T 724982e80d3b455a91f8a564d374a65e P faa9b2a805cd43eca626c68bc7e7168f [term 2 FOLLOWER]: Raft consensus is shut down!
I20260809 15:45:16.141350 21289 tablet_replica.cc:330] T d203a1017c6b44d0a62bbc654ee21d8c P faa9b2a805cd43eca626c68bc7e7168f: stopping tablet replica
I20260809 15:45:16.141799 21289 raft_consensus.cc:2229] T d203a1017c6b44d0a62bbc654ee21d8c P faa9b2a805cd43eca626c68bc7e7168f [term 4 LEADER]: Raft consensus shutting down.
I20260809 15:45:16.142441 21289 raft_consensus.cc:2258] T d203a1017c6b44d0a62bbc654ee21d8c P faa9b2a805cd43eca626c68bc7e7168f [term 4 FOLLOWER]: Raft consensus is shut down!
I20260809 15:45:16.143940 21289 tablet_replica.cc:330] T ae87ef2626584e6bacd9f1af2c8991d8 P faa9b2a805cd43eca626c68bc7e7168f: stopping tablet replica
I20260809 15:45:16.144363 21289 raft_consensus.cc:2229] T ae87ef2626584e6bacd9f1af2c8991d8 P faa9b2a805cd43eca626c68bc7e7168f [term 4 FOLLOWER]: Raft consensus shutting down.
I20260809 15:45:16.144711 21289 raft_consensus.cc:2258] T ae87ef2626584e6bacd9f1af2c8991d8 P faa9b2a805cd43eca626c68bc7e7168f [term 4 FOLLOWER]: Raft consensus is shut down!
I20260809 15:45:16.146076 21289 tablet_replica.cc:330] T d18bbcc7ebd14b479ddacb4aad814d47 P faa9b2a805cd43eca626c68bc7e7168f: stopping tablet replica
I20260809 15:45:16.146497 21289 raft_consensus.cc:2229] T d18bbcc7ebd14b479ddacb4aad814d47 P faa9b2a805cd43eca626c68bc7e7168f [term 3 LEADER]: Raft consensus shutting down.
I20260809 15:45:16.147114 21289 raft_consensus.cc:2258] T d18bbcc7ebd14b479ddacb4aad814d47 P faa9b2a805cd43eca626c68bc7e7168f [term 3 FOLLOWER]: Raft consensus is shut down!
I20260809 15:45:16.148571 21289 tablet_replica.cc:330] T 4fe458f602674472bedadfdd3c7ee1e6 P faa9b2a805cd43eca626c68bc7e7168f: stopping tablet replica
I20260809 15:45:16.148954 21289 raft_consensus.cc:2229] T 4fe458f602674472bedadfdd3c7ee1e6 P faa9b2a805cd43eca626c68bc7e7168f [term 2 FOLLOWER]: Raft consensus shutting down.
I20260809 15:45:16.149279 21289 raft_consensus.cc:2258] T 4fe458f602674472bedadfdd3c7ee1e6 P faa9b2a805cd43eca626c68bc7e7168f [term 2 FOLLOWER]: Raft consensus is shut down!
I20260809 15:45:16.150667 21289 tablet_replica.cc:330] T ab976a6064754a689b7c88c5fa968b86 P faa9b2a805cd43eca626c68bc7e7168f: stopping tablet replica
I20260809 15:45:16.151036 21289 raft_consensus.cc:2229] T ab976a6064754a689b7c88c5fa968b86 P faa9b2a805cd43eca626c68bc7e7168f [term 3 FOLLOWER]: Raft consensus shutting down.
I20260809 15:45:16.151386 21289 raft_consensus.cc:2258] T ab976a6064754a689b7c88c5fa968b86 P faa9b2a805cd43eca626c68bc7e7168f [term 3 FOLLOWER]: Raft consensus is shut down!
W20260809 15:45:16.220028 21574 consensus_peers.cc:487] T ae87ef2626584e6bacd9f1af2c8991d8 P 6517ff96fe0a42a2bde064dc7f1ecc97 -> Peer faa9b2a805cd43eca626c68bc7e7168f (127.20.202.65:37685): Couldn't send request to peer faa9b2a805cd43eca626c68bc7e7168f. Status: Network error: Client connection negotiation failed: client connection to 127.20.202.65:37685: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260809 15:45:16.230232 21289 tablet_server.cc:194] TabletServer@127.20.202.65:0 shutdown complete.
W20260809 15:45:16.257907 21709 consensus_peers.cc:487] T ab976a6064754a689b7c88c5fa968b86 P dee2d58fd60f4b39a553c007f76ac760 -> Peer faa9b2a805cd43eca626c68bc7e7168f (127.20.202.65:37685): Couldn't send request to peer faa9b2a805cd43eca626c68bc7e7168f. Status: Network error: Client connection negotiation failed: client connection to 127.20.202.65:37685: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260809 15:45:16.264281 21289 tablet_server.cc:178] TabletServer@127.20.202.66:0 shutting down...
I20260809 15:45:16.290916 21289 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260809 15:45:16.292229 21289 tablet_replica.cc:330] T 6dd9896939284d9cb2c03e9c21a92dfa P 6517ff96fe0a42a2bde064dc7f1ecc97: stopping tablet replica
I20260809 15:45:16.293960 21289 raft_consensus.cc:2229] T 6dd9896939284d9cb2c03e9c21a92dfa P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260809 15:45:16.295019 21289 raft_consensus.cc:2258] T 6dd9896939284d9cb2c03e9c21a92dfa P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 2 FOLLOWER]: Raft consensus is shut down!
W20260809 15:45:16.297406 21710 consensus_peers.cc:487] T 6dd9896939284d9cb2c03e9c21a92dfa P dee2d58fd60f4b39a553c007f76ac760 -> Peer 6517ff96fe0a42a2bde064dc7f1ecc97 (127.20.202.66:42701): Couldn't send request to peer 6517ff96fe0a42a2bde064dc7f1ecc97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260809 15:45:16.298163 21289 tablet_replica.cc:330] T 8d9c4e17e76743439089a2a53c45d01d P 6517ff96fe0a42a2bde064dc7f1ecc97: stopping tablet replica
I20260809 15:45:16.298640 21289 raft_consensus.cc:2229] T 8d9c4e17e76743439089a2a53c45d01d P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 3 LEADER]: Raft consensus shutting down.
I20260809 15:45:16.299789 21289 raft_consensus.cc:2258] T 8d9c4e17e76743439089a2a53c45d01d P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260809 15:45:16.301587 21289 tablet_replica.cc:330] T 2f46e8d793294ce384acbf2f5584ecd2 P 6517ff96fe0a42a2bde064dc7f1ecc97: stopping tablet replica
I20260809 15:45:16.302284 21289 raft_consensus.cc:2229] T 2f46e8d793294ce384acbf2f5584ecd2 P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260809 15:45:16.302914 21289 raft_consensus.cc:2258] T 2f46e8d793294ce384acbf2f5584ecd2 P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 3 FOLLOWER]: Raft consensus is shut down!
W20260809 15:45:16.303159 21574 consensus_peers.cc:487] T 4fe458f602674472bedadfdd3c7ee1e6 P 6517ff96fe0a42a2bde064dc7f1ecc97 -> Peer faa9b2a805cd43eca626c68bc7e7168f (127.20.202.65:37685): Couldn't send request to peer faa9b2a805cd43eca626c68bc7e7168f. Status: Network error: Client connection negotiation failed: client connection to 127.20.202.65:37685: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260809 15:45:16.304769 21289 tablet_replica.cc:330] T 724982e80d3b455a91f8a564d374a65e P 6517ff96fe0a42a2bde064dc7f1ecc97: stopping tablet replica
I20260809 15:45:16.305296 21289 raft_consensus.cc:2229] T 724982e80d3b455a91f8a564d374a65e P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260809 15:45:16.305754 21289 raft_consensus.cc:2258] T 724982e80d3b455a91f8a564d374a65e P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260809 15:45:16.307382 21289 tablet_replica.cc:330] T ab976a6064754a689b7c88c5fa968b86 P 6517ff96fe0a42a2bde064dc7f1ecc97: stopping tablet replica
I20260809 15:45:16.307824 21289 raft_consensus.cc:2229] T ab976a6064754a689b7c88c5fa968b86 P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260809 15:45:16.308269 21289 raft_consensus.cc:2258] T ab976a6064754a689b7c88c5fa968b86 P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260809 15:45:16.309729 21289 tablet_replica.cc:330] T d18bbcc7ebd14b479ddacb4aad814d47 P 6517ff96fe0a42a2bde064dc7f1ecc97: stopping tablet replica
I20260809 15:45:16.310200 21289 raft_consensus.cc:2229] T d18bbcc7ebd14b479ddacb4aad814d47 P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260809 15:45:16.310617 21289 raft_consensus.cc:2258] T d18bbcc7ebd14b479ddacb4aad814d47 P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260809 15:45:16.312219 21289 tablet_replica.cc:330] T 4fe458f602674472bedadfdd3c7ee1e6 P 6517ff96fe0a42a2bde064dc7f1ecc97: stopping tablet replica
I20260809 15:45:16.312642 21289 raft_consensus.cc:2229] T 4fe458f602674472bedadfdd3c7ee1e6 P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 2 LEADER]: Raft consensus shutting down.
I20260809 15:45:16.313508 21289 raft_consensus.cc:2258] T 4fe458f602674472bedadfdd3c7ee1e6 P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260809 15:45:16.315088 21289 tablet_replica.cc:330] T d203a1017c6b44d0a62bbc654ee21d8c P 6517ff96fe0a42a2bde064dc7f1ecc97: stopping tablet replica
I20260809 15:45:16.315567 21289 raft_consensus.cc:2229] T d203a1017c6b44d0a62bbc654ee21d8c P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260809 15:45:16.316018 21289 raft_consensus.cc:2258] T d203a1017c6b44d0a62bbc654ee21d8c P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260809 15:45:16.317524 21289 tablet_replica.cc:330] T ae87ef2626584e6bacd9f1af2c8991d8 P 6517ff96fe0a42a2bde064dc7f1ecc97: stopping tablet replica
I20260809 15:45:16.317938 21289 raft_consensus.cc:2229] T ae87ef2626584e6bacd9f1af2c8991d8 P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 4 LEADER]: Raft consensus shutting down.
I20260809 15:45:16.318905 21289 raft_consensus.cc:2258] T ae87ef2626584e6bacd9f1af2c8991d8 P 6517ff96fe0a42a2bde064dc7f1ecc97 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260809 15:45:16.345908 21289 tablet_server.cc:194] TabletServer@127.20.202.66:0 shutdown complete.
W20260809 15:45:16.352741 21710 consensus_peers.cc:487] T ab976a6064754a689b7c88c5fa968b86 P dee2d58fd60f4b39a553c007f76ac760 -> Peer 6517ff96fe0a42a2bde064dc7f1ecc97 (127.20.202.66:42701): Couldn't send request to peer 6517ff96fe0a42a2bde064dc7f1ecc97. Status: Network error: Client connection negotiation failed: client connection to 127.20.202.66:42701: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260809 15:45:16.363454 21289 tablet_server.cc:178] TabletServer@127.20.202.67:0 shutting down...
I20260809 15:45:16.386710 21289 ts_tablet_manager.cc:1500] Shutting down tablet manager...
W20260809 15:45:16.387410 21709 consensus_peers.cc:487] T 6dd9896939284d9cb2c03e9c21a92dfa P dee2d58fd60f4b39a553c007f76ac760 -> Peer faa9b2a805cd43eca626c68bc7e7168f (127.20.202.65:37685): Couldn't send request to peer faa9b2a805cd43eca626c68bc7e7168f. Status: Network error: Client connection negotiation failed: client connection to 127.20.202.65:37685: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260809 15:45:16.387691 21289 tablet_replica.cc:330] T 724982e80d3b455a91f8a564d374a65e P dee2d58fd60f4b39a553c007f76ac760: stopping tablet replica
I20260809 15:45:16.388480 21289 raft_consensus.cc:2229] T 724982e80d3b455a91f8a564d374a65e P dee2d58fd60f4b39a553c007f76ac760 [term 2 LEADER]: Raft consensus shutting down.
I20260809 15:45:16.389547 21289 raft_consensus.cc:2258] T 724982e80d3b455a91f8a564d374a65e P dee2d58fd60f4b39a553c007f76ac760 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260809 15:45:16.391229 21289 tablet_replica.cc:330] T 4fe458f602674472bedadfdd3c7ee1e6 P dee2d58fd60f4b39a553c007f76ac760: stopping tablet replica
I20260809 15:45:16.391891 21289 raft_consensus.cc:2229] T 4fe458f602674472bedadfdd3c7ee1e6 P dee2d58fd60f4b39a553c007f76ac760 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260809 15:45:16.392318 21289 raft_consensus.cc:2258] T 4fe458f602674472bedadfdd3c7ee1e6 P dee2d58fd60f4b39a553c007f76ac760 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260809 15:45:16.393855 21289 tablet_replica.cc:330] T d18bbcc7ebd14b479ddacb4aad814d47 P dee2d58fd60f4b39a553c007f76ac760: stopping tablet replica
I20260809 15:45:16.394384 21289 raft_consensus.cc:2229] T d18bbcc7ebd14b479ddacb4aad814d47 P dee2d58fd60f4b39a553c007f76ac760 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260809 15:45:16.394811 21289 raft_consensus.cc:2258] T d18bbcc7ebd14b479ddacb4aad814d47 P dee2d58fd60f4b39a553c007f76ac760 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260809 15:45:16.396283 21289 tablet_replica.cc:330] T 2f46e8d793294ce384acbf2f5584ecd2 P dee2d58fd60f4b39a553c007f76ac760: stopping tablet replica
I20260809 15:45:16.396711 21289 raft_consensus.cc:2229] T 2f46e8d793294ce384acbf2f5584ecd2 P dee2d58fd60f4b39a553c007f76ac760 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260809 15:45:16.397185 21289 raft_consensus.cc:2258] T 2f46e8d793294ce384acbf2f5584ecd2 P dee2d58fd60f4b39a553c007f76ac760 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260809 15:45:16.398757 21289 tablet_replica.cc:330] T 8d9c4e17e76743439089a2a53c45d01d P dee2d58fd60f4b39a553c007f76ac760: stopping tablet replica
I20260809 15:45:16.399187 21289 raft_consensus.cc:2229] T 8d9c4e17e76743439089a2a53c45d01d P dee2d58fd60f4b39a553c007f76ac760 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260809 15:45:16.399689 21289 raft_consensus.cc:2258] T 8d9c4e17e76743439089a2a53c45d01d P dee2d58fd60f4b39a553c007f76ac760 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260809 15:45:16.401199 21289 tablet_replica.cc:330] T ab976a6064754a689b7c88c5fa968b86 P dee2d58fd60f4b39a553c007f76ac760: stopping tablet replica
I20260809 15:45:16.401615 21289 raft_consensus.cc:2229] T ab976a6064754a689b7c88c5fa968b86 P dee2d58fd60f4b39a553c007f76ac760 [term 3 LEADER]: Raft consensus shutting down.
I20260809 15:45:16.402405 21289 raft_consensus.cc:2258] T ab976a6064754a689b7c88c5fa968b86 P dee2d58fd60f4b39a553c007f76ac760 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260809 15:45:16.404006 21289 tablet_replica.cc:330] T d203a1017c6b44d0a62bbc654ee21d8c P dee2d58fd60f4b39a553c007f76ac760: stopping tablet replica
I20260809 15:45:16.404439 21289 raft_consensus.cc:2229] T d203a1017c6b44d0a62bbc654ee21d8c P dee2d58fd60f4b39a553c007f76ac760 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260809 15:45:16.404870 21289 raft_consensus.cc:2258] T d203a1017c6b44d0a62bbc654ee21d8c P dee2d58fd60f4b39a553c007f76ac760 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260809 15:45:16.406394 21289 tablet_replica.cc:330] T ae87ef2626584e6bacd9f1af2c8991d8 P dee2d58fd60f4b39a553c007f76ac760: stopping tablet replica
I20260809 15:45:16.406812 21289 raft_consensus.cc:2229] T ae87ef2626584e6bacd9f1af2c8991d8 P dee2d58fd60f4b39a553c007f76ac760 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260809 15:45:16.407258 21289 raft_consensus.cc:2258] T ae87ef2626584e6bacd9f1af2c8991d8 P dee2d58fd60f4b39a553c007f76ac760 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260809 15:45:16.408763 21289 tablet_replica.cc:330] T 6dd9896939284d9cb2c03e9c21a92dfa P dee2d58fd60f4b39a553c007f76ac760: stopping tablet replica
I20260809 15:45:16.409150 21289 raft_consensus.cc:2229] T 6dd9896939284d9cb2c03e9c21a92dfa P dee2d58fd60f4b39a553c007f76ac760 [term 2 LEADER]: Raft consensus shutting down.
I20260809 15:45:16.409812 21289 raft_consensus.cc:2258] T 6dd9896939284d9cb2c03e9c21a92dfa P dee2d58fd60f4b39a553c007f76ac760 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260809 15:45:16.436654 21289 tablet_server.cc:194] TabletServer@127.20.202.67:0 shutdown complete.
I20260809 15:45:16.452389 21289 master.cc:537] Master@127.20.202.126:37791 shutting down...
I20260809 15:45:16.739836 21289 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 5f69f1903e1d4f6f932244dd3880569c [term 1 LEADER]: Raft consensus shutting down.
I20260809 15:45:16.740473 21289 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 5f69f1903e1d4f6f932244dd3880569c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 15:45:16.740810 21289 tablet_replica.cc:330] T 00000000000000000000000000000000 P 5f69f1903e1d4f6f932244dd3880569c: stopping tablet replica
I20260809 15:45:16.758111 21289 master.cc:552] Master@127.20.202.126:37791 shutdown complete.
6517ff96fe0a42a2bde064dc7f1ecc97 4
dee2d58fd60f4b39a553c007f76ac760 1
faa9b2a805cd43eca626c68bc7e7168f 4
6517ff96fe0a42a2bde064dc7f1ecc97 3
dee2d58fd60f4b39a553c007f76ac760 3
faa9b2a805cd43eca626c68bc7e7168f 3
6517ff96fe0a42a2bde064dc7f1ecc97 8
dee2d58fd60f4b39a553c007f76ac760 1
faa9b2a805cd43eca626c68bc7e7168f 0
6517ff96fe0a42a2bde064dc7f1ecc97 3
dee2d58fd60f4b39a553c007f76ac760 3
faa9b2a805cd43eca626c68bc7e7168f 3
[ OK ] LeaderRebalancerTest.FunctionalTestForDivided (30630 ms)
[ RUN ] LeaderRebalancerTest.FunctionalTestForNotDivided
I20260809 15:45:16.797255 21289 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.20.202.126:44859
I20260809 15:45:16.798204 21289 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:45:16.803614 22210 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:45:16.803839 22208 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:45:16.804194 22215 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:45:16.804982 21289 server_base.cc:1034] running on GCE node
I20260809 15:45:16.805807 21289 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260809 15:45:16.805986 21289 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260809 15:45:16.806119 21289 hybrid_clock.cc:648] HybridClock initialized: now 1786290316806105 us; error 0 us; skew 500 ppm
I20260809 15:45:16.806558 21289 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 15:45:16.808765 21289 webserver.cc:460] Webserver started at http://127.20.202.126:45077/ using document root <none> and password file <none>
I20260809 15:45:16.809216 21289 fs_manager.cc:359] Metadata directory not provided
I20260809 15:45:16.809382 21289 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:45:16.809645 21289 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 15:45:16.810808 21289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj2AJ_N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786290285951082-21289-0/minicluster-data/master-0-root/instance:
uuid: "b6998a43cbdd4d0eba73e55cb87cf709"
format_stamp: "Formatted at 2026-08-09 15:45:16 on dist-test-slave-gjrz"
I20260809 15:45:16.814855 21289 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260809 15:45:16.817741 22220 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:45:16.818361 21289 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 15:45:16.818606 21289 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj2AJ_N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786290285951082-21289-0/minicluster-data/master-0-root
uuid: "b6998a43cbdd4d0eba73e55cb87cf709"
format_stamp: "Formatted at 2026-08-09 15:45:16 on dist-test-slave-gjrz"
I20260809 15:45:16.818846 21289 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj2AJ_N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786290285951082-21289-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskj2AJ_N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786290285951082-21289-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskj2AJ_N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786290285951082-21289-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:45:16.842001 21289 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:45:16.843060 21289 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:45:16.878311 21289 rpc_server.cc:307] RPC server started. Bound to: 127.20.202.126:44859
I20260809 15:45:16.878435 22300 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.202.126:44859 every 8 connection(s)
I20260809 15:45:16.881902 22302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:16.890627 22302 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6998a43cbdd4d0eba73e55cb87cf709: Bootstrap starting.
I20260809 15:45:16.894646 22302 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b6998a43cbdd4d0eba73e55cb87cf709: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:16.898344 22302 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6998a43cbdd4d0eba73e55cb87cf709: No bootstrap required, opened a new log
I20260809 15:45:16.900187 22302 raft_consensus.cc:348] T 00000000000000000000000000000000 P b6998a43cbdd4d0eba73e55cb87cf709 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6998a43cbdd4d0eba73e55cb87cf709" member_type: VOTER }
I20260809 15:45:16.900614 22302 raft_consensus.cc:374] T 00000000000000000000000000000000 P b6998a43cbdd4d0eba73e55cb87cf709 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:16.900784 22302 raft_consensus.cc:729] T 00000000000000000000000000000000 P b6998a43cbdd4d0eba73e55cb87cf709 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6998a43cbdd4d0eba73e55cb87cf709, State: Initialized, Role: FOLLOWER
I20260809 15:45:16.901281 22302 consensus_queue.cc:260] T 00000000000000000000000000000000 P b6998a43cbdd4d0eba73e55cb87cf709 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6998a43cbdd4d0eba73e55cb87cf709" member_type: VOTER }
I20260809 15:45:16.901702 22302 raft_consensus.cc:388] T 00000000000000000000000000000000 P b6998a43cbdd4d0eba73e55cb87cf709 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 15:45:16.901901 22302 raft_consensus.cc:482] T 00000000000000000000000000000000 P b6998a43cbdd4d0eba73e55cb87cf709 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 15:45:16.902123 22302 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b6998a43cbdd4d0eba73e55cb87cf709 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:16.905429 22302 raft_consensus.cc:504] T 00000000000000000000000000000000 P b6998a43cbdd4d0eba73e55cb87cf709 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6998a43cbdd4d0eba73e55cb87cf709" member_type: VOTER }
I20260809 15:45:16.905864 22302 leader_election.cc:302] T 00000000000000000000000000000000 P b6998a43cbdd4d0eba73e55cb87cf709 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b6998a43cbdd4d0eba73e55cb87cf709; no voters:
I20260809 15:45:16.906852 22302 leader_election.cc:288] T 00000000000000000000000000000000 P b6998a43cbdd4d0eba73e55cb87cf709 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 15:45:16.907104 22306 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b6998a43cbdd4d0eba73e55cb87cf709 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:45:16.908321 22306 raft_consensus.cc:686] T 00000000000000000000000000000000 P b6998a43cbdd4d0eba73e55cb87cf709 [term 1 LEADER]: Becoming Leader. State: Replica: b6998a43cbdd4d0eba73e55cb87cf709, State: Running, Role: LEADER
I20260809 15:45:16.908944 22306 consensus_queue.cc:237] T 00000000000000000000000000000000 P b6998a43cbdd4d0eba73e55cb87cf709 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6998a43cbdd4d0eba73e55cb87cf709" member_type: VOTER }
I20260809 15:45:16.909672 22302 sys_catalog.cc:564] T 00000000000000000000000000000000 P b6998a43cbdd4d0eba73e55cb87cf709 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 15:45:16.915232 22307 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6998a43cbdd4d0eba73e55cb87cf709 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b6998a43cbdd4d0eba73e55cb87cf709" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6998a43cbdd4d0eba73e55cb87cf709" member_type: VOTER } }
I20260809 15:45:16.915315 22308 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6998a43cbdd4d0eba73e55cb87cf709 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b6998a43cbdd4d0eba73e55cb87cf709. Latest consensus state: current_term: 1 leader_uuid: "b6998a43cbdd4d0eba73e55cb87cf709" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6998a43cbdd4d0eba73e55cb87cf709" member_type: VOTER } }
I20260809 15:45:16.916015 22307 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6998a43cbdd4d0eba73e55cb87cf709 [sys.catalog]: This master's current role is: LEADER
I20260809 15:45:16.916049 22308 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6998a43cbdd4d0eba73e55cb87cf709 [sys.catalog]: This master's current role is: LEADER
I20260809 15:45:16.918088 22320 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 15:45:16.922916 22320 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 15:45:16.924883 21289 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260809 15:45:16.930843 22320 catalog_manager.cc:1347] Generated new cluster ID: 78d7f11ebcb747ff9938cca0acd45a68
I20260809 15:45:16.931088 22320 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 15:45:16.945317 22320 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 15:45:16.946812 22320 catalog_manager.cc:1504] Loading token signing keys...
I20260809 15:45:16.960443 22320 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b6998a43cbdd4d0eba73e55cb87cf709: Generated new TSK 0
I20260809 15:45:16.960984 22320 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 15:45:16.990607 21289 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:45:16.996037 22336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:45:16.996522 22345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:45:16.997996 22348 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:45:16.998523 21289 server_base.cc:1034] running on GCE node
I20260809 15:45:16.999354 21289 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260809 15:45:16.999529 21289 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260809 15:45:16.999675 21289 hybrid_clock.cc:648] HybridClock initialized: now 1786290316999658 us; error 0 us; skew 500 ppm
I20260809 15:45:17.000161 21289 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 15:45:17.006021 21289 webserver.cc:460] Webserver started at http://127.20.202.65:44703/ using document root <none> and password file <none>
I20260809 15:45:17.006474 21289 fs_manager.cc:359] Metadata directory not provided
I20260809 15:45:17.006647 21289 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:45:17.006891 21289 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 15:45:17.007925 21289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj2AJ_N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786290285951082-21289-0/minicluster-data/ts-0-root/instance:
uuid: "cab341909d314988b6893daac17235e7"
format_stamp: "Formatted at 2026-08-09 15:45:17 on dist-test-slave-gjrz"
I20260809 15:45:17.012079 21289 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260809 15:45:17.014815 22355 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:45:17.015460 21289 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.002s
I20260809 15:45:17.015710 21289 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj2AJ_N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786290285951082-21289-0/minicluster-data/ts-0-root
uuid: "cab341909d314988b6893daac17235e7"
format_stamp: "Formatted at 2026-08-09 15:45:17 on dist-test-slave-gjrz"
I20260809 15:45:17.015971 21289 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj2AJ_N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786290285951082-21289-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskj2AJ_N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786290285951082-21289-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskj2AJ_N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786290285951082-21289-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:45:17.027568 21289 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:45:17.028590 21289 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:45:17.029882 21289 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 15:45:17.031904 21289 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 15:45:17.032074 21289 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 15:45:17.032277 21289 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 15:45:17.032428 21289 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 15:45:17.069160 21289 rpc_server.cc:307] RPC server started. Bound to: 127.20.202.65:46511
I20260809 15:45:17.069254 22472 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.202.65:46511 every 8 connection(s)
I20260809 15:45:17.073328 21289 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:45:17.080142 22479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:45:17.080673 22480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:45:17.083407 22484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:45:17.083850 21289 server_base.cc:1034] running on GCE node
I20260809 15:45:17.084975 21289 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260809 15:45:17.085208 21289 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260809 15:45:17.085392 21289 hybrid_clock.cc:648] HybridClock initialized: now 1786290317085377 us; error 0 us; skew 500 ppm
I20260809 15:45:17.086055 21289 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 15:45:17.086874 22473 heartbeater.cc:347] Connected to a master server at 127.20.202.126:44859
I20260809 15:45:17.087226 22473 heartbeater.cc:464] Registering TS with master...
I20260809 15:45:17.087929 22473 heartbeater.cc:511] Master 127.20.202.126:44859 requested a full tablet report, sending...
I20260809 15:45:17.088716 21289 webserver.cc:460] Webserver started at http://127.20.202.66:46733/ using document root <none> and password file <none>
I20260809 15:45:17.089318 21289 fs_manager.cc:359] Metadata directory not provided
I20260809 15:45:17.089488 21289 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:45:17.089769 21289 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 15:45:17.089864 22244 ts_manager.cc:194] Registered new tserver with Master: cab341909d314988b6893daac17235e7 (127.20.202.65:46511)
I20260809 15:45:17.090868 21289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj2AJ_N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786290285951082-21289-0/minicluster-data/ts-1-root/instance:
uuid: "8586928e65c543c2b99cebf1dd7d700f"
format_stamp: "Formatted at 2026-08-09 15:45:17 on dist-test-slave-gjrz"
I20260809 15:45:17.091748 22244 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:37918
I20260809 15:45:17.095103 21289 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260809 15:45:17.098572 22489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:45:17.099216 21289 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 15:45:17.099495 21289 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj2AJ_N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786290285951082-21289-0/minicluster-data/ts-1-root
uuid: "8586928e65c543c2b99cebf1dd7d700f"
format_stamp: "Formatted at 2026-08-09 15:45:17 on dist-test-slave-gjrz"
I20260809 15:45:17.099738 21289 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj2AJ_N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786290285951082-21289-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskj2AJ_N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786290285951082-21289-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskj2AJ_N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786290285951082-21289-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:45:17.115787 21289 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:45:17.116803 21289 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:45:17.118124 21289 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 15:45:17.120414 21289 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 15:45:17.120586 21289 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 15:45:17.120828 21289 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 15:45:17.120961 21289 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 15:45:17.157205 21289 rpc_server.cc:307] RPC server started. Bound to: 127.20.202.66:45399
I20260809 15:45:17.157306 22601 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.202.66:45399 every 8 connection(s)
I20260809 15:45:17.161552 21289 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:45:17.168138 22618 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:45:17.169554 22620 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:45:17.173792 21289 server_base.cc:1034] running on GCE node
W20260809 15:45:17.174377 22623 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:45:17.174880 22604 heartbeater.cc:347] Connected to a master server at 127.20.202.126:44859
I20260809 15:45:17.175195 22604 heartbeater.cc:464] Registering TS with master...
I20260809 15:45:17.175302 21289 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260809 15:45:17.175563 21289 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260809 15:45:17.175715 21289 hybrid_clock.cc:648] HybridClock initialized: now 1786290317175703 us; error 0 us; skew 500 ppm
I20260809 15:45:17.175868 22604 heartbeater.cc:511] Master 127.20.202.126:44859 requested a full tablet report, sending...
I20260809 15:45:17.176255 21289 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 15:45:17.177636 22244 ts_manager.cc:194] Registered new tserver with Master: 8586928e65c543c2b99cebf1dd7d700f (127.20.202.66:45399)
I20260809 15:45:17.179052 22244 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:37932
I20260809 15:45:17.179152 21289 webserver.cc:460] Webserver started at http://127.20.202.67:34119/ using document root <none> and password file <none>
I20260809 15:45:17.179713 21289 fs_manager.cc:359] Metadata directory not provided
I20260809 15:45:17.179944 21289 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:45:17.180223 21289 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 15:45:17.181264 21289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj2AJ_N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786290285951082-21289-0/minicluster-data/ts-2-root/instance:
uuid: "27fddeff55ea476ebe7bd4881ae32035"
format_stamp: "Formatted at 2026-08-09 15:45:17 on dist-test-slave-gjrz"
I20260809 15:45:17.185357 21289 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260809 15:45:17.188256 22630 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:45:17.188915 21289 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 15:45:17.189172 21289 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj2AJ_N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786290285951082-21289-0/minicluster-data/ts-2-root
uuid: "27fddeff55ea476ebe7bd4881ae32035"
format_stamp: "Formatted at 2026-08-09 15:45:17 on dist-test-slave-gjrz"
I20260809 15:45:17.189412 21289 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj2AJ_N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786290285951082-21289-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskj2AJ_N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786290285951082-21289-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskj2AJ_N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786290285951082-21289-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:45:17.211977 21289 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:45:17.212970 21289 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:45:17.214246 21289 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 15:45:17.216269 21289 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 15:45:17.216454 21289 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 15:45:17.216661 21289 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 15:45:17.216801 21289 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 15:45:17.253800 21289 rpc_server.cc:307] RPC server started. Bound to: 127.20.202.67:33583
I20260809 15:45:17.253922 22738 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.202.67:33583 every 8 connection(s)
I20260809 15:45:17.267663 22741 heartbeater.cc:347] Connected to a master server at 127.20.202.126:44859
I20260809 15:45:17.267993 22741 heartbeater.cc:464] Registering TS with master...
I20260809 15:45:17.268612 22741 heartbeater.cc:511] Master 127.20.202.126:44859 requested a full tablet report, sending...
I20260809 15:45:17.270219 22244 ts_manager.cc:194] Registered new tserver with Master: 27fddeff55ea476ebe7bd4881ae32035 (127.20.202.67:33583)
I20260809 15:45:17.271304 21289 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.014433633s
I20260809 15:45:17.271656 22244 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:37944
I20260809 15:45:17.273670 21289 test_util.cc:268] Using random seed: 533965364
I20260809 15:45:17.295131 22244 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37960:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s""\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 15:45:17.297773 22244 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 15:45:17.372303 22542 tablet_service.cc:1467] Processing CreateTablet for tablet 6b1f47a4af1b4480ab2e69adb8be3cc3 (DEFAULT_TABLE table=test-workload [id=4d635a5919924282ac23aeb965203491]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260809 15:45:17.372557 22541 tablet_service.cc:1467] Processing CreateTablet for tablet 8c8c4f25c5364a919da21f5fca365c45 (DEFAULT_TABLE table=test-workload [id=4d635a5919924282ac23aeb965203491]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260809 15:45:17.373705 22542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b1f47a4af1b4480ab2e69adb8be3cc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:17.374385 22540 tablet_service.cc:1467] Processing CreateTablet for tablet 9b4926c9270e4754bac26cc4c3da4cf9 (DEFAULT_TABLE table=test-workload [id=4d635a5919924282ac23aeb965203491]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260809 15:45:17.376250 22535 tablet_service.cc:1467] Processing CreateTablet for tablet 1154606dbfd849c7833c096541e8c58f (DEFAULT_TABLE table=test-workload [id=4d635a5919924282ac23aeb965203491]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260809 15:45:17.378080 22534 tablet_service.cc:1467] Processing CreateTablet for tablet 4e0d833302ff456e96ab09aa12d4c5ba (DEFAULT_TABLE table=test-workload [id=4d635a5919924282ac23aeb965203491]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260809 15:45:17.378609 22405 tablet_service.cc:1467] Processing CreateTablet for tablet 6b1f47a4af1b4480ab2e69adb8be3cc3 (DEFAULT_TABLE table=test-workload [id=4d635a5919924282ac23aeb965203491]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260809 15:45:17.379570 22405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b1f47a4af1b4480ab2e69adb8be3cc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:17.379943 22533 tablet_service.cc:1467] Processing CreateTablet for tablet ab0d7bb4b86540b28101d1c813ee214e (DEFAULT_TABLE table=test-workload [id=4d635a5919924282ac23aeb965203491]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260809 15:45:17.380684 22404 tablet_service.cc:1467] Processing CreateTablet for tablet 8c8c4f25c5364a919da21f5fca365c45 (DEFAULT_TABLE table=test-workload [id=4d635a5919924282ac23aeb965203491]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260809 15:45:17.381906 22404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c8c4f25c5364a919da21f5fca365c45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:17.383797 22673 tablet_service.cc:1467] Processing CreateTablet for tablet 9b4926c9270e4754bac26cc4c3da4cf9 (DEFAULT_TABLE table=test-workload [id=4d635a5919924282ac23aeb965203491]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260809 15:45:17.384986 22673 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b4926c9270e4754bac26cc4c3da4cf9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:17.386291 22531 tablet_service.cc:1467] Processing CreateTablet for tablet 6d6e21b2d7b746469c42c6d80ad8e001 (DEFAULT_TABLE table=test-workload [id=4d635a5919924282ac23aeb965203491]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260809 15:45:17.388377 22403 tablet_service.cc:1467] Processing CreateTablet for tablet 9b4926c9270e4754bac26cc4c3da4cf9 (DEFAULT_TABLE table=test-workload [id=4d635a5919924282ac23aeb965203491]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260809 15:45:17.388567 22669 tablet_service.cc:1467] Processing CreateTablet for tablet 6d6e21b2d7b746469c42c6d80ad8e001 (DEFAULT_TABLE table=test-workload [id=4d635a5919924282ac23aeb965203491]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260809 15:45:17.389825 22669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d6e21b2d7b746469c42c6d80ad8e001. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:17.389956 22529 tablet_service.cc:1467] Processing CreateTablet for tablet 21aa1094e1cf4d40b51e5916cb6a2a96 (DEFAULT_TABLE table=test-workload [id=4d635a5919924282ac23aeb965203491]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260809 15:45:17.390476 22399 tablet_service.cc:1467] Processing CreateTablet for tablet 1154606dbfd849c7833c096541e8c58f (DEFAULT_TABLE table=test-workload [id=4d635a5919924282ac23aeb965203491]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260809 15:45:17.392549 22398 tablet_service.cc:1467] Processing CreateTablet for tablet 4e0d833302ff456e96ab09aa12d4c5ba (DEFAULT_TABLE table=test-workload [id=4d635a5919924282ac23aeb965203491]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260809 15:45:17.394591 22395 tablet_service.cc:1467] Processing CreateTablet for tablet ab0d7bb4b86540b28101d1c813ee214e (DEFAULT_TABLE table=test-workload [id=4d635a5919924282ac23aeb965203491]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260809 15:45:17.396001 22528 tablet_service.cc:1467] Processing CreateTablet for tablet 6eb2494806b34a2bad3215770bcb4c7c (DEFAULT_TABLE table=test-workload [id=4d635a5919924282ac23aeb965203491]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260809 15:45:17.396878 22675 tablet_service.cc:1467] Processing CreateTablet for tablet 6b1f47a4af1b4480ab2e69adb8be3cc3 (DEFAULT_TABLE table=test-workload [id=4d635a5919924282ac23aeb965203491]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260809 15:45:17.398061 22675 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b1f47a4af1b4480ab2e69adb8be3cc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:17.398933 22670 tablet_service.cc:1467] Processing CreateTablet for tablet ab0d7bb4b86540b28101d1c813ee214e (DEFAULT_TABLE table=test-workload [id=4d635a5919924282ac23aeb965203491]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260809 15:45:17.400275 22670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab0d7bb4b86540b28101d1c813ee214e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:17.400506 22674 tablet_service.cc:1467] Processing CreateTablet for tablet 8c8c4f25c5364a919da21f5fca365c45 (DEFAULT_TABLE table=test-workload [id=4d635a5919924282ac23aeb965203491]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260809 15:45:17.401684 22674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c8c4f25c5364a919da21f5fca365c45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:17.402729 22668 tablet_service.cc:1467] Processing CreateTablet for tablet 21aa1094e1cf4d40b51e5916cb6a2a96 (DEFAULT_TABLE table=test-workload [id=4d635a5919924282ac23aeb965203491]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260809 15:45:17.403512 22672 tablet_service.cc:1467] Processing CreateTablet for tablet 1154606dbfd849c7833c096541e8c58f (DEFAULT_TABLE table=test-workload [id=4d635a5919924282ac23aeb965203491]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260809 15:45:17.403952 22668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21aa1094e1cf4d40b51e5916cb6a2a96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:17.403769 22664 tablet_service.cc:1467] Processing CreateTablet for tablet 36cb01872dee4485b764d163e0bf4e20 (DEFAULT_TABLE table=test-workload [id=4d635a5919924282ac23aeb965203491]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260809 15:45:17.404685 22672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1154606dbfd849c7833c096541e8c58f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:17.405944 22665 tablet_service.cc:1467] Processing CreateTablet for tablet 6eb2494806b34a2bad3215770bcb4c7c (DEFAULT_TABLE table=test-workload [id=4d635a5919924282ac23aeb965203491]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260809 15:45:17.407105 22665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6eb2494806b34a2bad3215770bcb4c7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:17.407505 22671 tablet_service.cc:1467] Processing CreateTablet for tablet 4e0d833302ff456e96ab09aa12d4c5ba (DEFAULT_TABLE table=test-workload [id=4d635a5919924282ac23aeb965203491]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260809 15:45:17.408656 22671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e0d833302ff456e96ab09aa12d4c5ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:17.410308 22664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36cb01872dee4485b764d163e0bf4e20. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:17.410915 22527 tablet_service.cc:1467] Processing CreateTablet for tablet 36cb01872dee4485b764d163e0bf4e20 (DEFAULT_TABLE table=test-workload [id=4d635a5919924282ac23aeb965203491]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260809 15:45:17.415177 22540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b4926c9270e4754bac26cc4c3da4cf9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:17.417629 22535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1154606dbfd849c7833c096541e8c58f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:17.418838 22534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e0d833302ff456e96ab09aa12d4c5ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:17.421254 22533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab0d7bb4b86540b28101d1c813ee214e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:17.423794 22531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d6e21b2d7b746469c42c6d80ad8e001. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:17.426131 22529 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21aa1094e1cf4d40b51e5916cb6a2a96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:17.426832 22541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c8c4f25c5364a919da21f5fca365c45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:17.431555 22528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6eb2494806b34a2bad3215770bcb4c7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:17.435709 22527 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36cb01872dee4485b764d163e0bf4e20. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:17.440819 22391 tablet_service.cc:1467] Processing CreateTablet for tablet 6d6e21b2d7b746469c42c6d80ad8e001 (DEFAULT_TABLE table=test-workload [id=4d635a5919924282ac23aeb965203491]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260809 15:45:17.443493 22389 tablet_service.cc:1467] Processing CreateTablet for tablet 21aa1094e1cf4d40b51e5916cb6a2a96 (DEFAULT_TABLE table=test-workload [id=4d635a5919924282ac23aeb965203491]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260809 15:45:17.447904 22764 tablet_bootstrap.cc:492] T 6b1f47a4af1b4480ab2e69adb8be3cc3 P cab341909d314988b6893daac17235e7: Bootstrap starting.
I20260809 15:45:17.451793 22765 tablet_bootstrap.cc:492] T 6d6e21b2d7b746469c42c6d80ad8e001 P 27fddeff55ea476ebe7bd4881ae32035: Bootstrap starting.
I20260809 15:45:17.457310 22387 tablet_service.cc:1467] Processing CreateTablet for tablet 36cb01872dee4485b764d163e0bf4e20 (DEFAULT_TABLE table=test-workload [id=4d635a5919924282ac23aeb965203491]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260809 15:45:17.475050 22403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b4926c9270e4754bac26cc4c3da4cf9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:17.477636 22398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e0d833302ff456e96ab09aa12d4c5ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:17.478493 22765 tablet_bootstrap.cc:654] T 6d6e21b2d7b746469c42c6d80ad8e001 P 27fddeff55ea476ebe7bd4881ae32035: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:17.479537 22388 tablet_service.cc:1467] Processing CreateTablet for tablet 6eb2494806b34a2bad3215770bcb4c7c (DEFAULT_TABLE table=test-workload [id=4d635a5919924282ac23aeb965203491]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260809 15:45:17.480780 22388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6eb2494806b34a2bad3215770bcb4c7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:17.483592 22389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21aa1094e1cf4d40b51e5916cb6a2a96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:17.487593 22391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d6e21b2d7b746469c42c6d80ad8e001. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:17.488494 22399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1154606dbfd849c7833c096541e8c58f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:17.489773 22387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36cb01872dee4485b764d163e0bf4e20. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:17.496490 22395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab0d7bb4b86540b28101d1c813ee214e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:17.522467 22764 tablet_bootstrap.cc:654] T 6b1f47a4af1b4480ab2e69adb8be3cc3 P cab341909d314988b6893daac17235e7: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:17.532370 22766 tablet_bootstrap.cc:492] T 21aa1094e1cf4d40b51e5916cb6a2a96 P 8586928e65c543c2b99cebf1dd7d700f: Bootstrap starting.
I20260809 15:45:17.550331 22765 tablet_bootstrap.cc:492] T 6d6e21b2d7b746469c42c6d80ad8e001 P 27fddeff55ea476ebe7bd4881ae32035: No bootstrap required, opened a new log
I20260809 15:45:17.550782 22765 ts_tablet_manager.cc:1397] T 6d6e21b2d7b746469c42c6d80ad8e001 P 27fddeff55ea476ebe7bd4881ae32035: Time spent bootstrapping tablet: real 0.099s user 0.021s sys 0.034s
I20260809 15:45:17.553277 22765 raft_consensus.cc:348] T 6d6e21b2d7b746469c42c6d80ad8e001 P 27fddeff55ea476ebe7bd4881ae32035 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } } peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } }
I20260809 15:45:17.553951 22765 raft_consensus.cc:374] T 6d6e21b2d7b746469c42c6d80ad8e001 P 27fddeff55ea476ebe7bd4881ae32035 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:17.554226 22765 raft_consensus.cc:729] T 6d6e21b2d7b746469c42c6d80ad8e001 P 27fddeff55ea476ebe7bd4881ae32035 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27fddeff55ea476ebe7bd4881ae32035, State: Initialized, Role: FOLLOWER
I20260809 15:45:17.554929 22765 consensus_queue.cc:260] T 6d6e21b2d7b746469c42c6d80ad8e001 P 27fddeff55ea476ebe7bd4881ae32035 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } } peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } }
I20260809 15:45:17.560652 22766 tablet_bootstrap.cc:654] T 21aa1094e1cf4d40b51e5916cb6a2a96 P 8586928e65c543c2b99cebf1dd7d700f: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:17.567054 22764 tablet_bootstrap.cc:492] T 6b1f47a4af1b4480ab2e69adb8be3cc3 P cab341909d314988b6893daac17235e7: No bootstrap required, opened a new log
I20260809 15:45:17.567554 22764 ts_tablet_manager.cc:1397] T 6b1f47a4af1b4480ab2e69adb8be3cc3 P cab341909d314988b6893daac17235e7: Time spent bootstrapping tablet: real 0.120s user 0.007s sys 0.005s
I20260809 15:45:17.569722 22741 heartbeater.cc:503] Master 127.20.202.126:44859 was elected leader, sending a full tablet report...
I20260809 15:45:17.570170 22765 ts_tablet_manager.cc:1428] T 6d6e21b2d7b746469c42c6d80ad8e001 P 27fddeff55ea476ebe7bd4881ae32035: Time spent starting tablet: real 0.019s user 0.009s sys 0.009s
I20260809 15:45:17.570170 22764 raft_consensus.cc:348] T 6b1f47a4af1b4480ab2e69adb8be3cc3 P cab341909d314988b6893daac17235e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } } peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } }
I20260809 15:45:17.571046 22765 tablet_bootstrap.cc:492] T 21aa1094e1cf4d40b51e5916cb6a2a96 P 27fddeff55ea476ebe7bd4881ae32035: Bootstrap starting.
I20260809 15:45:17.572794 22766 tablet_bootstrap.cc:492] T 21aa1094e1cf4d40b51e5916cb6a2a96 P 8586928e65c543c2b99cebf1dd7d700f: No bootstrap required, opened a new log
I20260809 15:45:17.573203 22766 ts_tablet_manager.cc:1397] T 21aa1094e1cf4d40b51e5916cb6a2a96 P 8586928e65c543c2b99cebf1dd7d700f: Time spent bootstrapping tablet: real 0.041s user 0.006s sys 0.003s
I20260809 15:45:17.575433 22765 tablet_bootstrap.cc:654] T 21aa1094e1cf4d40b51e5916cb6a2a96 P 27fddeff55ea476ebe7bd4881ae32035: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:17.575793 22766 raft_consensus.cc:348] T 21aa1094e1cf4d40b51e5916cb6a2a96 P 8586928e65c543c2b99cebf1dd7d700f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } } peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } }
I20260809 15:45:17.576501 22766 raft_consensus.cc:374] T 21aa1094e1cf4d40b51e5916cb6a2a96 P 8586928e65c543c2b99cebf1dd7d700f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:17.576792 22766 raft_consensus.cc:729] T 21aa1094e1cf4d40b51e5916cb6a2a96 P 8586928e65c543c2b99cebf1dd7d700f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8586928e65c543c2b99cebf1dd7d700f, State: Initialized, Role: FOLLOWER
I20260809 15:45:17.577436 22766 consensus_queue.cc:260] T 21aa1094e1cf4d40b51e5916cb6a2a96 P 8586928e65c543c2b99cebf1dd7d700f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } } peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } }
I20260809 15:45:17.571125 22764 raft_consensus.cc:374] T 6b1f47a4af1b4480ab2e69adb8be3cc3 P cab341909d314988b6893daac17235e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:17.578923 22764 raft_consensus.cc:729] T 6b1f47a4af1b4480ab2e69adb8be3cc3 P cab341909d314988b6893daac17235e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cab341909d314988b6893daac17235e7, State: Initialized, Role: FOLLOWER
I20260809 15:45:17.579757 22604 heartbeater.cc:503] Master 127.20.202.126:44859 was elected leader, sending a full tablet report...
I20260809 15:45:17.580117 22765 tablet_bootstrap.cc:492] T 21aa1094e1cf4d40b51e5916cb6a2a96 P 27fddeff55ea476ebe7bd4881ae32035: No bootstrap required, opened a new log
I20260809 15:45:17.580543 22765 ts_tablet_manager.cc:1397] T 21aa1094e1cf4d40b51e5916cb6a2a96 P 27fddeff55ea476ebe7bd4881ae32035: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20260809 15:45:17.581435 22766 ts_tablet_manager.cc:1428] T 21aa1094e1cf4d40b51e5916cb6a2a96 P 8586928e65c543c2b99cebf1dd7d700f: Time spent starting tablet: real 0.008s user 0.006s sys 0.001s
I20260809 15:45:17.582682 22765 raft_consensus.cc:348] T 21aa1094e1cf4d40b51e5916cb6a2a96 P 27fddeff55ea476ebe7bd4881ae32035 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } } peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } }
I20260809 15:45:17.583289 22765 raft_consensus.cc:374] T 21aa1094e1cf4d40b51e5916cb6a2a96 P 27fddeff55ea476ebe7bd4881ae32035 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:17.583562 22766 tablet_bootstrap.cc:492] T 6eb2494806b34a2bad3215770bcb4c7c P 8586928e65c543c2b99cebf1dd7d700f: Bootstrap starting.
I20260809 15:45:17.579674 22764 consensus_queue.cc:260] T 6b1f47a4af1b4480ab2e69adb8be3cc3 P cab341909d314988b6893daac17235e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } } peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } }
I20260809 15:45:17.583611 22765 raft_consensus.cc:729] T 21aa1094e1cf4d40b51e5916cb6a2a96 P 27fddeff55ea476ebe7bd4881ae32035 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27fddeff55ea476ebe7bd4881ae32035, State: Initialized, Role: FOLLOWER
I20260809 15:45:17.587167 22765 consensus_queue.cc:260] T 21aa1094e1cf4d40b51e5916cb6a2a96 P 27fddeff55ea476ebe7bd4881ae32035 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } } peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } }
I20260809 15:45:17.589509 22766 tablet_bootstrap.cc:654] T 6eb2494806b34a2bad3215770bcb4c7c P 8586928e65c543c2b99cebf1dd7d700f: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:17.589624 22765 ts_tablet_manager.cc:1428] T 21aa1094e1cf4d40b51e5916cb6a2a96 P 27fddeff55ea476ebe7bd4881ae32035: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20260809 15:45:17.590730 22765 tablet_bootstrap.cc:492] T 8c8c4f25c5364a919da21f5fca365c45 P 27fddeff55ea476ebe7bd4881ae32035: Bootstrap starting.
I20260809 15:45:17.600554 22765 tablet_bootstrap.cc:654] T 8c8c4f25c5364a919da21f5fca365c45 P 27fddeff55ea476ebe7bd4881ae32035: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:17.618378 22764 ts_tablet_manager.cc:1428] T 6b1f47a4af1b4480ab2e69adb8be3cc3 P cab341909d314988b6893daac17235e7: Time spent starting tablet: real 0.051s user 0.012s sys 0.025s
I20260809 15:45:17.619251 22473 heartbeater.cc:503] Master 127.20.202.126:44859 was elected leader, sending a full tablet report...
I20260809 15:45:17.619592 22764 tablet_bootstrap.cc:492] T 1154606dbfd849c7833c096541e8c58f P cab341909d314988b6893daac17235e7: Bootstrap starting.
I20260809 15:45:17.619936 22766 tablet_bootstrap.cc:492] T 6eb2494806b34a2bad3215770bcb4c7c P 8586928e65c543c2b99cebf1dd7d700f: No bootstrap required, opened a new log
I20260809 15:45:17.620357 22766 ts_tablet_manager.cc:1397] T 6eb2494806b34a2bad3215770bcb4c7c P 8586928e65c543c2b99cebf1dd7d700f: Time spent bootstrapping tablet: real 0.037s user 0.017s sys 0.018s
I20260809 15:45:17.622882 22766 raft_consensus.cc:348] T 6eb2494806b34a2bad3215770bcb4c7c P 8586928e65c543c2b99cebf1dd7d700f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } } peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } }
I20260809 15:45:17.623572 22766 raft_consensus.cc:374] T 6eb2494806b34a2bad3215770bcb4c7c P 8586928e65c543c2b99cebf1dd7d700f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:17.623828 22766 raft_consensus.cc:729] T 6eb2494806b34a2bad3215770bcb4c7c P 8586928e65c543c2b99cebf1dd7d700f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8586928e65c543c2b99cebf1dd7d700f, State: Initialized, Role: FOLLOWER
I20260809 15:45:17.624363 22764 tablet_bootstrap.cc:654] T 1154606dbfd849c7833c096541e8c58f P cab341909d314988b6893daac17235e7: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:17.624415 22765 tablet_bootstrap.cc:492] T 8c8c4f25c5364a919da21f5fca365c45 P 27fddeff55ea476ebe7bd4881ae32035: No bootstrap required, opened a new log
I20260809 15:45:17.624939 22765 ts_tablet_manager.cc:1397] T 8c8c4f25c5364a919da21f5fca365c45 P 27fddeff55ea476ebe7bd4881ae32035: Time spent bootstrapping tablet: real 0.034s user 0.019s sys 0.006s
I20260809 15:45:17.624517 22766 consensus_queue.cc:260] T 6eb2494806b34a2bad3215770bcb4c7c P 8586928e65c543c2b99cebf1dd7d700f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } } peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } }
I20260809 15:45:17.626628 22766 ts_tablet_manager.cc:1428] T 6eb2494806b34a2bad3215770bcb4c7c P 8586928e65c543c2b99cebf1dd7d700f: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260809 15:45:17.627482 22766 tablet_bootstrap.cc:492] T 9b4926c9270e4754bac26cc4c3da4cf9 P 8586928e65c543c2b99cebf1dd7d700f: Bootstrap starting.
I20260809 15:45:17.627391 22765 raft_consensus.cc:348] T 8c8c4f25c5364a919da21f5fca365c45 P 27fddeff55ea476ebe7bd4881ae32035 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } } peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } }
I20260809 15:45:17.628095 22765 raft_consensus.cc:374] T 8c8c4f25c5364a919da21f5fca365c45 P 27fddeff55ea476ebe7bd4881ae32035 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:17.628366 22765 raft_consensus.cc:729] T 8c8c4f25c5364a919da21f5fca365c45 P 27fddeff55ea476ebe7bd4881ae32035 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27fddeff55ea476ebe7bd4881ae32035, State: Initialized, Role: FOLLOWER
I20260809 15:45:17.628988 22765 consensus_queue.cc:260] T 8c8c4f25c5364a919da21f5fca365c45 P 27fddeff55ea476ebe7bd4881ae32035 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } } peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } }
I20260809 15:45:17.630964 22765 ts_tablet_manager.cc:1428] T 8c8c4f25c5364a919da21f5fca365c45 P 27fddeff55ea476ebe7bd4881ae32035: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20260809 15:45:17.631839 22765 tablet_bootstrap.cc:492] T 6eb2494806b34a2bad3215770bcb4c7c P 27fddeff55ea476ebe7bd4881ae32035: Bootstrap starting.
I20260809 15:45:17.633092 22766 tablet_bootstrap.cc:654] T 9b4926c9270e4754bac26cc4c3da4cf9 P 8586928e65c543c2b99cebf1dd7d700f: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:17.637156 22765 tablet_bootstrap.cc:654] T 6eb2494806b34a2bad3215770bcb4c7c P 27fddeff55ea476ebe7bd4881ae32035: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:17.638320 22777 raft_consensus.cc:482] T 21aa1094e1cf4d40b51e5916cb6a2a96 P 8586928e65c543c2b99cebf1dd7d700f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:45:17.638808 22777 raft_consensus.cc:504] T 21aa1094e1cf4d40b51e5916cb6a2a96 P 8586928e65c543c2b99cebf1dd7d700f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } } peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } }
I20260809 15:45:17.647354 22777 leader_election.cc:288] T 21aa1094e1cf4d40b51e5916cb6a2a96 P 8586928e65c543c2b99cebf1dd7d700f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cab341909d314988b6893daac17235e7 (127.20.202.65:46511), 27fddeff55ea476ebe7bd4881ae32035 (127.20.202.67:33583)
I20260809 15:45:17.668380 22436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21aa1094e1cf4d40b51e5916cb6a2a96" candidate_uuid: "8586928e65c543c2b99cebf1dd7d700f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab341909d314988b6893daac17235e7" is_pre_election: true
W20260809 15:45:17.670068 22492 leader_election.cc:341] T 21aa1094e1cf4d40b51e5916cb6a2a96 P 8586928e65c543c2b99cebf1dd7d700f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cab341909d314988b6893daac17235e7 (127.20.202.65:46511): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:17.701941 22690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21aa1094e1cf4d40b51e5916cb6a2a96" candidate_uuid: "8586928e65c543c2b99cebf1dd7d700f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27fddeff55ea476ebe7bd4881ae32035" is_pre_election: true
I20260809 15:45:17.702613 22690 raft_consensus.cc:2454] T 21aa1094e1cf4d40b51e5916cb6a2a96 P 27fddeff55ea476ebe7bd4881ae32035 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8586928e65c543c2b99cebf1dd7d700f in term 0.
I20260809 15:45:17.703187 22777 raft_consensus.cc:482] T 6eb2494806b34a2bad3215770bcb4c7c P 8586928e65c543c2b99cebf1dd7d700f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:45:17.703655 22777 raft_consensus.cc:504] T 6eb2494806b34a2bad3215770bcb4c7c P 8586928e65c543c2b99cebf1dd7d700f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } } peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } }
I20260809 15:45:17.703975 22493 leader_election.cc:302] T 21aa1094e1cf4d40b51e5916cb6a2a96 P 8586928e65c543c2b99cebf1dd7d700f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27fddeff55ea476ebe7bd4881ae32035, 8586928e65c543c2b99cebf1dd7d700f; no voters: cab341909d314988b6893daac17235e7
I20260809 15:45:17.705463 22765 tablet_bootstrap.cc:492] T 6eb2494806b34a2bad3215770bcb4c7c P 27fddeff55ea476ebe7bd4881ae32035: No bootstrap required, opened a new log
I20260809 15:45:17.705719 22764 tablet_bootstrap.cc:492] T 1154606dbfd849c7833c096541e8c58f P cab341909d314988b6893daac17235e7: No bootstrap required, opened a new log
I20260809 15:45:17.705500 22777 leader_election.cc:288] T 6eb2494806b34a2bad3215770bcb4c7c P 8586928e65c543c2b99cebf1dd7d700f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cab341909d314988b6893daac17235e7 (127.20.202.65:46511), 27fddeff55ea476ebe7bd4881ae32035 (127.20.202.67:33583)
I20260809 15:45:17.706182 22765 ts_tablet_manager.cc:1397] T 6eb2494806b34a2bad3215770bcb4c7c P 27fddeff55ea476ebe7bd4881ae32035: Time spent bootstrapping tablet: real 0.075s user 0.039s sys 0.032s
I20260809 15:45:17.709427 22764 ts_tablet_manager.cc:1397] T 1154606dbfd849c7833c096541e8c58f P cab341909d314988b6893daac17235e7: Time spent bootstrapping tablet: real 0.090s user 0.024s sys 0.051s
I20260809 15:45:17.707015 22436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eb2494806b34a2bad3215770bcb4c7c" candidate_uuid: "8586928e65c543c2b99cebf1dd7d700f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab341909d314988b6893daac17235e7" is_pre_election: true
I20260809 15:45:17.706173 22777 raft_consensus.cc:2781] T 21aa1094e1cf4d40b51e5916cb6a2a96 P 8586928e65c543c2b99cebf1dd7d700f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 15:45:17.710014 22777 raft_consensus.cc:482] T 21aa1094e1cf4d40b51e5916cb6a2a96 P 8586928e65c543c2b99cebf1dd7d700f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:45:17.710326 22777 raft_consensus.cc:3037] T 21aa1094e1cf4d40b51e5916cb6a2a96 P 8586928e65c543c2b99cebf1dd7d700f [term 0 FOLLOWER]: Advancing to term 1
W20260809 15:45:17.710638 22492 leader_election.cc:341] T 6eb2494806b34a2bad3215770bcb4c7c P 8586928e65c543c2b99cebf1dd7d700f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cab341909d314988b6893daac17235e7 (127.20.202.65:46511): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:17.712206 22765 raft_consensus.cc:348] T 6eb2494806b34a2bad3215770bcb4c7c P 27fddeff55ea476ebe7bd4881ae32035 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } } peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } }
I20260809 15:45:17.712949 22765 raft_consensus.cc:374] T 6eb2494806b34a2bad3215770bcb4c7c P 27fddeff55ea476ebe7bd4881ae32035 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:17.713165 22766 tablet_bootstrap.cc:492] T 9b4926c9270e4754bac26cc4c3da4cf9 P 8586928e65c543c2b99cebf1dd7d700f: No bootstrap required, opened a new log
I20260809 15:45:17.713284 22765 raft_consensus.cc:729] T 6eb2494806b34a2bad3215770bcb4c7c P 27fddeff55ea476ebe7bd4881ae32035 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27fddeff55ea476ebe7bd4881ae32035, State: Initialized, Role: FOLLOWER
I20260809 15:45:17.713706 22766 ts_tablet_manager.cc:1397] T 9b4926c9270e4754bac26cc4c3da4cf9 P 8586928e65c543c2b99cebf1dd7d700f: Time spent bootstrapping tablet: real 0.086s user 0.018s sys 0.053s
I20260809 15:45:17.714195 22765 consensus_queue.cc:260] T 6eb2494806b34a2bad3215770bcb4c7c P 27fddeff55ea476ebe7bd4881ae32035 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } } peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } }
I20260809 15:45:17.716435 22765 ts_tablet_manager.cc:1428] T 6eb2494806b34a2bad3215770bcb4c7c P 27fddeff55ea476ebe7bd4881ae32035: Time spent starting tablet: real 0.007s user 0.006s sys 0.001s
I20260809 15:45:17.716342 22766 raft_consensus.cc:348] T 9b4926c9270e4754bac26cc4c3da4cf9 P 8586928e65c543c2b99cebf1dd7d700f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } } peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } }
I20260809 15:45:17.717300 22766 raft_consensus.cc:374] T 9b4926c9270e4754bac26cc4c3da4cf9 P 8586928e65c543c2b99cebf1dd7d700f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:17.717708 22765 tablet_bootstrap.cc:492] T ab0d7bb4b86540b28101d1c813ee214e P 27fddeff55ea476ebe7bd4881ae32035: Bootstrap starting.
I20260809 15:45:17.717682 22766 raft_consensus.cc:729] T 9b4926c9270e4754bac26cc4c3da4cf9 P 8586928e65c543c2b99cebf1dd7d700f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8586928e65c543c2b99cebf1dd7d700f, State: Initialized, Role: FOLLOWER
I20260809 15:45:17.719116 22766 consensus_queue.cc:260] T 9b4926c9270e4754bac26cc4c3da4cf9 P 8586928e65c543c2b99cebf1dd7d700f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } } peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } }
I20260809 15:45:17.719667 22690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eb2494806b34a2bad3215770bcb4c7c" candidate_uuid: "8586928e65c543c2b99cebf1dd7d700f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27fddeff55ea476ebe7bd4881ae32035" is_pre_election: true
I20260809 15:45:17.720230 22690 raft_consensus.cc:2454] T 6eb2494806b34a2bad3215770bcb4c7c P 27fddeff55ea476ebe7bd4881ae32035 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8586928e65c543c2b99cebf1dd7d700f in term 0.
I20260809 15:45:17.721164 22766 ts_tablet_manager.cc:1428] T 9b4926c9270e4754bac26cc4c3da4cf9 P 8586928e65c543c2b99cebf1dd7d700f: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260809 15:45:17.721325 22493 leader_election.cc:302] T 6eb2494806b34a2bad3215770bcb4c7c P 8586928e65c543c2b99cebf1dd7d700f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27fddeff55ea476ebe7bd4881ae32035, 8586928e65c543c2b99cebf1dd7d700f; no voters: cab341909d314988b6893daac17235e7
I20260809 15:45:17.724313 22765 tablet_bootstrap.cc:654] T ab0d7bb4b86540b28101d1c813ee214e P 27fddeff55ea476ebe7bd4881ae32035: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:17.728389 22766 tablet_bootstrap.cc:492] T 6b1f47a4af1b4480ab2e69adb8be3cc3 P 8586928e65c543c2b99cebf1dd7d700f: Bootstrap starting.
I20260809 15:45:17.728528 22764 raft_consensus.cc:348] T 1154606dbfd849c7833c096541e8c58f P cab341909d314988b6893daac17235e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } } peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } }
I20260809 15:45:17.729280 22764 raft_consensus.cc:374] T 1154606dbfd849c7833c096541e8c58f P cab341909d314988b6893daac17235e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:17.729554 22764 raft_consensus.cc:729] T 1154606dbfd849c7833c096541e8c58f P cab341909d314988b6893daac17235e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cab341909d314988b6893daac17235e7, State: Initialized, Role: FOLLOWER
I20260809 15:45:17.730211 22764 consensus_queue.cc:260] T 1154606dbfd849c7833c096541e8c58f P cab341909d314988b6893daac17235e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } } peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } }
I20260809 15:45:17.737612 22777 raft_consensus.cc:504] T 21aa1094e1cf4d40b51e5916cb6a2a96 P 8586928e65c543c2b99cebf1dd7d700f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } } peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } }
I20260809 15:45:17.738724 22795 raft_consensus.cc:2781] T 6eb2494806b34a2bad3215770bcb4c7c P 8586928e65c543c2b99cebf1dd7d700f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 15:45:17.739108 22795 raft_consensus.cc:482] T 6eb2494806b34a2bad3215770bcb4c7c P 8586928e65c543c2b99cebf1dd7d700f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:45:17.739444 22795 raft_consensus.cc:3037] T 6eb2494806b34a2bad3215770bcb4c7c P 8586928e65c543c2b99cebf1dd7d700f [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:17.740568 22436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21aa1094e1cf4d40b51e5916cb6a2a96" candidate_uuid: "8586928e65c543c2b99cebf1dd7d700f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab341909d314988b6893daac17235e7"
W20260809 15:45:17.741796 22492 leader_election.cc:341] T 21aa1094e1cf4d40b51e5916cb6a2a96 P 8586928e65c543c2b99cebf1dd7d700f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cab341909d314988b6893daac17235e7 (127.20.202.65:46511): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:17.743619 22777 leader_election.cc:288] T 21aa1094e1cf4d40b51e5916cb6a2a96 P 8586928e65c543c2b99cebf1dd7d700f [CANDIDATE]: Term 1 election: Requested vote from peers cab341909d314988b6893daac17235e7 (127.20.202.65:46511), 27fddeff55ea476ebe7bd4881ae32035 (127.20.202.67:33583)
I20260809 15:45:17.744215 22764 ts_tablet_manager.cc:1428] T 1154606dbfd849c7833c096541e8c58f P cab341909d314988b6893daac17235e7: Time spent starting tablet: real 0.030s user 0.003s sys 0.003s
I20260809 15:45:17.750680 22764 tablet_bootstrap.cc:492] T 6d6e21b2d7b746469c42c6d80ad8e001 P cab341909d314988b6893daac17235e7: Bootstrap starting.
I20260809 15:45:17.750329 22795 raft_consensus.cc:504] T 6eb2494806b34a2bad3215770bcb4c7c P 8586928e65c543c2b99cebf1dd7d700f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } } peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } }
I20260809 15:45:17.753005 22436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eb2494806b34a2bad3215770bcb4c7c" candidate_uuid: "8586928e65c543c2b99cebf1dd7d700f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab341909d314988b6893daac17235e7"
W20260809 15:45:17.754237 22492 leader_election.cc:341] T 6eb2494806b34a2bad3215770bcb4c7c P 8586928e65c543c2b99cebf1dd7d700f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cab341909d314988b6893daac17235e7 (127.20.202.65:46511): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:17.755139 22766 tablet_bootstrap.cc:654] T 6b1f47a4af1b4480ab2e69adb8be3cc3 P 8586928e65c543c2b99cebf1dd7d700f: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:17.756589 22764 tablet_bootstrap.cc:654] T 6d6e21b2d7b746469c42c6d80ad8e001 P cab341909d314988b6893daac17235e7: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:17.759642 22690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21aa1094e1cf4d40b51e5916cb6a2a96" candidate_uuid: "8586928e65c543c2b99cebf1dd7d700f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27fddeff55ea476ebe7bd4881ae32035"
I20260809 15:45:17.759867 22795 leader_election.cc:288] T 6eb2494806b34a2bad3215770bcb4c7c P 8586928e65c543c2b99cebf1dd7d700f [CANDIDATE]: Term 1 election: Requested vote from peers cab341909d314988b6893daac17235e7 (127.20.202.65:46511), 27fddeff55ea476ebe7bd4881ae32035 (127.20.202.67:33583)
I20260809 15:45:17.760185 22690 raft_consensus.cc:3037] T 21aa1094e1cf4d40b51e5916cb6a2a96 P 27fddeff55ea476ebe7bd4881ae32035 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:17.761000 22689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eb2494806b34a2bad3215770bcb4c7c" candidate_uuid: "8586928e65c543c2b99cebf1dd7d700f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27fddeff55ea476ebe7bd4881ae32035"
I20260809 15:45:17.761596 22689 raft_consensus.cc:3037] T 6eb2494806b34a2bad3215770bcb4c7c P 27fddeff55ea476ebe7bd4881ae32035 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:17.762960 22766 tablet_bootstrap.cc:492] T 6b1f47a4af1b4480ab2e69adb8be3cc3 P 8586928e65c543c2b99cebf1dd7d700f: No bootstrap required, opened a new log
I20260809 15:45:17.763392 22766 ts_tablet_manager.cc:1397] T 6b1f47a4af1b4480ab2e69adb8be3cc3 P 8586928e65c543c2b99cebf1dd7d700f: Time spent bootstrapping tablet: real 0.035s user 0.008s sys 0.003s
I20260809 15:45:17.765826 22766 raft_consensus.cc:348] T 6b1f47a4af1b4480ab2e69adb8be3cc3 P 8586928e65c543c2b99cebf1dd7d700f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } } peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } }
I20260809 15:45:17.766469 22766 raft_consensus.cc:374] T 6b1f47a4af1b4480ab2e69adb8be3cc3 P 8586928e65c543c2b99cebf1dd7d700f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:17.766729 22766 raft_consensus.cc:729] T 6b1f47a4af1b4480ab2e69adb8be3cc3 P 8586928e65c543c2b99cebf1dd7d700f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8586928e65c543c2b99cebf1dd7d700f, State: Initialized, Role: FOLLOWER
I20260809 15:45:17.767136 22690 raft_consensus.cc:2454] T 21aa1094e1cf4d40b51e5916cb6a2a96 P 27fddeff55ea476ebe7bd4881ae32035 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8586928e65c543c2b99cebf1dd7d700f in term 1.
I20260809 15:45:17.767535 22766 consensus_queue.cc:260] T 6b1f47a4af1b4480ab2e69adb8be3cc3 P 8586928e65c543c2b99cebf1dd7d700f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } } peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } }
I20260809 15:45:17.768489 22689 raft_consensus.cc:2454] T 6eb2494806b34a2bad3215770bcb4c7c P 27fddeff55ea476ebe7bd4881ae32035 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8586928e65c543c2b99cebf1dd7d700f in term 1.
I20260809 15:45:17.769680 22765 tablet_bootstrap.cc:492] T ab0d7bb4b86540b28101d1c813ee214e P 27fddeff55ea476ebe7bd4881ae32035: No bootstrap required, opened a new log
I20260809 15:45:17.769584 22493 leader_election.cc:302] T 21aa1094e1cf4d40b51e5916cb6a2a96 P 8586928e65c543c2b99cebf1dd7d700f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27fddeff55ea476ebe7bd4881ae32035, 8586928e65c543c2b99cebf1dd7d700f; no voters: cab341909d314988b6893daac17235e7
I20260809 15:45:17.770280 22765 ts_tablet_manager.cc:1397] T ab0d7bb4b86540b28101d1c813ee214e P 27fddeff55ea476ebe7bd4881ae32035: Time spent bootstrapping tablet: real 0.053s user 0.015s sys 0.025s
I20260809 15:45:17.770496 22796 raft_consensus.cc:2781] T 21aa1094e1cf4d40b51e5916cb6a2a96 P 8586928e65c543c2b99cebf1dd7d700f [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:45:17.770993 22796 raft_consensus.cc:686] T 21aa1094e1cf4d40b51e5916cb6a2a96 P 8586928e65c543c2b99cebf1dd7d700f [term 1 LEADER]: Becoming Leader. State: Replica: 8586928e65c543c2b99cebf1dd7d700f, State: Running, Role: LEADER
I20260809 15:45:17.771938 22796 consensus_queue.cc:237] T 21aa1094e1cf4d40b51e5916cb6a2a96 P 8586928e65c543c2b99cebf1dd7d700f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } } peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } }
I20260809 15:45:17.773466 22765 raft_consensus.cc:348] T ab0d7bb4b86540b28101d1c813ee214e P 27fddeff55ea476ebe7bd4881ae32035 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } } peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } }
I20260809 15:45:17.774351 22765 raft_consensus.cc:374] T ab0d7bb4b86540b28101d1c813ee214e P 27fddeff55ea476ebe7bd4881ae32035 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:17.774753 22765 raft_consensus.cc:729] T ab0d7bb4b86540b28101d1c813ee214e P 27fddeff55ea476ebe7bd4881ae32035 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27fddeff55ea476ebe7bd4881ae32035, State: Initialized, Role: FOLLOWER
I20260809 15:45:17.775151 22766 ts_tablet_manager.cc:1428] T 6b1f47a4af1b4480ab2e69adb8be3cc3 P 8586928e65c543c2b99cebf1dd7d700f: Time spent starting tablet: real 0.011s user 0.002s sys 0.004s
I20260809 15:45:17.776037 22766 tablet_bootstrap.cc:492] T 6d6e21b2d7b746469c42c6d80ad8e001 P 8586928e65c543c2b99cebf1dd7d700f: Bootstrap starting.
I20260809 15:45:17.775727 22765 consensus_queue.cc:260] T ab0d7bb4b86540b28101d1c813ee214e P 27fddeff55ea476ebe7bd4881ae32035 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } } peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } }
I20260809 15:45:17.776348 22493 leader_election.cc:302] T 6eb2494806b34a2bad3215770bcb4c7c P 8586928e65c543c2b99cebf1dd7d700f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27fddeff55ea476ebe7bd4881ae32035, 8586928e65c543c2b99cebf1dd7d700f; no voters: cab341909d314988b6893daac17235e7
I20260809 15:45:17.777750 22795 raft_consensus.cc:2781] T 6eb2494806b34a2bad3215770bcb4c7c P 8586928e65c543c2b99cebf1dd7d700f [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:45:17.778187 22795 raft_consensus.cc:686] T 6eb2494806b34a2bad3215770bcb4c7c P 8586928e65c543c2b99cebf1dd7d700f [term 1 LEADER]: Becoming Leader. State: Replica: 8586928e65c543c2b99cebf1dd7d700f, State: Running, Role: LEADER
I20260809 15:45:17.778888 22795 consensus_queue.cc:237] T 6eb2494806b34a2bad3215770bcb4c7c P 8586928e65c543c2b99cebf1dd7d700f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } } peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } }
I20260809 15:45:17.781862 22766 tablet_bootstrap.cc:654] T 6d6e21b2d7b746469c42c6d80ad8e001 P 8586928e65c543c2b99cebf1dd7d700f: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:17.784654 22765 ts_tablet_manager.cc:1428] T ab0d7bb4b86540b28101d1c813ee214e P 27fddeff55ea476ebe7bd4881ae32035: Time spent starting tablet: real 0.014s user 0.001s sys 0.005s
I20260809 15:45:17.785563 22765 tablet_bootstrap.cc:492] T 4e0d833302ff456e96ab09aa12d4c5ba P 27fddeff55ea476ebe7bd4881ae32035: Bootstrap starting.
I20260809 15:45:17.787988 22766 tablet_bootstrap.cc:492] T 6d6e21b2d7b746469c42c6d80ad8e001 P 8586928e65c543c2b99cebf1dd7d700f: No bootstrap required, opened a new log
I20260809 15:45:17.788431 22766 ts_tablet_manager.cc:1397] T 6d6e21b2d7b746469c42c6d80ad8e001 P 8586928e65c543c2b99cebf1dd7d700f: Time spent bootstrapping tablet: real 0.013s user 0.010s sys 0.000s
I20260809 15:45:17.791054 22766 raft_consensus.cc:348] T 6d6e21b2d7b746469c42c6d80ad8e001 P 8586928e65c543c2b99cebf1dd7d700f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } } peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } }
I20260809 15:45:17.791580 22765 tablet_bootstrap.cc:654] T 4e0d833302ff456e96ab09aa12d4c5ba P 27fddeff55ea476ebe7bd4881ae32035: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:17.791755 22766 raft_consensus.cc:374] T 6d6e21b2d7b746469c42c6d80ad8e001 P 8586928e65c543c2b99cebf1dd7d700f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:17.792095 22766 raft_consensus.cc:729] T 6d6e21b2d7b746469c42c6d80ad8e001 P 8586928e65c543c2b99cebf1dd7d700f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8586928e65c543c2b99cebf1dd7d700f, State: Initialized, Role: FOLLOWER
I20260809 15:45:17.792784 22764 tablet_bootstrap.cc:492] T 6d6e21b2d7b746469c42c6d80ad8e001 P cab341909d314988b6893daac17235e7: No bootstrap required, opened a new log
I20260809 15:45:17.793345 22764 ts_tablet_manager.cc:1397] T 6d6e21b2d7b746469c42c6d80ad8e001 P cab341909d314988b6893daac17235e7: Time spent bootstrapping tablet: real 0.043s user 0.033s sys 0.001s
I20260809 15:45:17.792819 22766 consensus_queue.cc:260] T 6d6e21b2d7b746469c42c6d80ad8e001 P 8586928e65c543c2b99cebf1dd7d700f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } } peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } }
I20260809 15:45:17.795406 22766 ts_tablet_manager.cc:1428] T 6d6e21b2d7b746469c42c6d80ad8e001 P 8586928e65c543c2b99cebf1dd7d700f: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260809 15:45:17.796276 22766 tablet_bootstrap.cc:492] T 4e0d833302ff456e96ab09aa12d4c5ba P 8586928e65c543c2b99cebf1dd7d700f: Bootstrap starting.
I20260809 15:45:17.796003 22764 raft_consensus.cc:348] T 6d6e21b2d7b746469c42c6d80ad8e001 P cab341909d314988b6893daac17235e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } } peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } }
I20260809 15:45:17.796746 22764 raft_consensus.cc:374] T 6d6e21b2d7b746469c42c6d80ad8e001 P cab341909d314988b6893daac17235e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:17.797075 22764 raft_consensus.cc:729] T 6d6e21b2d7b746469c42c6d80ad8e001 P cab341909d314988b6893daac17235e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cab341909d314988b6893daac17235e7, State: Initialized, Role: FOLLOWER
I20260809 15:45:17.797842 22764 consensus_queue.cc:260] T 6d6e21b2d7b746469c42c6d80ad8e001 P cab341909d314988b6893daac17235e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } } peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } }
I20260809 15:45:17.802426 22764 ts_tablet_manager.cc:1428] T 6d6e21b2d7b746469c42c6d80ad8e001 P cab341909d314988b6893daac17235e7: Time spent starting tablet: real 0.009s user 0.005s sys 0.001s
I20260809 15:45:17.802928 22765 tablet_bootstrap.cc:492] T 4e0d833302ff456e96ab09aa12d4c5ba P 27fddeff55ea476ebe7bd4881ae32035: No bootstrap required, opened a new log
I20260809 15:45:17.803300 22764 tablet_bootstrap.cc:492] T 8c8c4f25c5364a919da21f5fca365c45 P cab341909d314988b6893daac17235e7: Bootstrap starting.
I20260809 15:45:17.803392 22765 ts_tablet_manager.cc:1397] T 4e0d833302ff456e96ab09aa12d4c5ba P 27fddeff55ea476ebe7bd4881ae32035: Time spent bootstrapping tablet: real 0.018s user 0.013s sys 0.003s
I20260809 15:45:17.803212 22243 catalog_manager.cc:5495] T 21aa1094e1cf4d40b51e5916cb6a2a96 P 8586928e65c543c2b99cebf1dd7d700f reported cstate change: term changed from 0 to 1, leader changed from <none> to 8586928e65c543c2b99cebf1dd7d700f (127.20.202.66). New cstate: current_term: 1 leader_uuid: "8586928e65c543c2b99cebf1dd7d700f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } health_report { overall_health: UNKNOWN } } }
I20260809 15:45:17.806106 22765 raft_consensus.cc:348] T 4e0d833302ff456e96ab09aa12d4c5ba P 27fddeff55ea476ebe7bd4881ae32035 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } } peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } }
I20260809 15:45:17.806906 22765 raft_consensus.cc:374] T 4e0d833302ff456e96ab09aa12d4c5ba P 27fddeff55ea476ebe7bd4881ae32035 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:17.807188 22765 raft_consensus.cc:729] T 4e0d833302ff456e96ab09aa12d4c5ba P 27fddeff55ea476ebe7bd4881ae32035 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27fddeff55ea476ebe7bd4881ae32035, State: Initialized, Role: FOLLOWER
I20260809 15:45:17.809005 22764 tablet_bootstrap.cc:654] T 8c8c4f25c5364a919da21f5fca365c45 P cab341909d314988b6893daac17235e7: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:17.809376 22766 tablet_bootstrap.cc:654] T 4e0d833302ff456e96ab09aa12d4c5ba P 8586928e65c543c2b99cebf1dd7d700f: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:17.807912 22765 consensus_queue.cc:260] T 4e0d833302ff456e96ab09aa12d4c5ba P 27fddeff55ea476ebe7bd4881ae32035 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } } peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } }
I20260809 15:45:17.815078 22765 ts_tablet_manager.cc:1428] T 4e0d833302ff456e96ab09aa12d4c5ba P 27fddeff55ea476ebe7bd4881ae32035: Time spent starting tablet: real 0.011s user 0.005s sys 0.000s
I20260809 15:45:17.816005 22765 tablet_bootstrap.cc:492] T 36cb01872dee4485b764d163e0bf4e20 P 27fddeff55ea476ebe7bd4881ae32035: Bootstrap starting.
I20260809 15:45:17.821607 22765 tablet_bootstrap.cc:654] T 36cb01872dee4485b764d163e0bf4e20 P 27fddeff55ea476ebe7bd4881ae32035: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:17.820817 22242 catalog_manager.cc:5495] T 6eb2494806b34a2bad3215770bcb4c7c P 8586928e65c543c2b99cebf1dd7d700f reported cstate change: term changed from 0 to 1, leader changed from <none> to 8586928e65c543c2b99cebf1dd7d700f (127.20.202.66). New cstate: current_term: 1 leader_uuid: "8586928e65c543c2b99cebf1dd7d700f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } health_report { overall_health: UNKNOWN } } }
I20260809 15:45:17.832867 22766 tablet_bootstrap.cc:492] T 4e0d833302ff456e96ab09aa12d4c5ba P 8586928e65c543c2b99cebf1dd7d700f: No bootstrap required, opened a new log
I20260809 15:45:17.833290 22766 ts_tablet_manager.cc:1397] T 4e0d833302ff456e96ab09aa12d4c5ba P 8586928e65c543c2b99cebf1dd7d700f: Time spent bootstrapping tablet: real 0.037s user 0.021s sys 0.006s
I20260809 15:45:17.834199 22765 tablet_bootstrap.cc:492] T 36cb01872dee4485b764d163e0bf4e20 P 27fddeff55ea476ebe7bd4881ae32035: No bootstrap required, opened a new log
I20260809 15:45:17.834604 22765 ts_tablet_manager.cc:1397] T 36cb01872dee4485b764d163e0bf4e20 P 27fddeff55ea476ebe7bd4881ae32035: Time spent bootstrapping tablet: real 0.019s user 0.011s sys 0.004s
I20260809 15:45:17.835768 22766 raft_consensus.cc:348] T 4e0d833302ff456e96ab09aa12d4c5ba P 8586928e65c543c2b99cebf1dd7d700f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } } peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } }
I20260809 15:45:17.836334 22764 tablet_bootstrap.cc:492] T 8c8c4f25c5364a919da21f5fca365c45 P cab341909d314988b6893daac17235e7: No bootstrap required, opened a new log
I20260809 15:45:17.836467 22766 raft_consensus.cc:374] T 4e0d833302ff456e96ab09aa12d4c5ba P 8586928e65c543c2b99cebf1dd7d700f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:17.836736 22764 ts_tablet_manager.cc:1397] T 8c8c4f25c5364a919da21f5fca365c45 P cab341909d314988b6893daac17235e7: Time spent bootstrapping tablet: real 0.034s user 0.008s sys 0.008s
I20260809 15:45:17.836808 22766 raft_consensus.cc:729] T 4e0d833302ff456e96ab09aa12d4c5ba P 8586928e65c543c2b99cebf1dd7d700f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8586928e65c543c2b99cebf1dd7d700f, State: Initialized, Role: FOLLOWER
I20260809 15:45:17.836876 22765 raft_consensus.cc:348] T 36cb01872dee4485b764d163e0bf4e20 P 27fddeff55ea476ebe7bd4881ae32035 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } } peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } }
I20260809 15:45:17.837671 22765 raft_consensus.cc:374] T 36cb01872dee4485b764d163e0bf4e20 P 27fddeff55ea476ebe7bd4881ae32035 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:17.837976 22765 raft_consensus.cc:729] T 36cb01872dee4485b764d163e0bf4e20 P 27fddeff55ea476ebe7bd4881ae32035 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27fddeff55ea476ebe7bd4881ae32035, State: Initialized, Role: FOLLOWER
I20260809 15:45:17.837591 22766 consensus_queue.cc:260] T 4e0d833302ff456e96ab09aa12d4c5ba P 8586928e65c543c2b99cebf1dd7d700f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } } peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } }
I20260809 15:45:17.838613 22765 consensus_queue.cc:260] T 36cb01872dee4485b764d163e0bf4e20 P 27fddeff55ea476ebe7bd4881ae32035 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } } peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } }
I20260809 15:45:17.838830 22764 raft_consensus.cc:348] T 8c8c4f25c5364a919da21f5fca365c45 P cab341909d314988b6893daac17235e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } } peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } }
I20260809 15:45:17.839535 22764 raft_consensus.cc:374] T 8c8c4f25c5364a919da21f5fca365c45 P cab341909d314988b6893daac17235e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:17.839702 22766 ts_tablet_manager.cc:1428] T 4e0d833302ff456e96ab09aa12d4c5ba P 8586928e65c543c2b99cebf1dd7d700f: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260809 15:45:17.839833 22764 raft_consensus.cc:729] T 8c8c4f25c5364a919da21f5fca365c45 P cab341909d314988b6893daac17235e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cab341909d314988b6893daac17235e7, State: Initialized, Role: FOLLOWER
I20260809 15:45:17.840572 22766 tablet_bootstrap.cc:492] T 36cb01872dee4485b764d163e0bf4e20 P 8586928e65c543c2b99cebf1dd7d700f: Bootstrap starting.
I20260809 15:45:17.840485 22764 consensus_queue.cc:260] T 8c8c4f25c5364a919da21f5fca365c45 P cab341909d314988b6893daac17235e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } } peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } }
I20260809 15:45:17.844619 22765 ts_tablet_manager.cc:1428] T 36cb01872dee4485b764d163e0bf4e20 P 27fddeff55ea476ebe7bd4881ae32035: Time spent starting tablet: real 0.010s user 0.007s sys 0.000s
I20260809 15:45:17.845475 22765 tablet_bootstrap.cc:492] T 9b4926c9270e4754bac26cc4c3da4cf9 P 27fddeff55ea476ebe7bd4881ae32035: Bootstrap starting.
I20260809 15:45:17.846207 22766 tablet_bootstrap.cc:654] T 36cb01872dee4485b764d163e0bf4e20 P 8586928e65c543c2b99cebf1dd7d700f: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:17.850807 22765 tablet_bootstrap.cc:654] T 9b4926c9270e4754bac26cc4c3da4cf9 P 27fddeff55ea476ebe7bd4881ae32035: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:17.853089 22796 raft_consensus.cc:482] T 6b1f47a4af1b4480ab2e69adb8be3cc3 P 8586928e65c543c2b99cebf1dd7d700f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:45:17.853499 22796 raft_consensus.cc:504] T 6b1f47a4af1b4480ab2e69adb8be3cc3 P 8586928e65c543c2b99cebf1dd7d700f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } } peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } }
I20260809 15:45:17.855355 22796 leader_election.cc:288] T 6b1f47a4af1b4480ab2e69adb8be3cc3 P 8586928e65c543c2b99cebf1dd7d700f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27fddeff55ea476ebe7bd4881ae32035 (127.20.202.67:33583), cab341909d314988b6893daac17235e7 (127.20.202.65:46511)
I20260809 15:45:17.855420 22764 ts_tablet_manager.cc:1428] T 8c8c4f25c5364a919da21f5fca365c45 P cab341909d314988b6893daac17235e7: Time spent starting tablet: real 0.018s user 0.004s sys 0.001s
I20260809 15:45:17.856352 22764 tablet_bootstrap.cc:492] T 9b4926c9270e4754bac26cc4c3da4cf9 P cab341909d314988b6893daac17235e7: Bootstrap starting.
I20260809 15:45:17.856873 22689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b1f47a4af1b4480ab2e69adb8be3cc3" candidate_uuid: "8586928e65c543c2b99cebf1dd7d700f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27fddeff55ea476ebe7bd4881ae32035" is_pre_election: true
W20260809 15:45:17.858470 22493 leader_election.cc:341] T 6b1f47a4af1b4480ab2e69adb8be3cc3 P 8586928e65c543c2b99cebf1dd7d700f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 27fddeff55ea476ebe7bd4881ae32035 (127.20.202.67:33583): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:17.861074 22436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b1f47a4af1b4480ab2e69adb8be3cc3" candidate_uuid: "8586928e65c543c2b99cebf1dd7d700f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab341909d314988b6893daac17235e7" is_pre_election: true
I20260809 15:45:17.861527 22436 raft_consensus.cc:2454] T 6b1f47a4af1b4480ab2e69adb8be3cc3 P cab341909d314988b6893daac17235e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8586928e65c543c2b99cebf1dd7d700f in term 0.
I20260809 15:45:17.862450 22492 leader_election.cc:302] T 6b1f47a4af1b4480ab2e69adb8be3cc3 P 8586928e65c543c2b99cebf1dd7d700f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8586928e65c543c2b99cebf1dd7d700f, cab341909d314988b6893daac17235e7; no voters: 27fddeff55ea476ebe7bd4881ae32035
I20260809 15:45:17.863566 22796 raft_consensus.cc:2781] T 6b1f47a4af1b4480ab2e69adb8be3cc3 P 8586928e65c543c2b99cebf1dd7d700f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 15:45:17.863893 22796 raft_consensus.cc:482] T 6b1f47a4af1b4480ab2e69adb8be3cc3 P 8586928e65c543c2b99cebf1dd7d700f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:45:17.864202 22796 raft_consensus.cc:3037] T 6b1f47a4af1b4480ab2e69adb8be3cc3 P 8586928e65c543c2b99cebf1dd7d700f [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:17.867594 22764 tablet_bootstrap.cc:654] T 9b4926c9270e4754bac26cc4c3da4cf9 P cab341909d314988b6893daac17235e7: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:17.870644 22796 raft_consensus.cc:504] T 6b1f47a4af1b4480ab2e69adb8be3cc3 P 8586928e65c543c2b99cebf1dd7d700f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } } peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } }
I20260809 15:45:17.873198 22765 tablet_bootstrap.cc:492] T 9b4926c9270e4754bac26cc4c3da4cf9 P 27fddeff55ea476ebe7bd4881ae32035: No bootstrap required, opened a new log
I20260809 15:45:17.873446 22796 leader_election.cc:288] T 6b1f47a4af1b4480ab2e69adb8be3cc3 P 8586928e65c543c2b99cebf1dd7d700f [CANDIDATE]: Term 1 election: Requested vote from peers 27fddeff55ea476ebe7bd4881ae32035 (127.20.202.67:33583), cab341909d314988b6893daac17235e7 (127.20.202.65:46511)
I20260809 15:45:17.873549 22764 tablet_bootstrap.cc:492] T 9b4926c9270e4754bac26cc4c3da4cf9 P cab341909d314988b6893daac17235e7: No bootstrap required, opened a new log
I20260809 15:45:17.873618 22765 ts_tablet_manager.cc:1397] T 9b4926c9270e4754bac26cc4c3da4cf9 P 27fddeff55ea476ebe7bd4881ae32035: Time spent bootstrapping tablet: real 0.028s user 0.013s sys 0.013s
I20260809 15:45:17.874014 22764 ts_tablet_manager.cc:1397] T 9b4926c9270e4754bac26cc4c3da4cf9 P cab341909d314988b6893daac17235e7: Time spent bootstrapping tablet: real 0.018s user 0.009s sys 0.001s
I20260809 15:45:17.874478 22689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b1f47a4af1b4480ab2e69adb8be3cc3" candidate_uuid: "8586928e65c543c2b99cebf1dd7d700f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27fddeff55ea476ebe7bd4881ae32035"
I20260809 15:45:17.875244 22766 tablet_bootstrap.cc:492] T 36cb01872dee4485b764d163e0bf4e20 P 8586928e65c543c2b99cebf1dd7d700f: No bootstrap required, opened a new log
I20260809 15:45:17.875669 22766 ts_tablet_manager.cc:1397] T 36cb01872dee4485b764d163e0bf4e20 P 8586928e65c543c2b99cebf1dd7d700f: Time spent bootstrapping tablet: real 0.035s user 0.019s sys 0.013s
W20260809 15:45:17.876223 22493 leader_election.cc:341] T 6b1f47a4af1b4480ab2e69adb8be3cc3 P 8586928e65c543c2b99cebf1dd7d700f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 27fddeff55ea476ebe7bd4881ae32035 (127.20.202.67:33583): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:17.876313 22765 raft_consensus.cc:348] T 9b4926c9270e4754bac26cc4c3da4cf9 P 27fddeff55ea476ebe7bd4881ae32035 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } } peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } }
I20260809 15:45:17.876943 22765 raft_consensus.cc:374] T 9b4926c9270e4754bac26cc4c3da4cf9 P 27fddeff55ea476ebe7bd4881ae32035 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:17.876610 22764 raft_consensus.cc:348] T 9b4926c9270e4754bac26cc4c3da4cf9 P cab341909d314988b6893daac17235e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } } peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } }
I20260809 15:45:17.877033 22436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b1f47a4af1b4480ab2e69adb8be3cc3" candidate_uuid: "8586928e65c543c2b99cebf1dd7d700f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab341909d314988b6893daac17235e7"
I20260809 15:45:17.877205 22765 raft_consensus.cc:729] T 9b4926c9270e4754bac26cc4c3da4cf9 P 27fddeff55ea476ebe7bd4881ae32035 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27fddeff55ea476ebe7bd4881ae32035, State: Initialized, Role: FOLLOWER
I20260809 15:45:17.877293 22764 raft_consensus.cc:374] T 9b4926c9270e4754bac26cc4c3da4cf9 P cab341909d314988b6893daac17235e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:17.877641 22436 raft_consensus.cc:3037] T 6b1f47a4af1b4480ab2e69adb8be3cc3 P cab341909d314988b6893daac17235e7 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:17.877753 22764 raft_consensus.cc:729] T 9b4926c9270e4754bac26cc4c3da4cf9 P cab341909d314988b6893daac17235e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cab341909d314988b6893daac17235e7, State: Initialized, Role: FOLLOWER
I20260809 15:45:17.877800 22765 consensus_queue.cc:260] T 9b4926c9270e4754bac26cc4c3da4cf9 P 27fddeff55ea476ebe7bd4881ae32035 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } } peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } }
I20260809 15:45:17.878070 22766 raft_consensus.cc:348] T 36cb01872dee4485b764d163e0bf4e20 P 8586928e65c543c2b99cebf1dd7d700f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } } peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } }
I20260809 15:45:17.879853 22765 ts_tablet_manager.cc:1428] T 9b4926c9270e4754bac26cc4c3da4cf9 P 27fddeff55ea476ebe7bd4881ae32035: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260809 15:45:17.883596 22766 raft_consensus.cc:374] T 36cb01872dee4485b764d163e0bf4e20 P 8586928e65c543c2b99cebf1dd7d700f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:17.884032 22766 raft_consensus.cc:729] T 36cb01872dee4485b764d163e0bf4e20 P 8586928e65c543c2b99cebf1dd7d700f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8586928e65c543c2b99cebf1dd7d700f, State: Initialized, Role: FOLLOWER
I20260809 15:45:17.878669 22764 consensus_queue.cc:260] T 9b4926c9270e4754bac26cc4c3da4cf9 P cab341909d314988b6893daac17235e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } } peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } }
I20260809 15:45:17.884435 22765 tablet_bootstrap.cc:492] T 6b1f47a4af1b4480ab2e69adb8be3cc3 P 27fddeff55ea476ebe7bd4881ae32035: Bootstrap starting.
I20260809 15:45:17.884727 22766 consensus_queue.cc:260] T 36cb01872dee4485b764d163e0bf4e20 P 8586928e65c543c2b99cebf1dd7d700f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } } peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } }
I20260809 15:45:17.888710 22766 ts_tablet_manager.cc:1428] T 36cb01872dee4485b764d163e0bf4e20 P 8586928e65c543c2b99cebf1dd7d700f: Time spent starting tablet: real 0.013s user 0.002s sys 0.008s
I20260809 15:45:17.889330 22765 tablet_bootstrap.cc:654] T 6b1f47a4af1b4480ab2e69adb8be3cc3 P 27fddeff55ea476ebe7bd4881ae32035: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:17.889544 22766 tablet_bootstrap.cc:492] T 1154606dbfd849c7833c096541e8c58f P 8586928e65c543c2b99cebf1dd7d700f: Bootstrap starting.
I20260809 15:45:17.894321 22764 ts_tablet_manager.cc:1428] T 9b4926c9270e4754bac26cc4c3da4cf9 P cab341909d314988b6893daac17235e7: Time spent starting tablet: real 0.020s user 0.005s sys 0.000s
I20260809 15:45:17.895056 22766 tablet_bootstrap.cc:654] T 1154606dbfd849c7833c096541e8c58f P 8586928e65c543c2b99cebf1dd7d700f: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:17.897121 22781 raft_consensus.cc:482] T 8c8c4f25c5364a919da21f5fca365c45 P cab341909d314988b6893daac17235e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:45:17.897584 22781 raft_consensus.cc:504] T 8c8c4f25c5364a919da21f5fca365c45 P cab341909d314988b6893daac17235e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } } peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } }
I20260809 15:45:17.899828 22765 tablet_bootstrap.cc:492] T 6b1f47a4af1b4480ab2e69adb8be3cc3 P 27fddeff55ea476ebe7bd4881ae32035: No bootstrap required, opened a new log
I20260809 15:45:17.899969 22781 leader_election.cc:288] T 8c8c4f25c5364a919da21f5fca365c45 P cab341909d314988b6893daac17235e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8586928e65c543c2b99cebf1dd7d700f (127.20.202.66:45399), 27fddeff55ea476ebe7bd4881ae32035 (127.20.202.67:33583)
I20260809 15:45:17.900286 22765 ts_tablet_manager.cc:1397] T 6b1f47a4af1b4480ab2e69adb8be3cc3 P 27fddeff55ea476ebe7bd4881ae32035: Time spent bootstrapping tablet: real 0.016s user 0.010s sys 0.005s
I20260809 15:45:17.902447 22765 raft_consensus.cc:348] T 6b1f47a4af1b4480ab2e69adb8be3cc3 P 27fddeff55ea476ebe7bd4881ae32035 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } } peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } }
I20260809 15:45:17.903075 22765 raft_consensus.cc:374] T 6b1f47a4af1b4480ab2e69adb8be3cc3 P 27fddeff55ea476ebe7bd4881ae32035 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:17.903340 22765 raft_consensus.cc:729] T 6b1f47a4af1b4480ab2e69adb8be3cc3 P 27fddeff55ea476ebe7bd4881ae32035 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27fddeff55ea476ebe7bd4881ae32035, State: Initialized, Role: FOLLOWER
I20260809 15:45:17.904053 22436 raft_consensus.cc:2454] T 6b1f47a4af1b4480ab2e69adb8be3cc3 P cab341909d314988b6893daac17235e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8586928e65c543c2b99cebf1dd7d700f in term 1.
I20260809 15:45:17.903986 22765 consensus_queue.cc:260] T 6b1f47a4af1b4480ab2e69adb8be3cc3 P 27fddeff55ea476ebe7bd4881ae32035 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } } peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } }
I20260809 15:45:17.906098 22765 ts_tablet_manager.cc:1428] T 6b1f47a4af1b4480ab2e69adb8be3cc3 P 27fddeff55ea476ebe7bd4881ae32035: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260809 15:45:17.907893 22765 tablet_bootstrap.cc:492] T 1154606dbfd849c7833c096541e8c58f P 27fddeff55ea476ebe7bd4881ae32035: Bootstrap starting.
I20260809 15:45:17.912515 22765 tablet_bootstrap.cc:654] T 1154606dbfd849c7833c096541e8c58f P 27fddeff55ea476ebe7bd4881ae32035: Neither blocks nor log segments found. Creating new log.
W20260809 15:45:17.919881 22321 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260809 15:45:17.919926 22764 tablet_bootstrap.cc:492] T 6eb2494806b34a2bad3215770bcb4c7c P cab341909d314988b6893daac17235e7: Bootstrap starting.
I20260809 15:45:17.924486 22772 raft_consensus.cc:482] T 8c8c4f25c5364a919da21f5fca365c45 P 27fddeff55ea476ebe7bd4881ae32035 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:45:17.924960 22772 raft_consensus.cc:504] T 8c8c4f25c5364a919da21f5fca365c45 P 27fddeff55ea476ebe7bd4881ae32035 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } } peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } }
I20260809 15:45:17.929675 22796 raft_consensus.cc:482] T 4e0d833302ff456e96ab09aa12d4c5ba P 8586928e65c543c2b99cebf1dd7d700f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:45:17.930121 22796 raft_consensus.cc:504] T 4e0d833302ff456e96ab09aa12d4c5ba P 8586928e65c543c2b99cebf1dd7d700f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } } peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } }
I20260809 15:45:17.930867 22772 leader_election.cc:288] T 8c8c4f25c5364a919da21f5fca365c45 P 27fddeff55ea476ebe7bd4881ae32035 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cab341909d314988b6893daac17235e7 (127.20.202.65:46511), 8586928e65c543c2b99cebf1dd7d700f (127.20.202.66:45399)
I20260809 15:45:17.931982 22796 leader_election.cc:288] T 4e0d833302ff456e96ab09aa12d4c5ba P 8586928e65c543c2b99cebf1dd7d700f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cab341909d314988b6893daac17235e7 (127.20.202.65:46511), 27fddeff55ea476ebe7bd4881ae32035 (127.20.202.67:33583)
I20260809 15:45:17.936590 22764 tablet_bootstrap.cc:654] T 6eb2494806b34a2bad3215770bcb4c7c P cab341909d314988b6893daac17235e7: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:17.941926 22492 leader_election.cc:302] T 6b1f47a4af1b4480ab2e69adb8be3cc3 P 8586928e65c543c2b99cebf1dd7d700f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8586928e65c543c2b99cebf1dd7d700f, cab341909d314988b6893daac17235e7; no voters: 27fddeff55ea476ebe7bd4881ae32035
I20260809 15:45:17.943084 22796 raft_consensus.cc:2781] T 6b1f47a4af1b4480ab2e69adb8be3cc3 P 8586928e65c543c2b99cebf1dd7d700f [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:45:17.943571 22796 raft_consensus.cc:686] T 6b1f47a4af1b4480ab2e69adb8be3cc3 P 8586928e65c543c2b99cebf1dd7d700f [term 1 LEADER]: Becoming Leader. State: Replica: 8586928e65c543c2b99cebf1dd7d700f, State: Running, Role: LEADER
I20260809 15:45:17.944365 22796 consensus_queue.cc:237] T 6b1f47a4af1b4480ab2e69adb8be3cc3 P 8586928e65c543c2b99cebf1dd7d700f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } } peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } }
I20260809 15:45:17.948508 22436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e0d833302ff456e96ab09aa12d4c5ba" candidate_uuid: "8586928e65c543c2b99cebf1dd7d700f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab341909d314988b6893daac17235e7" is_pre_election: true
I20260809 15:45:17.951464 22766 tablet_bootstrap.cc:492] T 1154606dbfd849c7833c096541e8c58f P 8586928e65c543c2b99cebf1dd7d700f: No bootstrap required, opened a new log
I20260809 15:45:17.951949 22766 ts_tablet_manager.cc:1397] T 1154606dbfd849c7833c096541e8c58f P 8586928e65c543c2b99cebf1dd7d700f: Time spent bootstrapping tablet: real 0.063s user 0.027s sys 0.025s
W20260809 15:45:17.952421 22492 leader_election.cc:341] T 4e0d833302ff456e96ab09aa12d4c5ba P 8586928e65c543c2b99cebf1dd7d700f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cab341909d314988b6893daac17235e7 (127.20.202.65:46511): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:17.954509 22766 raft_consensus.cc:348] T 1154606dbfd849c7833c096541e8c58f P 8586928e65c543c2b99cebf1dd7d700f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } } peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } }
I20260809 15:45:17.955197 22766 raft_consensus.cc:374] T 1154606dbfd849c7833c096541e8c58f P 8586928e65c543c2b99cebf1dd7d700f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:17.955477 22766 raft_consensus.cc:729] T 1154606dbfd849c7833c096541e8c58f P 8586928e65c543c2b99cebf1dd7d700f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8586928e65c543c2b99cebf1dd7d700f, State: Initialized, Role: FOLLOWER
I20260809 15:45:17.955652 22242 catalog_manager.cc:5495] T 6b1f47a4af1b4480ab2e69adb8be3cc3 P 8586928e65c543c2b99cebf1dd7d700f reported cstate change: term changed from 0 to 1, leader changed from <none> to 8586928e65c543c2b99cebf1dd7d700f (127.20.202.66). New cstate: current_term: 1 leader_uuid: "8586928e65c543c2b99cebf1dd7d700f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } health_report { overall_health: UNKNOWN } } }
I20260809 15:45:17.956113 22766 consensus_queue.cc:260] T 1154606dbfd849c7833c096541e8c58f P 8586928e65c543c2b99cebf1dd7d700f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } } peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } }
I20260809 15:45:17.960104 22556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c8c4f25c5364a919da21f5fca365c45" candidate_uuid: "cab341909d314988b6893daac17235e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8586928e65c543c2b99cebf1dd7d700f" is_pre_election: true
W20260809 15:45:17.962109 22361 leader_election.cc:341] T 8c8c4f25c5364a919da21f5fca365c45 P cab341909d314988b6893daac17235e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8586928e65c543c2b99cebf1dd7d700f (127.20.202.66:45399): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:17.962670 22766 ts_tablet_manager.cc:1428] T 1154606dbfd849c7833c096541e8c58f P 8586928e65c543c2b99cebf1dd7d700f: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
I20260809 15:45:17.963250 22766 tablet_bootstrap.cc:492] T ab0d7bb4b86540b28101d1c813ee214e P 8586928e65c543c2b99cebf1dd7d700f: Bootstrap starting.
I20260809 15:45:17.965297 22765 tablet_bootstrap.cc:492] T 1154606dbfd849c7833c096541e8c58f P 27fddeff55ea476ebe7bd4881ae32035: No bootstrap required, opened a new log
I20260809 15:45:17.965734 22765 ts_tablet_manager.cc:1397] T 1154606dbfd849c7833c096541e8c58f P 27fddeff55ea476ebe7bd4881ae32035: Time spent bootstrapping tablet: real 0.058s user 0.011s sys 0.030s
I20260809 15:45:17.967845 22766 tablet_bootstrap.cc:654] T ab0d7bb4b86540b28101d1c813ee214e P 8586928e65c543c2b99cebf1dd7d700f: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:17.969878 22765 raft_consensus.cc:348] T 1154606dbfd849c7833c096541e8c58f P 27fddeff55ea476ebe7bd4881ae32035 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } } peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } }
I20260809 15:45:17.970438 22765 raft_consensus.cc:374] T 1154606dbfd849c7833c096541e8c58f P 27fddeff55ea476ebe7bd4881ae32035 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:17.970634 22765 raft_consensus.cc:729] T 1154606dbfd849c7833c096541e8c58f P 27fddeff55ea476ebe7bd4881ae32035 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27fddeff55ea476ebe7bd4881ae32035, State: Initialized, Role: FOLLOWER
I20260809 15:45:17.971144 22765 consensus_queue.cc:260] T 1154606dbfd849c7833c096541e8c58f P 27fddeff55ea476ebe7bd4881ae32035 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } } peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } }
I20260809 15:45:17.973001 22765 ts_tablet_manager.cc:1428] T 1154606dbfd849c7833c096541e8c58f P 27fddeff55ea476ebe7bd4881ae32035: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260809 15:45:17.975678 22764 tablet_bootstrap.cc:492] T 6eb2494806b34a2bad3215770bcb4c7c P cab341909d314988b6893daac17235e7: No bootstrap required, opened a new log
I20260809 15:45:17.976121 22772 raft_consensus.cc:482] T 9b4926c9270e4754bac26cc4c3da4cf9 P 27fddeff55ea476ebe7bd4881ae32035 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:45:17.977487 22772 raft_consensus.cc:504] T 9b4926c9270e4754bac26cc4c3da4cf9 P 27fddeff55ea476ebe7bd4881ae32035 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } } peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } }
I20260809 15:45:17.979295 22772 leader_election.cc:288] T 9b4926c9270e4754bac26cc4c3da4cf9 P 27fddeff55ea476ebe7bd4881ae32035 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cab341909d314988b6893daac17235e7 (127.20.202.65:46511), 8586928e65c543c2b99cebf1dd7d700f (127.20.202.66:45399)
I20260809 15:45:17.976823 22689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e0d833302ff456e96ab09aa12d4c5ba" candidate_uuid: "8586928e65c543c2b99cebf1dd7d700f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27fddeff55ea476ebe7bd4881ae32035" is_pre_election: true
I20260809 15:45:17.979969 22689 raft_consensus.cc:2454] T 4e0d833302ff456e96ab09aa12d4c5ba P 27fddeff55ea476ebe7bd4881ae32035 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8586928e65c543c2b99cebf1dd7d700f in term 0.
I20260809 15:45:17.981279 22493 leader_election.cc:302] T 4e0d833302ff456e96ab09aa12d4c5ba P 8586928e65c543c2b99cebf1dd7d700f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27fddeff55ea476ebe7bd4881ae32035, 8586928e65c543c2b99cebf1dd7d700f; no voters: cab341909d314988b6893daac17235e7
I20260809 15:45:17.982313 22796 raft_consensus.cc:2781] T 4e0d833302ff456e96ab09aa12d4c5ba P 8586928e65c543c2b99cebf1dd7d700f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 15:45:17.982640 22796 raft_consensus.cc:482] T 4e0d833302ff456e96ab09aa12d4c5ba P 8586928e65c543c2b99cebf1dd7d700f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:45:17.982935 22796 raft_consensus.cc:3037] T 4e0d833302ff456e96ab09aa12d4c5ba P 8586928e65c543c2b99cebf1dd7d700f [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:17.987468 22781 raft_consensus.cc:482] T 1154606dbfd849c7833c096541e8c58f P cab341909d314988b6893daac17235e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:45:17.987880 22781 raft_consensus.cc:504] T 1154606dbfd849c7833c096541e8c58f P cab341909d314988b6893daac17235e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } } peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } }
I20260809 15:45:17.989245 22796 raft_consensus.cc:504] T 4e0d833302ff456e96ab09aa12d4c5ba P 8586928e65c543c2b99cebf1dd7d700f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } } peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } }
I20260809 15:45:17.989687 22781 leader_election.cc:288] T 1154606dbfd849c7833c096541e8c58f P cab341909d314988b6893daac17235e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8586928e65c543c2b99cebf1dd7d700f (127.20.202.66:45399), 27fddeff55ea476ebe7bd4881ae32035 (127.20.202.67:33583)
I20260809 15:45:17.990945 22556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1154606dbfd849c7833c096541e8c58f" candidate_uuid: "cab341909d314988b6893daac17235e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8586928e65c543c2b99cebf1dd7d700f" is_pre_election: true
I20260809 15:45:17.991539 22556 raft_consensus.cc:2454] T 1154606dbfd849c7833c096541e8c58f P 8586928e65c543c2b99cebf1dd7d700f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab341909d314988b6893daac17235e7 in term 0.
I20260809 15:45:17.992161 22796 leader_election.cc:288] T 4e0d833302ff456e96ab09aa12d4c5ba P 8586928e65c543c2b99cebf1dd7d700f [CANDIDATE]: Term 1 election: Requested vote from peers cab341909d314988b6893daac17235e7 (127.20.202.65:46511), 27fddeff55ea476ebe7bd4881ae32035 (127.20.202.67:33583)
I20260809 15:45:17.992565 22361 leader_election.cc:302] T 1154606dbfd849c7833c096541e8c58f P cab341909d314988b6893daac17235e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8586928e65c543c2b99cebf1dd7d700f, cab341909d314988b6893daac17235e7; no voters:
I20260809 15:45:17.993227 22764 ts_tablet_manager.cc:1397] T 6eb2494806b34a2bad3215770bcb4c7c P cab341909d314988b6893daac17235e7: Time spent bootstrapping tablet: real 0.073s user 0.006s sys 0.015s
I20260809 15:45:17.994139 22436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c8c4f25c5364a919da21f5fca365c45" candidate_uuid: "27fddeff55ea476ebe7bd4881ae32035" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab341909d314988b6893daac17235e7" is_pre_election: true
I20260809 15:45:17.996512 22436 raft_consensus.cc:2454] T 8c8c4f25c5364a919da21f5fca365c45 P cab341909d314988b6893daac17235e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27fddeff55ea476ebe7bd4881ae32035 in term 0.
I20260809 15:45:17.994608 22781 raft_consensus.cc:2781] T 1154606dbfd849c7833c096541e8c58f P cab341909d314988b6893daac17235e7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 15:45:17.997069 22766 tablet_bootstrap.cc:492] T ab0d7bb4b86540b28101d1c813ee214e P 8586928e65c543c2b99cebf1dd7d700f: No bootstrap required, opened a new log
I20260809 15:45:17.997192 22781 raft_consensus.cc:482] T 1154606dbfd849c7833c096541e8c58f P cab341909d314988b6893daac17235e7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:45:17.997494 22766 ts_tablet_manager.cc:1397] T ab0d7bb4b86540b28101d1c813ee214e P 8586928e65c543c2b99cebf1dd7d700f: Time spent bootstrapping tablet: real 0.034s user 0.023s sys 0.007s
I20260809 15:45:17.997520 22781 raft_consensus.cc:3037] T 1154606dbfd849c7833c096541e8c58f P cab341909d314988b6893daac17235e7 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:17.994580 22433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b4926c9270e4754bac26cc4c3da4cf9" candidate_uuid: "27fddeff55ea476ebe7bd4881ae32035" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab341909d314988b6893daac17235e7" is_pre_election: true
I20260809 15:45:17.998528 22764 raft_consensus.cc:348] T 6eb2494806b34a2bad3215770bcb4c7c P cab341909d314988b6893daac17235e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } } peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } }
I20260809 15:45:17.995508 22430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e0d833302ff456e96ab09aa12d4c5ba" candidate_uuid: "8586928e65c543c2b99cebf1dd7d700f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab341909d314988b6893daac17235e7"
I20260809 15:45:17.999225 22764 raft_consensus.cc:374] T 6eb2494806b34a2bad3215770bcb4c7c P cab341909d314988b6893daac17235e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:17.998842 22433 raft_consensus.cc:2454] T 9b4926c9270e4754bac26cc4c3da4cf9 P cab341909d314988b6893daac17235e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27fddeff55ea476ebe7bd4881ae32035 in term 0.
I20260809 15:45:17.999804 22766 raft_consensus.cc:348] T ab0d7bb4b86540b28101d1c813ee214e P 8586928e65c543c2b99cebf1dd7d700f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } } peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } }
I20260809 15:45:18.000429 22766 raft_consensus.cc:374] T ab0d7bb4b86540b28101d1c813ee214e P 8586928e65c543c2b99cebf1dd7d700f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:18.000705 22766 raft_consensus.cc:729] T ab0d7bb4b86540b28101d1c813ee214e P 8586928e65c543c2b99cebf1dd7d700f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8586928e65c543c2b99cebf1dd7d700f, State: Initialized, Role: FOLLOWER
I20260809 15:45:18.000645 22556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c8c4f25c5364a919da21f5fca365c45" candidate_uuid: "27fddeff55ea476ebe7bd4881ae32035" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8586928e65c543c2b99cebf1dd7d700f" is_pre_election: true
I20260809 15:45:18.000746 22689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e0d833302ff456e96ab09aa12d4c5ba" candidate_uuid: "8586928e65c543c2b99cebf1dd7d700f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27fddeff55ea476ebe7bd4881ae32035"
W20260809 15:45:18.000284 22492 leader_election.cc:341] T 4e0d833302ff456e96ab09aa12d4c5ba P 8586928e65c543c2b99cebf1dd7d700f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cab341909d314988b6893daac17235e7 (127.20.202.65:46511): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:18.001408 22766 consensus_queue.cc:260] T ab0d7bb4b86540b28101d1c813ee214e P 8586928e65c543c2b99cebf1dd7d700f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } } peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } }
I20260809 15:45:18.001368 22689 raft_consensus.cc:3037] T 4e0d833302ff456e96ab09aa12d4c5ba P 27fddeff55ea476ebe7bd4881ae32035 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:18.002470 22556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b4926c9270e4754bac26cc4c3da4cf9" candidate_uuid: "27fddeff55ea476ebe7bd4881ae32035" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8586928e65c543c2b99cebf1dd7d700f" is_pre_election: true
I20260809 15:45:18.003021 22556 raft_consensus.cc:2454] T 9b4926c9270e4754bac26cc4c3da4cf9 P 8586928e65c543c2b99cebf1dd7d700f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27fddeff55ea476ebe7bd4881ae32035 in term 0.
I20260809 15:45:18.003350 22766 ts_tablet_manager.cc:1428] T ab0d7bb4b86540b28101d1c813ee214e P 8586928e65c543c2b99cebf1dd7d700f: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
W20260809 15:45:18.004210 22635 leader_election.cc:341] T 8c8c4f25c5364a919da21f5fca365c45 P 27fddeff55ea476ebe7bd4881ae32035 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8586928e65c543c2b99cebf1dd7d700f (127.20.202.66:45399): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:18.004830 22635 leader_election.cc:302] T 9b4926c9270e4754bac26cc4c3da4cf9 P 27fddeff55ea476ebe7bd4881ae32035 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27fddeff55ea476ebe7bd4881ae32035, 8586928e65c543c2b99cebf1dd7d700f; no voters:
I20260809 15:45:18.004660 22781 raft_consensus.cc:504] T 1154606dbfd849c7833c096541e8c58f P cab341909d314988b6893daac17235e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } } peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } }
I20260809 15:45:18.005667 22772 raft_consensus.cc:2781] T 9b4926c9270e4754bac26cc4c3da4cf9 P 27fddeff55ea476ebe7bd4881ae32035 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 15:45:18.005992 22772 raft_consensus.cc:482] T 9b4926c9270e4754bac26cc4c3da4cf9 P 27fddeff55ea476ebe7bd4881ae32035 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:45:18.006319 22772 raft_consensus.cc:3037] T 9b4926c9270e4754bac26cc4c3da4cf9 P 27fddeff55ea476ebe7bd4881ae32035 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:18.006960 22556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1154606dbfd849c7833c096541e8c58f" candidate_uuid: "cab341909d314988b6893daac17235e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8586928e65c543c2b99cebf1dd7d700f"
I20260809 15:45:18.007512 22556 raft_consensus.cc:3037] T 1154606dbfd849c7833c096541e8c58f P 8586928e65c543c2b99cebf1dd7d700f [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:17.999536 22764 raft_consensus.cc:729] T 6eb2494806b34a2bad3215770bcb4c7c P cab341909d314988b6893daac17235e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cab341909d314988b6893daac17235e7, State: Initialized, Role: FOLLOWER
I20260809 15:45:18.008491 22764 consensus_queue.cc:260] T 6eb2494806b34a2bad3215770bcb4c7c P cab341909d314988b6893daac17235e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } } peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } }
I20260809 15:45:18.013093 22690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c8c4f25c5364a919da21f5fca365c45" candidate_uuid: "cab341909d314988b6893daac17235e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27fddeff55ea476ebe7bd4881ae32035" is_pre_election: true
I20260809 15:45:18.013312 22687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1154606dbfd849c7833c096541e8c58f" candidate_uuid: "cab341909d314988b6893daac17235e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27fddeff55ea476ebe7bd4881ae32035" is_pre_election: true
I20260809 15:45:18.013677 22690 raft_consensus.cc:2454] T 8c8c4f25c5364a919da21f5fca365c45 P 27fddeff55ea476ebe7bd4881ae32035 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab341909d314988b6893daac17235e7 in term 0.
I20260809 15:45:18.013974 22687 raft_consensus.cc:2454] T 1154606dbfd849c7833c096541e8c58f P 27fddeff55ea476ebe7bd4881ae32035 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab341909d314988b6893daac17235e7 in term 0.
I20260809 15:45:18.018332 22556 raft_consensus.cc:2454] T 1154606dbfd849c7833c096541e8c58f P 8586928e65c543c2b99cebf1dd7d700f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab341909d314988b6893daac17235e7 in term 1.
I20260809 15:45:18.018795 22360 leader_election.cc:302] T 8c8c4f25c5364a919da21f5fca365c45 P cab341909d314988b6893daac17235e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27fddeff55ea476ebe7bd4881ae32035, cab341909d314988b6893daac17235e7; no voters: 8586928e65c543c2b99cebf1dd7d700f
I20260809 15:45:18.018730 22766 tablet_bootstrap.cc:492] T 8c8c4f25c5364a919da21f5fca365c45 P 8586928e65c543c2b99cebf1dd7d700f: Bootstrap starting.
I20260809 15:45:18.020850 22772 raft_consensus.cc:504] T 9b4926c9270e4754bac26cc4c3da4cf9 P 27fddeff55ea476ebe7bd4881ae32035 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } } peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } }
I20260809 15:45:18.022811 22772 leader_election.cc:288] T 9b4926c9270e4754bac26cc4c3da4cf9 P 27fddeff55ea476ebe7bd4881ae32035 [CANDIDATE]: Term 1 election: Requested vote from peers cab341909d314988b6893daac17235e7 (127.20.202.65:46511), 8586928e65c543c2b99cebf1dd7d700f (127.20.202.66:45399)
I20260809 15:45:18.023653 22772 raft_consensus.cc:482] T 36cb01872dee4485b764d163e0bf4e20 P 27fddeff55ea476ebe7bd4881ae32035 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:45:18.018496 22689 raft_consensus.cc:2454] T 4e0d833302ff456e96ab09aa12d4c5ba P 27fddeff55ea476ebe7bd4881ae32035 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8586928e65c543c2b99cebf1dd7d700f in term 1.
I20260809 15:45:18.006130 22633 leader_election.cc:302] T 8c8c4f25c5364a919da21f5fca365c45 P 27fddeff55ea476ebe7bd4881ae32035 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27fddeff55ea476ebe7bd4881ae32035, cab341909d314988b6893daac17235e7; no voters: 8586928e65c543c2b99cebf1dd7d700f
I20260809 15:45:18.024874 22766 tablet_bootstrap.cc:654] T 8c8c4f25c5364a919da21f5fca365c45 P 8586928e65c543c2b99cebf1dd7d700f: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:18.025983 22772 raft_consensus.cc:504] T 36cb01872dee4485b764d163e0bf4e20 P 27fddeff55ea476ebe7bd4881ae32035 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } } peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } }
I20260809 15:45:18.027900 22361 leader_election.cc:302] T 1154606dbfd849c7833c096541e8c58f P cab341909d314988b6893daac17235e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8586928e65c543c2b99cebf1dd7d700f, cab341909d314988b6893daac17235e7; no voters:
I20260809 15:45:18.028582 22772 leader_election.cc:288] T 36cb01872dee4485b764d163e0bf4e20 P 27fddeff55ea476ebe7bd4881ae32035 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cab341909d314988b6893daac17235e7 (127.20.202.65:46511), 8586928e65c543c2b99cebf1dd7d700f (127.20.202.66:45399)
I20260809 15:45:18.031312 22781 leader_election.cc:288] T 1154606dbfd849c7833c096541e8c58f P cab341909d314988b6893daac17235e7 [CANDIDATE]: Term 1 election: Requested vote from peers 8586928e65c543c2b99cebf1dd7d700f (127.20.202.66:45399), 27fddeff55ea476ebe7bd4881ae32035 (127.20.202.67:33583)
I20260809 15:45:18.031796 22781 raft_consensus.cc:2781] T 8c8c4f25c5364a919da21f5fca365c45 P cab341909d314988b6893daac17235e7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 15:45:18.032117 22781 raft_consensus.cc:482] T 8c8c4f25c5364a919da21f5fca365c45 P cab341909d314988b6893daac17235e7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:45:18.032423 22781 raft_consensus.cc:3037] T 8c8c4f25c5364a919da21f5fca365c45 P cab341909d314988b6893daac17235e7 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:18.029623 22772 raft_consensus.cc:2781] T 8c8c4f25c5364a919da21f5fca365c45 P 27fddeff55ea476ebe7bd4881ae32035 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 15:45:18.035640 22772 raft_consensus.cc:482] T 8c8c4f25c5364a919da21f5fca365c45 P 27fddeff55ea476ebe7bd4881ae32035 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:45:18.035965 22772 raft_consensus.cc:3037] T 8c8c4f25c5364a919da21f5fca365c45 P 27fddeff55ea476ebe7bd4881ae32035 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:18.039999 22493 leader_election.cc:302] T 4e0d833302ff456e96ab09aa12d4c5ba P 8586928e65c543c2b99cebf1dd7d700f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27fddeff55ea476ebe7bd4881ae32035, 8586928e65c543c2b99cebf1dd7d700f; no voters: cab341909d314988b6893daac17235e7
I20260809 15:45:18.040261 22556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b4926c9270e4754bac26cc4c3da4cf9" candidate_uuid: "27fddeff55ea476ebe7bd4881ae32035" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8586928e65c543c2b99cebf1dd7d700f"
I20260809 15:45:18.040812 22556 raft_consensus.cc:3037] T 9b4926c9270e4754bac26cc4c3da4cf9 P 8586928e65c543c2b99cebf1dd7d700f [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:18.041154 22796 raft_consensus.cc:2781] T 4e0d833302ff456e96ab09aa12d4c5ba P 8586928e65c543c2b99cebf1dd7d700f [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:45:18.040952 22781 raft_consensus.cc:504] T 8c8c4f25c5364a919da21f5fca365c45 P cab341909d314988b6893daac17235e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } } peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } }
I20260809 15:45:18.041646 22796 raft_consensus.cc:686] T 4e0d833302ff456e96ab09aa12d4c5ba P 8586928e65c543c2b99cebf1dd7d700f [term 1 LEADER]: Becoming Leader. State: Replica: 8586928e65c543c2b99cebf1dd7d700f, State: Running, Role: LEADER
I20260809 15:45:18.042867 22781 leader_election.cc:288] T 8c8c4f25c5364a919da21f5fca365c45 P cab341909d314988b6893daac17235e7 [CANDIDATE]: Term 1 election: Requested vote from peers 8586928e65c543c2b99cebf1dd7d700f (127.20.202.66:45399), 27fddeff55ea476ebe7bd4881ae32035 (127.20.202.67:33583)
I20260809 15:45:18.042456 22796 consensus_queue.cc:237] T 4e0d833302ff456e96ab09aa12d4c5ba P 8586928e65c543c2b99cebf1dd7d700f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } } peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } }
I20260809 15:45:18.045480 22764 ts_tablet_manager.cc:1428] T 6eb2494806b34a2bad3215770bcb4c7c P cab341909d314988b6893daac17235e7: Time spent starting tablet: real 0.049s user 0.026s sys 0.013s
I20260809 15:45:18.045641 22555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36cb01872dee4485b764d163e0bf4e20" candidate_uuid: "27fddeff55ea476ebe7bd4881ae32035" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8586928e65c543c2b99cebf1dd7d700f" is_pre_election: true
I20260809 15:45:18.046378 22764 tablet_bootstrap.cc:492] T 4e0d833302ff456e96ab09aa12d4c5ba P cab341909d314988b6893daac17235e7: Bootstrap starting.
I20260809 15:45:18.046283 22555 raft_consensus.cc:2454] T 36cb01872dee4485b764d163e0bf4e20 P 8586928e65c543c2b99cebf1dd7d700f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27fddeff55ea476ebe7bd4881ae32035 in term 0.
I20260809 15:45:18.047327 22433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b4926c9270e4754bac26cc4c3da4cf9" candidate_uuid: "27fddeff55ea476ebe7bd4881ae32035" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab341909d314988b6893daac17235e7"
I20260809 15:45:18.051427 22433 raft_consensus.cc:3037] T 9b4926c9270e4754bac26cc4c3da4cf9 P cab341909d314988b6893daac17235e7 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:18.051551 22635 leader_election.cc:302] T 36cb01872dee4485b764d163e0bf4e20 P 27fddeff55ea476ebe7bd4881ae32035 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27fddeff55ea476ebe7bd4881ae32035, 8586928e65c543c2b99cebf1dd7d700f; no voters:
I20260809 15:45:18.056085 22555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c8c4f25c5364a919da21f5fca365c45" candidate_uuid: "cab341909d314988b6893daac17235e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8586928e65c543c2b99cebf1dd7d700f"
W20260809 15:45:18.057338 22361 leader_election.cc:341] T 8c8c4f25c5364a919da21f5fca365c45 P cab341909d314988b6893daac17235e7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8586928e65c543c2b99cebf1dd7d700f (127.20.202.66:45399): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:18.058270 22430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36cb01872dee4485b764d163e0bf4e20" candidate_uuid: "27fddeff55ea476ebe7bd4881ae32035" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab341909d314988b6893daac17235e7" is_pre_election: true
I20260809 15:45:18.053586 22556 raft_consensus.cc:2454] T 9b4926c9270e4754bac26cc4c3da4cf9 P 8586928e65c543c2b99cebf1dd7d700f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27fddeff55ea476ebe7bd4881ae32035 in term 1.
I20260809 15:45:18.058666 22689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1154606dbfd849c7833c096541e8c58f" candidate_uuid: "cab341909d314988b6893daac17235e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27fddeff55ea476ebe7bd4881ae32035"
I20260809 15:45:18.054868 22844 raft_consensus.cc:2781] T 1154606dbfd849c7833c096541e8c58f P cab341909d314988b6893daac17235e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:45:18.059610 22690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c8c4f25c5364a919da21f5fca365c45" candidate_uuid: "cab341909d314988b6893daac17235e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27fddeff55ea476ebe7bd4881ae32035"
I20260809 15:45:18.061069 22244 catalog_manager.cc:5495] T 4e0d833302ff456e96ab09aa12d4c5ba P 8586928e65c543c2b99cebf1dd7d700f reported cstate change: term changed from 0 to 1, leader changed from <none> to 8586928e65c543c2b99cebf1dd7d700f (127.20.202.66). New cstate: current_term: 1 leader_uuid: "8586928e65c543c2b99cebf1dd7d700f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } health_report { overall_health: UNKNOWN } } }
I20260809 15:45:18.065549 22764 tablet_bootstrap.cc:654] T 4e0d833302ff456e96ab09aa12d4c5ba P cab341909d314988b6893daac17235e7: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:18.085144 22689 raft_consensus.cc:3037] T 1154606dbfd849c7833c096541e8c58f P 27fddeff55ea476ebe7bd4881ae32035 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:18.099511 22766 tablet_bootstrap.cc:492] T 8c8c4f25c5364a919da21f5fca365c45 P 8586928e65c543c2b99cebf1dd7d700f: No bootstrap required, opened a new log
I20260809 15:45:18.099936 22766 ts_tablet_manager.cc:1397] T 8c8c4f25c5364a919da21f5fca365c45 P 8586928e65c543c2b99cebf1dd7d700f: Time spent bootstrapping tablet: real 0.081s user 0.007s sys 0.031s
W20260809 15:45:18.087479 22633 leader_election.cc:341] T 36cb01872dee4485b764d163e0bf4e20 P 27fddeff55ea476ebe7bd4881ae32035 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cab341909d314988b6893daac17235e7 (127.20.202.65:46511): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:18.085512 22772 raft_consensus.cc:504] T 8c8c4f25c5364a919da21f5fca365c45 P 27fddeff55ea476ebe7bd4881ae32035 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } } peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } }
I20260809 15:45:18.062156 22433 raft_consensus.cc:2454] T 9b4926c9270e4754bac26cc4c3da4cf9 P cab341909d314988b6893daac17235e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27fddeff55ea476ebe7bd4881ae32035 in term 1.
I20260809 15:45:18.104525 22853 raft_consensus.cc:2781] T 36cb01872dee4485b764d163e0bf4e20 P 27fddeff55ea476ebe7bd4881ae32035 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 15:45:18.104709 22633 leader_election.cc:302] T 9b4926c9270e4754bac26cc4c3da4cf9 P 27fddeff55ea476ebe7bd4881ae32035 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27fddeff55ea476ebe7bd4881ae32035, cab341909d314988b6893daac17235e7; no voters:
I20260809 15:45:18.104929 22853 raft_consensus.cc:482] T 36cb01872dee4485b764d163e0bf4e20 P 27fddeff55ea476ebe7bd4881ae32035 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:45:18.105259 22853 raft_consensus.cc:3037] T 36cb01872dee4485b764d163e0bf4e20 P 27fddeff55ea476ebe7bd4881ae32035 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:18.100771 22844 raft_consensus.cc:686] T 1154606dbfd849c7833c096541e8c58f P cab341909d314988b6893daac17235e7 [term 1 LEADER]: Becoming Leader. State: Replica: cab341909d314988b6893daac17235e7, State: Running, Role: LEADER
I20260809 15:45:18.085244 22843 raft_consensus.cc:482] T 6d6e21b2d7b746469c42c6d80ad8e001 P 27fddeff55ea476ebe7bd4881ae32035 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:45:18.105434 22766 raft_consensus.cc:348] T 8c8c4f25c5364a919da21f5fca365c45 P 8586928e65c543c2b99cebf1dd7d700f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } } peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } }
I20260809 15:45:18.106564 22766 raft_consensus.cc:374] T 8c8c4f25c5364a919da21f5fca365c45 P 8586928e65c543c2b99cebf1dd7d700f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:18.106828 22766 raft_consensus.cc:729] T 8c8c4f25c5364a919da21f5fca365c45 P 8586928e65c543c2b99cebf1dd7d700f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8586928e65c543c2b99cebf1dd7d700f, State: Initialized, Role: FOLLOWER
I20260809 15:45:18.107173 22857 raft_consensus.cc:2781] T 9b4926c9270e4754bac26cc4c3da4cf9 P 27fddeff55ea476ebe7bd4881ae32035 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:45:18.107466 22766 consensus_queue.cc:260] T 8c8c4f25c5364a919da21f5fca365c45 P 8586928e65c543c2b99cebf1dd7d700f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } } peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } }
I20260809 15:45:18.108405 22857 raft_consensus.cc:686] T 9b4926c9270e4754bac26cc4c3da4cf9 P 27fddeff55ea476ebe7bd4881ae32035 [term 1 LEADER]: Becoming Leader. State: Replica: 27fddeff55ea476ebe7bd4881ae32035, State: Running, Role: LEADER
I20260809 15:45:18.109459 22766 ts_tablet_manager.cc:1428] T 8c8c4f25c5364a919da21f5fca365c45 P 8586928e65c543c2b99cebf1dd7d700f: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20260809 15:45:18.109175 22857 consensus_queue.cc:237] T 9b4926c9270e4754bac26cc4c3da4cf9 P 27fddeff55ea476ebe7bd4881ae32035 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } } peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } }
I20260809 15:45:18.106060 22690 raft_consensus.cc:2379] T 8c8c4f25c5364a919da21f5fca365c45 P 27fddeff55ea476ebe7bd4881ae32035 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate cab341909d314988b6893daac17235e7 in current term 1: Already voted for candidate 27fddeff55ea476ebe7bd4881ae32035 in this term.
I20260809 15:45:18.110610 22844 consensus_queue.cc:237] T 1154606dbfd849c7833c096541e8c58f P cab341909d314988b6893daac17235e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } } peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } }
I20260809 15:45:18.111814 22843 raft_consensus.cc:504] T 6d6e21b2d7b746469c42c6d80ad8e001 P 27fddeff55ea476ebe7bd4881ae32035 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } } peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } }
I20260809 15:45:18.114789 22843 leader_election.cc:288] T 6d6e21b2d7b746469c42c6d80ad8e001 P 27fddeff55ea476ebe7bd4881ae32035 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cab341909d314988b6893daac17235e7 (127.20.202.65:46511), 8586928e65c543c2b99cebf1dd7d700f (127.20.202.66:45399)
I20260809 15:45:18.116407 22433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d6e21b2d7b746469c42c6d80ad8e001" candidate_uuid: "27fddeff55ea476ebe7bd4881ae32035" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab341909d314988b6893daac17235e7" is_pre_election: true
I20260809 15:45:18.117023 22433 raft_consensus.cc:2454] T 6d6e21b2d7b746469c42c6d80ad8e001 P cab341909d314988b6893daac17235e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27fddeff55ea476ebe7bd4881ae32035 in term 0.
I20260809 15:45:18.118155 22360 leader_election.cc:302] T 8c8c4f25c5364a919da21f5fca365c45 P cab341909d314988b6893daac17235e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab341909d314988b6893daac17235e7; no voters: 27fddeff55ea476ebe7bd4881ae32035, 8586928e65c543c2b99cebf1dd7d700f
I20260809 15:45:18.119460 22845 raft_consensus.cc:2726] T 8c8c4f25c5364a919da21f5fca365c45 P cab341909d314988b6893daac17235e7 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260809 15:45:18.120047 22796 raft_consensus.cc:482] T 6d6e21b2d7b746469c42c6d80ad8e001 P 8586928e65c543c2b99cebf1dd7d700f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:45:18.120247 22633 leader_election.cc:302] T 6d6e21b2d7b746469c42c6d80ad8e001 P 27fddeff55ea476ebe7bd4881ae32035 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27fddeff55ea476ebe7bd4881ae32035, cab341909d314988b6893daac17235e7; no voters:
W20260809 15:45:18.101011 22635 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:45399, user_credentials={real_user=slave}} blocked reactor thread for 49549.1us
I20260809 15:45:18.120491 22796 raft_consensus.cc:504] T 6d6e21b2d7b746469c42c6d80ad8e001 P 8586928e65c543c2b99cebf1dd7d700f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } } peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } }
I20260809 15:45:18.122040 22556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d6e21b2d7b746469c42c6d80ad8e001" candidate_uuid: "27fddeff55ea476ebe7bd4881ae32035" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8586928e65c543c2b99cebf1dd7d700f" is_pre_election: true
I20260809 15:45:18.122474 22796 leader_election.cc:288] T 6d6e21b2d7b746469c42c6d80ad8e001 P 8586928e65c543c2b99cebf1dd7d700f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cab341909d314988b6893daac17235e7 (127.20.202.65:46511), 27fddeff55ea476ebe7bd4881ae32035 (127.20.202.67:33583)
I20260809 15:45:18.122692 22772 leader_election.cc:288] T 8c8c4f25c5364a919da21f5fca365c45 P 27fddeff55ea476ebe7bd4881ae32035 [CANDIDATE]: Term 1 election: Requested vote from peers cab341909d314988b6893daac17235e7 (127.20.202.65:46511), 8586928e65c543c2b99cebf1dd7d700f (127.20.202.66:45399)
I20260809 15:45:18.122604 22556 raft_consensus.cc:2454] T 6d6e21b2d7b746469c42c6d80ad8e001 P 8586928e65c543c2b99cebf1dd7d700f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27fddeff55ea476ebe7bd4881ae32035 in term 0.
I20260809 15:45:18.123227 22772 raft_consensus.cc:2781] T 6d6e21b2d7b746469c42c6d80ad8e001 P 27fddeff55ea476ebe7bd4881ae32035 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 15:45:18.123550 22772 raft_consensus.cc:482] T 6d6e21b2d7b746469c42c6d80ad8e001 P 27fddeff55ea476ebe7bd4881ae32035 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:45:18.124006 22433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c8c4f25c5364a919da21f5fca365c45" candidate_uuid: "27fddeff55ea476ebe7bd4881ae32035" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab341909d314988b6893daac17235e7"
I20260809 15:45:18.123706 22430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d6e21b2d7b746469c42c6d80ad8e001" candidate_uuid: "8586928e65c543c2b99cebf1dd7d700f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab341909d314988b6893daac17235e7" is_pre_election: true
I20260809 15:45:18.127442 22772 raft_consensus.cc:3037] T 6d6e21b2d7b746469c42c6d80ad8e001 P 27fddeff55ea476ebe7bd4881ae32035 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:18.129055 22430 raft_consensus.cc:2454] T 6d6e21b2d7b746469c42c6d80ad8e001 P cab341909d314988b6893daac17235e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8586928e65c543c2b99cebf1dd7d700f in term 0.
I20260809 15:45:18.129114 22556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c8c4f25c5364a919da21f5fca365c45" candidate_uuid: "27fddeff55ea476ebe7bd4881ae32035" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8586928e65c543c2b99cebf1dd7d700f"
I20260809 15:45:18.128703 22433 raft_consensus.cc:2379] T 8c8c4f25c5364a919da21f5fca365c45 P cab341909d314988b6893daac17235e7 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 27fddeff55ea476ebe7bd4881ae32035 in current term 1: Already voted for candidate cab341909d314988b6893daac17235e7 in this term.
I20260809 15:45:18.129724 22556 raft_consensus.cc:3037] T 8c8c4f25c5364a919da21f5fca365c45 P 8586928e65c543c2b99cebf1dd7d700f [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:18.126601 22689 raft_consensus.cc:2454] T 1154606dbfd849c7833c096541e8c58f P 27fddeff55ea476ebe7bd4881ae32035 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab341909d314988b6893daac17235e7 in term 1.
I20260809 15:45:18.132046 22492 leader_election.cc:302] T 6d6e21b2d7b746469c42c6d80ad8e001 P 8586928e65c543c2b99cebf1dd7d700f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8586928e65c543c2b99cebf1dd7d700f, cab341909d314988b6893daac17235e7; no voters:
I20260809 15:45:18.132640 22690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d6e21b2d7b746469c42c6d80ad8e001" candidate_uuid: "8586928e65c543c2b99cebf1dd7d700f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27fddeff55ea476ebe7bd4881ae32035" is_pre_election: true
I20260809 15:45:18.133776 22796 raft_consensus.cc:2781] T 6d6e21b2d7b746469c42c6d80ad8e001 P 8586928e65c543c2b99cebf1dd7d700f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 15:45:18.134442 22796 raft_consensus.cc:482] T 6d6e21b2d7b746469c42c6d80ad8e001 P 8586928e65c543c2b99cebf1dd7d700f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:45:18.134825 22796 raft_consensus.cc:3037] T 6d6e21b2d7b746469c42c6d80ad8e001 P 8586928e65c543c2b99cebf1dd7d700f [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:18.135722 22556 raft_consensus.cc:2454] T 8c8c4f25c5364a919da21f5fca365c45 P 8586928e65c543c2b99cebf1dd7d700f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27fddeff55ea476ebe7bd4881ae32035 in term 1.
I20260809 15:45:18.136930 22635 leader_election.cc:302] T 8c8c4f25c5364a919da21f5fca365c45 P 27fddeff55ea476ebe7bd4881ae32035 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27fddeff55ea476ebe7bd4881ae32035, 8586928e65c543c2b99cebf1dd7d700f; no voters: cab341909d314988b6893daac17235e7
I20260809 15:45:18.137949 22772 raft_consensus.cc:504] T 6d6e21b2d7b746469c42c6d80ad8e001 P 27fddeff55ea476ebe7bd4881ae32035 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } } peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } }
I20260809 15:45:18.138569 22764 tablet_bootstrap.cc:492] T 4e0d833302ff456e96ab09aa12d4c5ba P cab341909d314988b6893daac17235e7: No bootstrap required, opened a new log
I20260809 15:45:18.141600 22764 ts_tablet_manager.cc:1397] T 4e0d833302ff456e96ab09aa12d4c5ba P cab341909d314988b6893daac17235e7: Time spent bootstrapping tablet: real 0.095s user 0.031s sys 0.026s
I20260809 15:45:18.141829 22244 catalog_manager.cc:5495] T 9b4926c9270e4754bac26cc4c3da4cf9 P 27fddeff55ea476ebe7bd4881ae32035 reported cstate change: term changed from 0 to 1, leader changed from <none> to 27fddeff55ea476ebe7bd4881ae32035 (127.20.202.67). New cstate: current_term: 1 leader_uuid: "27fddeff55ea476ebe7bd4881ae32035" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } health_report { overall_health: HEALTHY } } }
I20260809 15:45:18.142619 22796 raft_consensus.cc:504] T 6d6e21b2d7b746469c42c6d80ad8e001 P 8586928e65c543c2b99cebf1dd7d700f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } } peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } }
I20260809 15:45:18.144141 22764 raft_consensus.cc:348] T 4e0d833302ff456e96ab09aa12d4c5ba P cab341909d314988b6893daac17235e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } } peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } }
I20260809 15:45:18.131738 22853 raft_consensus.cc:504] T 36cb01872dee4485b764d163e0bf4e20 P 27fddeff55ea476ebe7bd4881ae32035 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } } peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } }
I20260809 15:45:18.145255 22796 leader_election.cc:288] T 6d6e21b2d7b746469c42c6d80ad8e001 P 8586928e65c543c2b99cebf1dd7d700f [CANDIDATE]: Term 1 election: Requested vote from peers cab341909d314988b6893daac17235e7 (127.20.202.65:46511), 27fddeff55ea476ebe7bd4881ae32035 (127.20.202.67:33583)
I20260809 15:45:18.146358 22853 leader_election.cc:288] T 36cb01872dee4485b764d163e0bf4e20 P 27fddeff55ea476ebe7bd4881ae32035 [CANDIDATE]: Term 1 election: Requested vote from peers cab341909d314988b6893daac17235e7 (127.20.202.65:46511), 8586928e65c543c2b99cebf1dd7d700f (127.20.202.66:45399)
I20260809 15:45:18.146466 22433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d6e21b2d7b746469c42c6d80ad8e001" candidate_uuid: "8586928e65c543c2b99cebf1dd7d700f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab341909d314988b6893daac17235e7"
I20260809 15:45:18.147074 22433 raft_consensus.cc:3037] T 6d6e21b2d7b746469c42c6d80ad8e001 P cab341909d314988b6893daac17235e7 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:18.147724 22556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36cb01872dee4485b764d163e0bf4e20" candidate_uuid: "27fddeff55ea476ebe7bd4881ae32035" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8586928e65c543c2b99cebf1dd7d700f"
I20260809 15:45:18.144845 22764 raft_consensus.cc:374] T 4e0d833302ff456e96ab09aa12d4c5ba P cab341909d314988b6893daac17235e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:18.148151 22772 leader_election.cc:288] T 6d6e21b2d7b746469c42c6d80ad8e001 P 27fddeff55ea476ebe7bd4881ae32035 [CANDIDATE]: Term 1 election: Requested vote from peers cab341909d314988b6893daac17235e7 (127.20.202.65:46511), 8586928e65c543c2b99cebf1dd7d700f (127.20.202.66:45399)
I20260809 15:45:18.148283 22556 raft_consensus.cc:3037] T 36cb01872dee4485b764d163e0bf4e20 P 8586928e65c543c2b99cebf1dd7d700f [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:18.148304 22764 raft_consensus.cc:729] T 4e0d833302ff456e96ab09aa12d4c5ba P cab341909d314988b6893daac17235e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cab341909d314988b6893daac17235e7, State: Initialized, Role: FOLLOWER
I20260809 15:45:18.145252 22242 catalog_manager.cc:5495] T 1154606dbfd849c7833c096541e8c58f P cab341909d314988b6893daac17235e7 reported cstate change: term changed from 0 to 1, leader changed from <none> to cab341909d314988b6893daac17235e7 (127.20.202.65). New cstate: current_term: 1 leader_uuid: "cab341909d314988b6893daac17235e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } health_report { overall_health: UNKNOWN } } }
I20260809 15:45:18.149292 22764 consensus_queue.cc:260] T 4e0d833302ff456e96ab09aa12d4c5ba P cab341909d314988b6893daac17235e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } } peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } }
I20260809 15:45:18.150473 22430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36cb01872dee4485b764d163e0bf4e20" candidate_uuid: "27fddeff55ea476ebe7bd4881ae32035" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab341909d314988b6893daac17235e7"
I20260809 15:45:18.151772 22690 raft_consensus.cc:2379] T 6d6e21b2d7b746469c42c6d80ad8e001 P 27fddeff55ea476ebe7bd4881ae32035 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8586928e65c543c2b99cebf1dd7d700f in current term 1: Already voted for candidate 27fddeff55ea476ebe7bd4881ae32035 in this term.
I20260809 15:45:18.152099 22764 ts_tablet_manager.cc:1428] T 4e0d833302ff456e96ab09aa12d4c5ba P cab341909d314988b6893daac17235e7: Time spent starting tablet: real 0.010s user 0.005s sys 0.000s
I20260809 15:45:18.153035 22764 tablet_bootstrap.cc:492] T 21aa1094e1cf4d40b51e5916cb6a2a96 P cab341909d314988b6893daac17235e7: Bootstrap starting.
I20260809 15:45:18.144932 22858 raft_consensus.cc:2781] T 8c8c4f25c5364a919da21f5fca365c45 P 27fddeff55ea476ebe7bd4881ae32035 [term 1 FOLLOWER]: Leader election won for term 1
W20260809 15:45:18.153264 22633 leader_election.cc:341] T 36cb01872dee4485b764d163e0bf4e20 P 27fddeff55ea476ebe7bd4881ae32035 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cab341909d314988b6893daac17235e7 (127.20.202.65:46511): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:18.168453 22436 raft_consensus.cc:3037] T 6eb2494806b34a2bad3215770bcb4c7c P cab341909d314988b6893daac17235e7 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:18.153893 22689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d6e21b2d7b746469c42c6d80ad8e001" candidate_uuid: "8586928e65c543c2b99cebf1dd7d700f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27fddeff55ea476ebe7bd4881ae32035"
I20260809 15:45:18.155652 22555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d6e21b2d7b746469c42c6d80ad8e001" candidate_uuid: "27fddeff55ea476ebe7bd4881ae32035" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8586928e65c543c2b99cebf1dd7d700f"
I20260809 15:45:18.169545 22689 raft_consensus.cc:2379] T 6d6e21b2d7b746469c42c6d80ad8e001 P 27fddeff55ea476ebe7bd4881ae32035 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 8586928e65c543c2b99cebf1dd7d700f in current term 1: Already voted for candidate 27fddeff55ea476ebe7bd4881ae32035 in this term.
I20260809 15:45:18.169783 22555 raft_consensus.cc:2379] T 6d6e21b2d7b746469c42c6d80ad8e001 P 8586928e65c543c2b99cebf1dd7d700f [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 27fddeff55ea476ebe7bd4881ae32035 in current term 1: Already voted for candidate 8586928e65c543c2b99cebf1dd7d700f in this term.
I20260809 15:45:18.154246 22430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d6e21b2d7b746469c42c6d80ad8e001" candidate_uuid: "27fddeff55ea476ebe7bd4881ae32035" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab341909d314988b6893daac17235e7"
I20260809 15:45:18.171062 22858 raft_consensus.cc:686] T 8c8c4f25c5364a919da21f5fca365c45 P 27fddeff55ea476ebe7bd4881ae32035 [term 1 LEADER]: Becoming Leader. State: Replica: 27fddeff55ea476ebe7bd4881ae32035, State: Running, Role: LEADER
I20260809 15:45:18.171741 22633 leader_election.cc:302] T 6d6e21b2d7b746469c42c6d80ad8e001 P 27fddeff55ea476ebe7bd4881ae32035 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27fddeff55ea476ebe7bd4881ae32035; no voters: 8586928e65c543c2b99cebf1dd7d700f, cab341909d314988b6893daac17235e7
I20260809 15:45:18.172461 22772 raft_consensus.cc:2726] T 6d6e21b2d7b746469c42c6d80ad8e001 P 27fddeff55ea476ebe7bd4881ae32035 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260809 15:45:18.171968 22858 consensus_queue.cc:237] T 8c8c4f25c5364a919da21f5fca365c45 P 27fddeff55ea476ebe7bd4881ae32035 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } } peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } }
I20260809 15:45:18.165663 22433 raft_consensus.cc:2454] T 6d6e21b2d7b746469c42c6d80ad8e001 P cab341909d314988b6893daac17235e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8586928e65c543c2b99cebf1dd7d700f in term 1.
I20260809 15:45:18.158677 22556 raft_consensus.cc:2454] T 36cb01872dee4485b764d163e0bf4e20 P 8586928e65c543c2b99cebf1dd7d700f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27fddeff55ea476ebe7bd4881ae32035 in term 1.
I20260809 15:45:18.176080 22764 tablet_bootstrap.cc:654] T 21aa1094e1cf4d40b51e5916cb6a2a96 P cab341909d314988b6893daac17235e7: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:18.176247 22635 leader_election.cc:302] T 36cb01872dee4485b764d163e0bf4e20 P 27fddeff55ea476ebe7bd4881ae32035 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27fddeff55ea476ebe7bd4881ae32035, 8586928e65c543c2b99cebf1dd7d700f; no voters: cab341909d314988b6893daac17235e7
I20260809 15:45:18.177066 22796 consensus_queue.cc:1035] T 6eb2494806b34a2bad3215770bcb4c7c P 8586928e65c543c2b99cebf1dd7d700f [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:45:18.177067 22772 raft_consensus.cc:2781] T 36cb01872dee4485b764d163e0bf4e20 P 27fddeff55ea476ebe7bd4881ae32035 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:45:18.179054 22772 raft_consensus.cc:686] T 36cb01872dee4485b764d163e0bf4e20 P 27fddeff55ea476ebe7bd4881ae32035 [term 1 LEADER]: Becoming Leader. State: Replica: 27fddeff55ea476ebe7bd4881ae32035, State: Running, Role: LEADER
I20260809 15:45:18.180334 22492 leader_election.cc:302] T 6d6e21b2d7b746469c42c6d80ad8e001 P 8586928e65c543c2b99cebf1dd7d700f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8586928e65c543c2b99cebf1dd7d700f, cab341909d314988b6893daac17235e7; no voters: 27fddeff55ea476ebe7bd4881ae32035
I20260809 15:45:18.180109 22772 consensus_queue.cc:237] T 36cb01872dee4485b764d163e0bf4e20 P 27fddeff55ea476ebe7bd4881ae32035 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } } peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } }
I20260809 15:45:18.181414 22796 raft_consensus.cc:2781] T 6d6e21b2d7b746469c42c6d80ad8e001 P 8586928e65c543c2b99cebf1dd7d700f [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:45:18.181829 22796 raft_consensus.cc:686] T 6d6e21b2d7b746469c42c6d80ad8e001 P 8586928e65c543c2b99cebf1dd7d700f [term 1 LEADER]: Becoming Leader. State: Replica: 8586928e65c543c2b99cebf1dd7d700f, State: Running, Role: LEADER
I20260809 15:45:18.182035 22764 tablet_bootstrap.cc:492] T 21aa1094e1cf4d40b51e5916cb6a2a96 P cab341909d314988b6893daac17235e7: No bootstrap required, opened a new log
I20260809 15:45:18.182483 22764 ts_tablet_manager.cc:1397] T 21aa1094e1cf4d40b51e5916cb6a2a96 P cab341909d314988b6893daac17235e7: Time spent bootstrapping tablet: real 0.030s user 0.011s sys 0.000s
I20260809 15:45:18.183872 22796 consensus_queue.cc:237] T 6d6e21b2d7b746469c42c6d80ad8e001 P 8586928e65c543c2b99cebf1dd7d700f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } } peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } }
I20260809 15:45:18.185110 22764 raft_consensus.cc:348] T 21aa1094e1cf4d40b51e5916cb6a2a96 P cab341909d314988b6893daac17235e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } } peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } }
I20260809 15:45:18.185796 22764 raft_consensus.cc:374] T 21aa1094e1cf4d40b51e5916cb6a2a96 P cab341909d314988b6893daac17235e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:18.186088 22764 raft_consensus.cc:729] T 21aa1094e1cf4d40b51e5916cb6a2a96 P cab341909d314988b6893daac17235e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cab341909d314988b6893daac17235e7, State: Initialized, Role: FOLLOWER
I20260809 15:45:18.186705 22764 consensus_queue.cc:260] T 21aa1094e1cf4d40b51e5916cb6a2a96 P cab341909d314988b6893daac17235e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } } peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } }
I20260809 15:45:18.188623 22764 ts_tablet_manager.cc:1428] T 21aa1094e1cf4d40b51e5916cb6a2a96 P cab341909d314988b6893daac17235e7: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260809 15:45:18.189525 22764 tablet_bootstrap.cc:492] T ab0d7bb4b86540b28101d1c813ee214e P cab341909d314988b6893daac17235e7: Bootstrap starting.
I20260809 15:45:18.190025 22242 catalog_manager.cc:5495] T 8c8c4f25c5364a919da21f5fca365c45 P 27fddeff55ea476ebe7bd4881ae32035 reported cstate change: term changed from 0 to 1, leader changed from <none> to 27fddeff55ea476ebe7bd4881ae32035 (127.20.202.67). New cstate: current_term: 1 leader_uuid: "27fddeff55ea476ebe7bd4881ae32035" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } health_report { overall_health: HEALTHY } } }
I20260809 15:45:18.194593 22764 tablet_bootstrap.cc:654] T ab0d7bb4b86540b28101d1c813ee214e P cab341909d314988b6893daac17235e7: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:18.203493 22242 catalog_manager.cc:5495] T 6d6e21b2d7b746469c42c6d80ad8e001 P 8586928e65c543c2b99cebf1dd7d700f reported cstate change: term changed from 0 to 1, leader changed from <none> to 8586928e65c543c2b99cebf1dd7d700f (127.20.202.66). New cstate: current_term: 1 leader_uuid: "8586928e65c543c2b99cebf1dd7d700f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } health_report { overall_health: UNKNOWN } } }
I20260809 15:45:18.208029 22764 tablet_bootstrap.cc:492] T ab0d7bb4b86540b28101d1c813ee214e P cab341909d314988b6893daac17235e7: No bootstrap required, opened a new log
I20260809 15:45:18.208477 22764 ts_tablet_manager.cc:1397] T ab0d7bb4b86540b28101d1c813ee214e P cab341909d314988b6893daac17235e7: Time spent bootstrapping tablet: real 0.019s user 0.012s sys 0.006s
I20260809 15:45:18.211148 22764 raft_consensus.cc:348] T ab0d7bb4b86540b28101d1c813ee214e P cab341909d314988b6893daac17235e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } } peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } }
I20260809 15:45:18.211557 22795 raft_consensus.cc:482] T ab0d7bb4b86540b28101d1c813ee214e P 8586928e65c543c2b99cebf1dd7d700f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:45:18.211859 22764 raft_consensus.cc:374] T ab0d7bb4b86540b28101d1c813ee214e P cab341909d314988b6893daac17235e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:18.212194 22764 raft_consensus.cc:729] T ab0d7bb4b86540b28101d1c813ee214e P cab341909d314988b6893daac17235e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cab341909d314988b6893daac17235e7, State: Initialized, Role: FOLLOWER
I20260809 15:45:18.213186 22764 consensus_queue.cc:260] T ab0d7bb4b86540b28101d1c813ee214e P cab341909d314988b6893daac17235e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } } peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } }
I20260809 15:45:18.212745 22795 raft_consensus.cc:504] T ab0d7bb4b86540b28101d1c813ee214e P 8586928e65c543c2b99cebf1dd7d700f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } } peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } }
I20260809 15:45:18.215124 22764 ts_tablet_manager.cc:1428] T ab0d7bb4b86540b28101d1c813ee214e P cab341909d314988b6893daac17235e7: Time spent starting tablet: real 0.006s user 0.003s sys 0.002s
I20260809 15:45:18.215978 22764 tablet_bootstrap.cc:492] T 36cb01872dee4485b764d163e0bf4e20 P cab341909d314988b6893daac17235e7: Bootstrap starting.
I20260809 15:45:18.215818 22433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab0d7bb4b86540b28101d1c813ee214e" candidate_uuid: "8586928e65c543c2b99cebf1dd7d700f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab341909d314988b6893daac17235e7" is_pre_election: true
I20260809 15:45:18.216466 22433 raft_consensus.cc:2454] T ab0d7bb4b86540b28101d1c813ee214e P cab341909d314988b6893daac17235e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8586928e65c543c2b99cebf1dd7d700f in term 0.
I20260809 15:45:18.215118 22795 leader_election.cc:288] T ab0d7bb4b86540b28101d1c813ee214e P 8586928e65c543c2b99cebf1dd7d700f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cab341909d314988b6893daac17235e7 (127.20.202.65:46511), 27fddeff55ea476ebe7bd4881ae32035 (127.20.202.67:33583)
I20260809 15:45:18.210812 22244 catalog_manager.cc:5495] T 36cb01872dee4485b764d163e0bf4e20 P 27fddeff55ea476ebe7bd4881ae32035 reported cstate change: term changed from 0 to 1, leader changed from <none> to 27fddeff55ea476ebe7bd4881ae32035 (127.20.202.67). New cstate: current_term: 1 leader_uuid: "27fddeff55ea476ebe7bd4881ae32035" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } health_report { overall_health: HEALTHY } } }
I20260809 15:45:18.217626 22492 leader_election.cc:302] T ab0d7bb4b86540b28101d1c813ee214e P 8586928e65c543c2b99cebf1dd7d700f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8586928e65c543c2b99cebf1dd7d700f, cab341909d314988b6893daac17235e7; no voters:
I20260809 15:45:18.218528 22795 raft_consensus.cc:2781] T ab0d7bb4b86540b28101d1c813ee214e P 8586928e65c543c2b99cebf1dd7d700f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 15:45:18.218936 22795 raft_consensus.cc:482] T ab0d7bb4b86540b28101d1c813ee214e P 8586928e65c543c2b99cebf1dd7d700f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:45:18.219521 22795 raft_consensus.cc:3037] T ab0d7bb4b86540b28101d1c813ee214e P 8586928e65c543c2b99cebf1dd7d700f [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:18.220913 22690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab0d7bb4b86540b28101d1c813ee214e" candidate_uuid: "8586928e65c543c2b99cebf1dd7d700f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27fddeff55ea476ebe7bd4881ae32035" is_pre_election: true
I20260809 15:45:18.221500 22690 raft_consensus.cc:2454] T ab0d7bb4b86540b28101d1c813ee214e P 27fddeff55ea476ebe7bd4881ae32035 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8586928e65c543c2b99cebf1dd7d700f in term 0.
I20260809 15:45:18.223506 22796 consensus_queue.cc:1035] T 6eb2494806b34a2bad3215770bcb4c7c P 8586928e65c543c2b99cebf1dd7d700f [LEADER]: Connected to new peer: Peer: permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:45:18.225603 22795 raft_consensus.cc:504] T ab0d7bb4b86540b28101d1c813ee214e P 8586928e65c543c2b99cebf1dd7d700f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } } peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } }
I20260809 15:45:18.227288 22764 tablet_bootstrap.cc:654] T 36cb01872dee4485b764d163e0bf4e20 P cab341909d314988b6893daac17235e7: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:18.237646 22795 leader_election.cc:288] T ab0d7bb4b86540b28101d1c813ee214e P 8586928e65c543c2b99cebf1dd7d700f [CANDIDATE]: Term 1 election: Requested vote from peers cab341909d314988b6893daac17235e7 (127.20.202.65:46511), 27fddeff55ea476ebe7bd4881ae32035 (127.20.202.67:33583)
I20260809 15:45:18.238641 22689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab0d7bb4b86540b28101d1c813ee214e" candidate_uuid: "8586928e65c543c2b99cebf1dd7d700f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27fddeff55ea476ebe7bd4881ae32035"
I20260809 15:45:18.239221 22689 raft_consensus.cc:3037] T ab0d7bb4b86540b28101d1c813ee214e P 27fddeff55ea476ebe7bd4881ae32035 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:18.240825 22858 raft_consensus.cc:482] T 6b1f47a4af1b4480ab2e69adb8be3cc3 P 27fddeff55ea476ebe7bd4881ae32035 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:45:18.241464 22858 raft_consensus.cc:504] T 6b1f47a4af1b4480ab2e69adb8be3cc3 P 27fddeff55ea476ebe7bd4881ae32035 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } } peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } }
I20260809 15:45:18.241825 22436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab0d7bb4b86540b28101d1c813ee214e" candidate_uuid: "8586928e65c543c2b99cebf1dd7d700f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab341909d314988b6893daac17235e7"
I20260809 15:45:18.242395 22436 raft_consensus.cc:3037] T ab0d7bb4b86540b28101d1c813ee214e P cab341909d314988b6893daac17235e7 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:18.243472 22858 leader_election.cc:288] T 6b1f47a4af1b4480ab2e69adb8be3cc3 P 27fddeff55ea476ebe7bd4881ae32035 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8586928e65c543c2b99cebf1dd7d700f (127.20.202.66:45399), cab341909d314988b6893daac17235e7 (127.20.202.65:46511)
I20260809 15:45:18.244120 22556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b1f47a4af1b4480ab2e69adb8be3cc3" candidate_uuid: "27fddeff55ea476ebe7bd4881ae32035" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8586928e65c543c2b99cebf1dd7d700f" is_pre_election: true
I20260809 15:45:18.248296 22433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b1f47a4af1b4480ab2e69adb8be3cc3" candidate_uuid: "27fddeff55ea476ebe7bd4881ae32035" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab341909d314988b6893daac17235e7" is_pre_election: true
I20260809 15:45:18.248723 22689 raft_consensus.cc:2454] T ab0d7bb4b86540b28101d1c813ee214e P 27fddeff55ea476ebe7bd4881ae32035 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8586928e65c543c2b99cebf1dd7d700f in term 1.
I20260809 15:45:18.249002 22433 raft_consensus.cc:2379] T 6b1f47a4af1b4480ab2e69adb8be3cc3 P cab341909d314988b6893daac17235e7 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27fddeff55ea476ebe7bd4881ae32035 in current term 1: Already voted for candidate 8586928e65c543c2b99cebf1dd7d700f in this term.
I20260809 15:45:18.249853 22493 leader_election.cc:302] T ab0d7bb4b86540b28101d1c813ee214e P 8586928e65c543c2b99cebf1dd7d700f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27fddeff55ea476ebe7bd4881ae32035, 8586928e65c543c2b99cebf1dd7d700f; no voters:
I20260809 15:45:18.250008 22633 leader_election.cc:302] T 6b1f47a4af1b4480ab2e69adb8be3cc3 P 27fddeff55ea476ebe7bd4881ae32035 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27fddeff55ea476ebe7bd4881ae32035; no voters: 8586928e65c543c2b99cebf1dd7d700f, cab341909d314988b6893daac17235e7
I20260809 15:45:18.251060 22777 raft_consensus.cc:2781] T ab0d7bb4b86540b28101d1c813ee214e P 8586928e65c543c2b99cebf1dd7d700f [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:45:18.251184 22772 raft_consensus.cc:3037] T 6b1f47a4af1b4480ab2e69adb8be3cc3 P 27fddeff55ea476ebe7bd4881ae32035 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:18.251055 22436 raft_consensus.cc:2454] T ab0d7bb4b86540b28101d1c813ee214e P cab341909d314988b6893daac17235e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8586928e65c543c2b99cebf1dd7d700f in term 1.
I20260809 15:45:18.251777 22777 raft_consensus.cc:686] T ab0d7bb4b86540b28101d1c813ee214e P 8586928e65c543c2b99cebf1dd7d700f [term 1 LEADER]: Becoming Leader. State: Replica: 8586928e65c543c2b99cebf1dd7d700f, State: Running, Role: LEADER
I20260809 15:45:18.253592 22436 raft_consensus.cc:3037] T 21aa1094e1cf4d40b51e5916cb6a2a96 P cab341909d314988b6893daac17235e7 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:18.254230 22777 consensus_queue.cc:237] T ab0d7bb4b86540b28101d1c813ee214e P 8586928e65c543c2b99cebf1dd7d700f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } } peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } }
I20260809 15:45:18.259363 22764 tablet_bootstrap.cc:492] T 36cb01872dee4485b764d163e0bf4e20 P cab341909d314988b6893daac17235e7: No bootstrap required, opened a new log
I20260809 15:45:18.259768 22764 ts_tablet_manager.cc:1397] T 36cb01872dee4485b764d163e0bf4e20 P cab341909d314988b6893daac17235e7: Time spent bootstrapping tablet: real 0.044s user 0.013s sys 0.014s
I20260809 15:45:18.262024 22764 raft_consensus.cc:348] T 36cb01872dee4485b764d163e0bf4e20 P cab341909d314988b6893daac17235e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } } peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } }
I20260809 15:45:18.262274 22777 consensus_queue.cc:1035] T 21aa1094e1cf4d40b51e5916cb6a2a96 P 8586928e65c543c2b99cebf1dd7d700f [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:45:18.262691 22764 raft_consensus.cc:374] T 36cb01872dee4485b764d163e0bf4e20 P cab341909d314988b6893daac17235e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:18.262962 22764 raft_consensus.cc:729] T 36cb01872dee4485b764d163e0bf4e20 P cab341909d314988b6893daac17235e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cab341909d314988b6893daac17235e7, State: Initialized, Role: FOLLOWER
I20260809 15:45:18.263573 22764 consensus_queue.cc:260] T 36cb01872dee4485b764d163e0bf4e20 P cab341909d314988b6893daac17235e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } } peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } }
I20260809 15:45:18.263482 22241 catalog_manager.cc:5495] T ab0d7bb4b86540b28101d1c813ee214e P 8586928e65c543c2b99cebf1dd7d700f reported cstate change: term changed from 0 to 1, leader changed from <none> to 8586928e65c543c2b99cebf1dd7d700f (127.20.202.66). New cstate: current_term: 1 leader_uuid: "8586928e65c543c2b99cebf1dd7d700f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } health_report { overall_health: UNKNOWN } } }
I20260809 15:45:18.265013 22772 raft_consensus.cc:2726] T 6b1f47a4af1b4480ab2e69adb8be3cc3 P 27fddeff55ea476ebe7bd4881ae32035 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 15:45:18.266099 22764 ts_tablet_manager.cc:1428] T 36cb01872dee4485b764d163e0bf4e20 P cab341909d314988b6893daac17235e7: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260809 15:45:18.278714 22879 consensus_queue.cc:1035] T 21aa1094e1cf4d40b51e5916cb6a2a96 P 8586928e65c543c2b99cebf1dd7d700f [LEADER]: Connected to new peer: Peer: permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:45:18.357721 22844 raft_consensus.cc:482] T 4e0d833302ff456e96ab09aa12d4c5ba P cab341909d314988b6893daac17235e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:45:18.358134 22844 raft_consensus.cc:504] T 4e0d833302ff456e96ab09aa12d4c5ba P cab341909d314988b6893daac17235e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } } peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } }
I20260809 15:45:18.359663 22844 leader_election.cc:288] T 4e0d833302ff456e96ab09aa12d4c5ba P cab341909d314988b6893daac17235e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8586928e65c543c2b99cebf1dd7d700f (127.20.202.66:45399), 27fddeff55ea476ebe7bd4881ae32035 (127.20.202.67:33583)
I20260809 15:45:18.360327 22556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e0d833302ff456e96ab09aa12d4c5ba" candidate_uuid: "cab341909d314988b6893daac17235e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8586928e65c543c2b99cebf1dd7d700f" is_pre_election: true
I20260809 15:45:18.360546 22689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e0d833302ff456e96ab09aa12d4c5ba" candidate_uuid: "cab341909d314988b6893daac17235e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27fddeff55ea476ebe7bd4881ae32035" is_pre_election: true
I20260809 15:45:18.361199 22689 raft_consensus.cc:2379] T 4e0d833302ff456e96ab09aa12d4c5ba P 27fddeff55ea476ebe7bd4881ae32035 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab341909d314988b6893daac17235e7 in current term 1: Already voted for candidate 8586928e65c543c2b99cebf1dd7d700f in this term.
I20260809 15:45:18.361975 22360 leader_election.cc:302] T 4e0d833302ff456e96ab09aa12d4c5ba P cab341909d314988b6893daac17235e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab341909d314988b6893daac17235e7; no voters: 27fddeff55ea476ebe7bd4881ae32035, 8586928e65c543c2b99cebf1dd7d700f
I20260809 15:45:18.362533 22844 raft_consensus.cc:3037] T 4e0d833302ff456e96ab09aa12d4c5ba P cab341909d314988b6893daac17235e7 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:18.366855 22844 raft_consensus.cc:2726] T 4e0d833302ff456e96ab09aa12d4c5ba P cab341909d314988b6893daac17235e7 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 15:45:18.438319 22879 consensus_queue.cc:1035] T 6b1f47a4af1b4480ab2e69adb8be3cc3 P 8586928e65c543c2b99cebf1dd7d700f [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:45:18.442070 22844 raft_consensus.cc:482] T 36cb01872dee4485b764d163e0bf4e20 P cab341909d314988b6893daac17235e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:45:18.442502 22844 raft_consensus.cc:504] T 36cb01872dee4485b764d163e0bf4e20 P cab341909d314988b6893daac17235e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } } peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } }
I20260809 15:45:18.444358 22844 leader_election.cc:288] T 36cb01872dee4485b764d163e0bf4e20 P cab341909d314988b6893daac17235e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8586928e65c543c2b99cebf1dd7d700f (127.20.202.66:45399), 27fddeff55ea476ebe7bd4881ae32035 (127.20.202.67:33583)
I20260809 15:45:18.446429 22796 consensus_queue.cc:1035] T 4e0d833302ff456e96ab09aa12d4c5ba P 8586928e65c543c2b99cebf1dd7d700f [LEADER]: Connected to new peer: Peer: permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:45:18.446786 22556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36cb01872dee4485b764d163e0bf4e20" candidate_uuid: "cab341909d314988b6893daac17235e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8586928e65c543c2b99cebf1dd7d700f" is_pre_election: true
I20260809 15:45:18.447342 22689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36cb01872dee4485b764d163e0bf4e20" candidate_uuid: "cab341909d314988b6893daac17235e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27fddeff55ea476ebe7bd4881ae32035" is_pre_election: true
I20260809 15:45:18.447723 22556 raft_consensus.cc:2379] T 36cb01872dee4485b764d163e0bf4e20 P 8586928e65c543c2b99cebf1dd7d700f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cab341909d314988b6893daac17235e7 in current term 1: Already voted for candidate 27fddeff55ea476ebe7bd4881ae32035 in this term.
I20260809 15:45:18.448688 22360 leader_election.cc:302] T 36cb01872dee4485b764d163e0bf4e20 P cab341909d314988b6893daac17235e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cab341909d314988b6893daac17235e7; no voters: 27fddeff55ea476ebe7bd4881ae32035, 8586928e65c543c2b99cebf1dd7d700f
I20260809 15:45:18.449591 22845 raft_consensus.cc:3037] T 36cb01872dee4485b764d163e0bf4e20 P cab341909d314988b6893daac17235e7 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:18.450842 22879 consensus_queue.cc:1035] T 6b1f47a4af1b4480ab2e69adb8be3cc3 P 8586928e65c543c2b99cebf1dd7d700f [LEADER]: Connected to new peer: Peer: permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:45:18.456669 22845 raft_consensus.cc:2726] T 36cb01872dee4485b764d163e0bf4e20 P cab341909d314988b6893daac17235e7 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 15:45:18.465039 22795 consensus_queue.cc:1035] T 4e0d833302ff456e96ab09aa12d4c5ba P 8586928e65c543c2b99cebf1dd7d700f [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:45:18.489660 22556 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6b1f47a4af1b4480ab2e69adb8be3cc3"
dest_uuid: "8586928e65c543c2b99cebf1dd7d700f"
mode: GRACEFUL
new_leader_uuid: "cab341909d314988b6893daac17235e7"
from {username='slave'} at 127.0.0.1:56494
I20260809 15:45:18.490199 22556 raft_consensus.cc:595] T 6b1f47a4af1b4480ab2e69adb8be3cc3 P 8586928e65c543c2b99cebf1dd7d700f [term 1 LEADER]: Received request to transfer leadership to TS cab341909d314988b6893daac17235e7
I20260809 15:45:18.501806 22689 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8c8c4f25c5364a919da21f5fca365c45"
dest_uuid: "27fddeff55ea476ebe7bd4881ae32035"
mode: GRACEFUL
new_leader_uuid: "cab341909d314988b6893daac17235e7"
from {username='slave'} at 127.0.0.1:51590
I20260809 15:45:18.502249 22689 raft_consensus.cc:595] T 8c8c4f25c5364a919da21f5fca365c45 P 27fddeff55ea476ebe7bd4881ae32035 [term 1 LEADER]: Received request to transfer leadership to TS cab341909d314988b6893daac17235e7
I20260809 15:45:18.506085 22689 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9b4926c9270e4754bac26cc4c3da4cf9"
dest_uuid: "27fddeff55ea476ebe7bd4881ae32035"
mode: GRACEFUL
new_leader_uuid: "cab341909d314988b6893daac17235e7"
from {username='slave'} at 127.0.0.1:51590
I20260809 15:45:18.506474 22689 raft_consensus.cc:595] T 9b4926c9270e4754bac26cc4c3da4cf9 P 27fddeff55ea476ebe7bd4881ae32035 [term 1 LEADER]: Received request to transfer leadership to TS cab341909d314988b6893daac17235e7
I20260809 15:45:18.512504 22556 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4e0d833302ff456e96ab09aa12d4c5ba"
dest_uuid: "8586928e65c543c2b99cebf1dd7d700f"
mode: GRACEFUL
new_leader_uuid: "cab341909d314988b6893daac17235e7"
from {username='slave'} at 127.0.0.1:56494
I20260809 15:45:18.512916 22556 raft_consensus.cc:595] T 4e0d833302ff456e96ab09aa12d4c5ba P 8586928e65c543c2b99cebf1dd7d700f [term 1 LEADER]: Received request to transfer leadership to TS cab341909d314988b6893daac17235e7
I20260809 15:45:18.538864 22845 consensus_queue.cc:1035] T 1154606dbfd849c7833c096541e8c58f P cab341909d314988b6893daac17235e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:45:18.548005 22844 consensus_queue.cc:1035] T 1154606dbfd849c7833c096541e8c58f P cab341909d314988b6893daac17235e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:45:18.594691 22879 consensus_queue.cc:1035] T 6d6e21b2d7b746469c42c6d80ad8e001 P 8586928e65c543c2b99cebf1dd7d700f [LEADER]: Connected to new peer: Peer: permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:45:18.607199 22777 consensus_queue.cc:1035] T 6d6e21b2d7b746469c42c6d80ad8e001 P 8586928e65c543c2b99cebf1dd7d700f [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:45:18.616361 22772 consensus_queue.cc:1035] T 9b4926c9270e4754bac26cc4c3da4cf9 P 27fddeff55ea476ebe7bd4881ae32035 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:45:18.621800 22858 raft_consensus.cc:979] T 9b4926c9270e4754bac26cc4c3da4cf9 P 27fddeff55ea476ebe7bd4881ae32035: : Instructing follower cab341909d314988b6893daac17235e7 to start an election
I20260809 15:45:18.622212 22857 raft_consensus.cc:1067] T 9b4926c9270e4754bac26cc4c3da4cf9 P 27fddeff55ea476ebe7bd4881ae32035 [term 1 LEADER]: Signalling peer cab341909d314988b6893daac17235e7 to start an election
I20260809 15:45:18.623694 22433 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9b4926c9270e4754bac26cc4c3da4cf9"
dest_uuid: "cab341909d314988b6893daac17235e7"
from {username='slave'} at 127.0.0.1:47182
I20260809 15:45:18.624476 22433 raft_consensus.cc:482] T 9b4926c9270e4754bac26cc4c3da4cf9 P cab341909d314988b6893daac17235e7 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 15:45:18.624846 22433 raft_consensus.cc:3037] T 9b4926c9270e4754bac26cc4c3da4cf9 P cab341909d314988b6893daac17235e7 [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:45:18.630002 22433 raft_consensus.cc:504] T 9b4926c9270e4754bac26cc4c3da4cf9 P cab341909d314988b6893daac17235e7 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } } peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } }
I20260809 15:45:18.632135 22555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b4926c9270e4754bac26cc4c3da4cf9" candidate_uuid: "cab341909d314988b6893daac17235e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8586928e65c543c2b99cebf1dd7d700f"
I20260809 15:45:18.632601 22555 raft_consensus.cc:3037] T 9b4926c9270e4754bac26cc4c3da4cf9 P 8586928e65c543c2b99cebf1dd7d700f [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:45:18.633121 22857 consensus_queue.cc:1035] T 9b4926c9270e4754bac26cc4c3da4cf9 P 27fddeff55ea476ebe7bd4881ae32035 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:45:18.633915 22689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b4926c9270e4754bac26cc4c3da4cf9" candidate_uuid: "cab341909d314988b6893daac17235e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "27fddeff55ea476ebe7bd4881ae32035"
I20260809 15:45:18.634479 22689 raft_consensus.cc:3032] T 9b4926c9270e4754bac26cc4c3da4cf9 P 27fddeff55ea476ebe7bd4881ae32035 [term 1 LEADER]: Stepping down as leader of term 1
I20260809 15:45:18.634766 22689 raft_consensus.cc:729] T 9b4926c9270e4754bac26cc4c3da4cf9 P 27fddeff55ea476ebe7bd4881ae32035 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 27fddeff55ea476ebe7bd4881ae32035, State: Running, Role: LEADER
I20260809 15:45:18.635468 22433 leader_election.cc:288] T 9b4926c9270e4754bac26cc4c3da4cf9 P cab341909d314988b6893daac17235e7 [CANDIDATE]: Term 2 election: Requested vote from peers 8586928e65c543c2b99cebf1dd7d700f (127.20.202.66:45399), 27fddeff55ea476ebe7bd4881ae32035 (127.20.202.67:33583)
I20260809 15:45:18.635438 22689 consensus_queue.cc:260] T 9b4926c9270e4754bac26cc4c3da4cf9 P 27fddeff55ea476ebe7bd4881ae32035 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } } peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } }
I20260809 15:45:18.636178 22689 raft_consensus.cc:3037] T 9b4926c9270e4754bac26cc4c3da4cf9 P 27fddeff55ea476ebe7bd4881ae32035 [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:45:18.637727 22555 raft_consensus.cc:2454] T 9b4926c9270e4754bac26cc4c3da4cf9 P 8586928e65c543c2b99cebf1dd7d700f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab341909d314988b6893daac17235e7 in term 2.
I20260809 15:45:18.638509 22556 raft_consensus.cc:1226] T 9b4926c9270e4754bac26cc4c3da4cf9 P 8586928e65c543c2b99cebf1dd7d700f [term 2 FOLLOWER]: Rejecting Update request from peer 27fddeff55ea476ebe7bd4881ae32035 for earlier term 1. Current term is 2. Ops: [1.1-1.1]
I20260809 15:45:18.638564 22361 leader_election.cc:302] T 9b4926c9270e4754bac26cc4c3da4cf9 P cab341909d314988b6893daac17235e7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8586928e65c543c2b99cebf1dd7d700f, cab341909d314988b6893daac17235e7; no voters:
I20260809 15:45:18.639417 22840 raft_consensus.cc:2781] T 9b4926c9270e4754bac26cc4c3da4cf9 P cab341909d314988b6893daac17235e7 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 15:45:18.639863 22840 raft_consensus.cc:686] T 9b4926c9270e4754bac26cc4c3da4cf9 P cab341909d314988b6893daac17235e7 [term 2 LEADER]: Becoming Leader. State: Replica: cab341909d314988b6893daac17235e7, State: Running, Role: LEADER
I20260809 15:45:18.640955 22689 raft_consensus.cc:2454] T 9b4926c9270e4754bac26cc4c3da4cf9 P 27fddeff55ea476ebe7bd4881ae32035 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab341909d314988b6893daac17235e7 in term 2.
I20260809 15:45:18.640641 22840 consensus_queue.cc:237] T 9b4926c9270e4754bac26cc4c3da4cf9 P cab341909d314988b6893daac17235e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } } peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } }
I20260809 15:45:18.646695 22242 catalog_manager.cc:5495] T 9b4926c9270e4754bac26cc4c3da4cf9 P cab341909d314988b6893daac17235e7 reported cstate change: term changed from 1 to 2, leader changed from 27fddeff55ea476ebe7bd4881ae32035 (127.20.202.67) to cab341909d314988b6893daac17235e7 (127.20.202.65). New cstate: current_term: 2 leader_uuid: "cab341909d314988b6893daac17235e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } health_report { overall_health: UNKNOWN } } }
I20260809 15:45:18.675208 22853 consensus_queue.cc:1035] T 36cb01872dee4485b764d163e0bf4e20 P 27fddeff55ea476ebe7bd4881ae32035 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:45:18.683895 22858 consensus_queue.cc:1035] T 36cb01872dee4485b764d163e0bf4e20 P 27fddeff55ea476ebe7bd4881ae32035 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:45:18.690575 22777 consensus_queue.cc:1035] T ab0d7bb4b86540b28101d1c813ee214e P 8586928e65c543c2b99cebf1dd7d700f [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:45:18.700266 22777 consensus_queue.cc:1035] T ab0d7bb4b86540b28101d1c813ee214e P 8586928e65c543c2b99cebf1dd7d700f [LEADER]: Connected to new peer: Peer: permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:45:18.792807 22858 consensus_queue.cc:1035] T 8c8c4f25c5364a919da21f5fca365c45 P 27fddeff55ea476ebe7bd4881ae32035 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:45:18.801599 22857 consensus_queue.cc:1035] T 8c8c4f25c5364a919da21f5fca365c45 P 27fddeff55ea476ebe7bd4881ae32035 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:45:18.808594 22857 raft_consensus.cc:979] T 8c8c4f25c5364a919da21f5fca365c45 P 27fddeff55ea476ebe7bd4881ae32035: : Instructing follower cab341909d314988b6893daac17235e7 to start an election
I20260809 15:45:18.808943 22772 raft_consensus.cc:1067] T 8c8c4f25c5364a919da21f5fca365c45 P 27fddeff55ea476ebe7bd4881ae32035 [term 1 LEADER]: Signalling peer cab341909d314988b6893daac17235e7 to start an election
I20260809 15:45:18.810220 22433 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8c8c4f25c5364a919da21f5fca365c45"
dest_uuid: "cab341909d314988b6893daac17235e7"
from {username='slave'} at 127.0.0.1:47182
I20260809 15:45:18.810788 22433 raft_consensus.cc:482] T 8c8c4f25c5364a919da21f5fca365c45 P cab341909d314988b6893daac17235e7 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 15:45:18.811159 22433 raft_consensus.cc:3037] T 8c8c4f25c5364a919da21f5fca365c45 P cab341909d314988b6893daac17235e7 [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:45:18.815454 22433 raft_consensus.cc:504] T 8c8c4f25c5364a919da21f5fca365c45 P cab341909d314988b6893daac17235e7 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } } peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } }
I20260809 15:45:18.817234 22433 leader_election.cc:288] T 8c8c4f25c5364a919da21f5fca365c45 P cab341909d314988b6893daac17235e7 [CANDIDATE]: Term 2 election: Requested vote from peers 8586928e65c543c2b99cebf1dd7d700f (127.20.202.66:45399), 27fddeff55ea476ebe7bd4881ae32035 (127.20.202.67:33583)
I20260809 15:45:18.817665 22556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c8c4f25c5364a919da21f5fca365c45" candidate_uuid: "cab341909d314988b6893daac17235e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8586928e65c543c2b99cebf1dd7d700f"
I20260809 15:45:18.817823 22689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c8c4f25c5364a919da21f5fca365c45" candidate_uuid: "cab341909d314988b6893daac17235e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "27fddeff55ea476ebe7bd4881ae32035"
I20260809 15:45:18.818220 22556 raft_consensus.cc:3037] T 8c8c4f25c5364a919da21f5fca365c45 P 8586928e65c543c2b99cebf1dd7d700f [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:45:18.818403 22689 raft_consensus.cc:3032] T 8c8c4f25c5364a919da21f5fca365c45 P 27fddeff55ea476ebe7bd4881ae32035 [term 1 LEADER]: Stepping down as leader of term 1
I20260809 15:45:18.818748 22689 raft_consensus.cc:729] T 8c8c4f25c5364a919da21f5fca365c45 P 27fddeff55ea476ebe7bd4881ae32035 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 27fddeff55ea476ebe7bd4881ae32035, State: Running, Role: LEADER
I20260809 15:45:18.819476 22689 consensus_queue.cc:260] T 8c8c4f25c5364a919da21f5fca365c45 P 27fddeff55ea476ebe7bd4881ae32035 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } } peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } }
I20260809 15:45:18.820616 22689 raft_consensus.cc:3037] T 8c8c4f25c5364a919da21f5fca365c45 P 27fddeff55ea476ebe7bd4881ae32035 [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:45:18.822865 22556 raft_consensus.cc:2454] T 8c8c4f25c5364a919da21f5fca365c45 P 8586928e65c543c2b99cebf1dd7d700f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab341909d314988b6893daac17235e7 in term 2.
I20260809 15:45:18.823627 22361 leader_election.cc:302] T 8c8c4f25c5364a919da21f5fca365c45 P cab341909d314988b6893daac17235e7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8586928e65c543c2b99cebf1dd7d700f, cab341909d314988b6893daac17235e7; no voters:
I20260809 15:45:18.824198 22840 raft_consensus.cc:2781] T 8c8c4f25c5364a919da21f5fca365c45 P cab341909d314988b6893daac17235e7 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 15:45:18.824635 22840 raft_consensus.cc:686] T 8c8c4f25c5364a919da21f5fca365c45 P cab341909d314988b6893daac17235e7 [term 2 LEADER]: Becoming Leader. State: Replica: cab341909d314988b6893daac17235e7, State: Running, Role: LEADER
I20260809 15:45:18.825618 22689 raft_consensus.cc:2454] T 8c8c4f25c5364a919da21f5fca365c45 P 27fddeff55ea476ebe7bd4881ae32035 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab341909d314988b6893daac17235e7 in term 2.
I20260809 15:45:18.825376 22840 consensus_queue.cc:237] T 8c8c4f25c5364a919da21f5fca365c45 P cab341909d314988b6893daac17235e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } } peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } }
I20260809 15:45:18.830724 22241 catalog_manager.cc:5495] T 8c8c4f25c5364a919da21f5fca365c45 P cab341909d314988b6893daac17235e7 reported cstate change: term changed from 1 to 2, leader changed from 27fddeff55ea476ebe7bd4881ae32035 (127.20.202.67) to cab341909d314988b6893daac17235e7 (127.20.202.65). New cstate: current_term: 2 leader_uuid: "cab341909d314988b6893daac17235e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } health_report { overall_health: UNKNOWN } } }
I20260809 15:45:18.957173 22777 raft_consensus.cc:979] T 6b1f47a4af1b4480ab2e69adb8be3cc3 P 8586928e65c543c2b99cebf1dd7d700f: : Instructing follower cab341909d314988b6893daac17235e7 to start an election
I20260809 15:45:18.957547 22796 raft_consensus.cc:1067] T 6b1f47a4af1b4480ab2e69adb8be3cc3 P 8586928e65c543c2b99cebf1dd7d700f [term 1 LEADER]: Signalling peer cab341909d314988b6893daac17235e7 to start an election
I20260809 15:45:18.958839 22433 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6b1f47a4af1b4480ab2e69adb8be3cc3"
dest_uuid: "cab341909d314988b6893daac17235e7"
from {username='slave'} at 127.0.0.1:47180
I20260809 15:45:18.959292 22433 raft_consensus.cc:482] T 6b1f47a4af1b4480ab2e69adb8be3cc3 P cab341909d314988b6893daac17235e7 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 15:45:18.959648 22433 raft_consensus.cc:3037] T 6b1f47a4af1b4480ab2e69adb8be3cc3 P cab341909d314988b6893daac17235e7 [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:45:18.961558 22777 raft_consensus.cc:979] T 4e0d833302ff456e96ab09aa12d4c5ba P 8586928e65c543c2b99cebf1dd7d700f: : Instructing follower cab341909d314988b6893daac17235e7 to start an election
I20260809 15:45:18.961951 22795 raft_consensus.cc:1067] T 4e0d833302ff456e96ab09aa12d4c5ba P 8586928e65c543c2b99cebf1dd7d700f [term 1 LEADER]: Signalling peer cab341909d314988b6893daac17235e7 to start an election
I20260809 15:45:18.963141 22436 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4e0d833302ff456e96ab09aa12d4c5ba"
dest_uuid: "cab341909d314988b6893daac17235e7"
from {username='slave'} at 127.0.0.1:47180
I20260809 15:45:18.963595 22436 raft_consensus.cc:482] T 4e0d833302ff456e96ab09aa12d4c5ba P cab341909d314988b6893daac17235e7 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 15:45:18.963850 22436 raft_consensus.cc:3037] T 4e0d833302ff456e96ab09aa12d4c5ba P cab341909d314988b6893daac17235e7 [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:45:18.965411 22433 raft_consensus.cc:504] T 6b1f47a4af1b4480ab2e69adb8be3cc3 P cab341909d314988b6893daac17235e7 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } } peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } }
I20260809 15:45:18.966909 22433 leader_election.cc:288] T 6b1f47a4af1b4480ab2e69adb8be3cc3 P cab341909d314988b6893daac17235e7 [CANDIDATE]: Term 2 election: Requested vote from peers 8586928e65c543c2b99cebf1dd7d700f (127.20.202.66:45399), 27fddeff55ea476ebe7bd4881ae32035 (127.20.202.67:33583)
I20260809 15:45:18.967592 22556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b1f47a4af1b4480ab2e69adb8be3cc3" candidate_uuid: "cab341909d314988b6893daac17235e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8586928e65c543c2b99cebf1dd7d700f"
I20260809 15:45:18.967833 22689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b1f47a4af1b4480ab2e69adb8be3cc3" candidate_uuid: "cab341909d314988b6893daac17235e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "27fddeff55ea476ebe7bd4881ae32035"
I20260809 15:45:18.968170 22556 raft_consensus.cc:3032] T 6b1f47a4af1b4480ab2e69adb8be3cc3 P 8586928e65c543c2b99cebf1dd7d700f [term 1 LEADER]: Stepping down as leader of term 1
I20260809 15:45:18.968325 22689 raft_consensus.cc:3037] T 6b1f47a4af1b4480ab2e69adb8be3cc3 P 27fddeff55ea476ebe7bd4881ae32035 [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:45:18.968546 22556 raft_consensus.cc:729] T 6b1f47a4af1b4480ab2e69adb8be3cc3 P 8586928e65c543c2b99cebf1dd7d700f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8586928e65c543c2b99cebf1dd7d700f, State: Running, Role: LEADER
I20260809 15:45:18.968412 22436 raft_consensus.cc:504] T 4e0d833302ff456e96ab09aa12d4c5ba P cab341909d314988b6893daac17235e7 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } } peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } }
I20260809 15:45:18.969341 22556 consensus_queue.cc:260] T 6b1f47a4af1b4480ab2e69adb8be3cc3 P 8586928e65c543c2b99cebf1dd7d700f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } } peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } }
I20260809 15:45:18.970340 22436 leader_election.cc:288] T 4e0d833302ff456e96ab09aa12d4c5ba P cab341909d314988b6893daac17235e7 [CANDIDATE]: Term 2 election: Requested vote from peers 8586928e65c543c2b99cebf1dd7d700f (127.20.202.66:45399), 27fddeff55ea476ebe7bd4881ae32035 (127.20.202.67:33583)
I20260809 15:45:18.970772 22555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e0d833302ff456e96ab09aa12d4c5ba" candidate_uuid: "cab341909d314988b6893daac17235e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8586928e65c543c2b99cebf1dd7d700f"
I20260809 15:45:18.970901 22690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e0d833302ff456e96ab09aa12d4c5ba" candidate_uuid: "cab341909d314988b6893daac17235e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "27fddeff55ea476ebe7bd4881ae32035"
I20260809 15:45:18.971549 22555 raft_consensus.cc:3032] T 4e0d833302ff456e96ab09aa12d4c5ba P 8586928e65c543c2b99cebf1dd7d700f [term 1 LEADER]: Stepping down as leader of term 1
I20260809 15:45:18.971632 22690 raft_consensus.cc:3037] T 4e0d833302ff456e96ab09aa12d4c5ba P 27fddeff55ea476ebe7bd4881ae32035 [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:45:18.971896 22555 raft_consensus.cc:729] T 4e0d833302ff456e96ab09aa12d4c5ba P 8586928e65c543c2b99cebf1dd7d700f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8586928e65c543c2b99cebf1dd7d700f, State: Running, Role: LEADER
I20260809 15:45:18.972681 22555 consensus_queue.cc:260] T 4e0d833302ff456e96ab09aa12d4c5ba P 8586928e65c543c2b99cebf1dd7d700f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } } peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } }
I20260809 15:45:18.973875 22555 raft_consensus.cc:3037] T 4e0d833302ff456e96ab09aa12d4c5ba P 8586928e65c543c2b99cebf1dd7d700f [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:45:18.974274 22689 raft_consensus.cc:2454] T 6b1f47a4af1b4480ab2e69adb8be3cc3 P 27fddeff55ea476ebe7bd4881ae32035 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab341909d314988b6893daac17235e7 in term 2.
I20260809 15:45:18.975580 22360 leader_election.cc:302] T 6b1f47a4af1b4480ab2e69adb8be3cc3 P cab341909d314988b6893daac17235e7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27fddeff55ea476ebe7bd4881ae32035, cab341909d314988b6893daac17235e7; no voters:
I20260809 15:45:18.976344 22840 raft_consensus.cc:2781] T 6b1f47a4af1b4480ab2e69adb8be3cc3 P cab341909d314988b6893daac17235e7 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 15:45:18.976735 22690 raft_consensus.cc:2454] T 4e0d833302ff456e96ab09aa12d4c5ba P 27fddeff55ea476ebe7bd4881ae32035 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab341909d314988b6893daac17235e7 in term 2.
I20260809 15:45:18.976781 22840 raft_consensus.cc:686] T 6b1f47a4af1b4480ab2e69adb8be3cc3 P cab341909d314988b6893daac17235e7 [term 2 LEADER]: Becoming Leader. State: Replica: cab341909d314988b6893daac17235e7, State: Running, Role: LEADER
I20260809 15:45:18.977413 22556 raft_consensus.cc:3037] T 6b1f47a4af1b4480ab2e69adb8be3cc3 P 8586928e65c543c2b99cebf1dd7d700f [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:45:18.977573 22840 consensus_queue.cc:237] T 6b1f47a4af1b4480ab2e69adb8be3cc3 P cab341909d314988b6893daac17235e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } } peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } }
I20260809 15:45:18.979192 22555 raft_consensus.cc:2454] T 4e0d833302ff456e96ab09aa12d4c5ba P 8586928e65c543c2b99cebf1dd7d700f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab341909d314988b6893daac17235e7 in term 2.
I20260809 15:45:18.980350 22360 leader_election.cc:302] T 4e0d833302ff456e96ab09aa12d4c5ba P cab341909d314988b6893daac17235e7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27fddeff55ea476ebe7bd4881ae32035, cab341909d314988b6893daac17235e7; no voters:
I20260809 15:45:18.981117 22840 raft_consensus.cc:2781] T 4e0d833302ff456e96ab09aa12d4c5ba P cab341909d314988b6893daac17235e7 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 15:45:18.981689 22840 raft_consensus.cc:686] T 4e0d833302ff456e96ab09aa12d4c5ba P cab341909d314988b6893daac17235e7 [term 2 LEADER]: Becoming Leader. State: Replica: cab341909d314988b6893daac17235e7, State: Running, Role: LEADER
I20260809 15:45:18.983047 22556 raft_consensus.cc:2454] T 6b1f47a4af1b4480ab2e69adb8be3cc3 P 8586928e65c543c2b99cebf1dd7d700f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab341909d314988b6893daac17235e7 in term 2.
I20260809 15:45:18.983003 22840 consensus_queue.cc:237] T 4e0d833302ff456e96ab09aa12d4c5ba P cab341909d314988b6893daac17235e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } } peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } }
I20260809 15:45:18.984373 22241 catalog_manager.cc:5495] T 6b1f47a4af1b4480ab2e69adb8be3cc3 P cab341909d314988b6893daac17235e7 reported cstate change: term changed from 1 to 2, leader changed from 8586928e65c543c2b99cebf1dd7d700f (127.20.202.66) to cab341909d314988b6893daac17235e7 (127.20.202.65). New cstate: current_term: 2 leader_uuid: "cab341909d314988b6893daac17235e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } health_report { overall_health: HEALTHY } } }
I20260809 15:45:18.994688 22241 catalog_manager.cc:5495] T 4e0d833302ff456e96ab09aa12d4c5ba P cab341909d314988b6893daac17235e7 reported cstate change: term changed from 1 to 2, leader changed from 8586928e65c543c2b99cebf1dd7d700f (127.20.202.66) to cab341909d314988b6893daac17235e7 (127.20.202.65). New cstate: current_term: 2 leader_uuid: "cab341909d314988b6893daac17235e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } health_report { overall_health: UNKNOWN } } }
I20260809 15:45:19.215755 22690 raft_consensus.cc:1261] T 9b4926c9270e4754bac26cc4c3da4cf9 P 27fddeff55ea476ebe7bd4881ae32035 [term 2 FOLLOWER]: Refusing update from remote peer cab341909d314988b6893daac17235e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 15:45:19.216789 22912 consensus_queue.cc:1035] T 9b4926c9270e4754bac26cc4c3da4cf9 P cab341909d314988b6893daac17235e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 15:45:19.219355 22556 raft_consensus.cc:1261] T 8c8c4f25c5364a919da21f5fca365c45 P 8586928e65c543c2b99cebf1dd7d700f [term 2 FOLLOWER]: Refusing update from remote peer cab341909d314988b6893daac17235e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 15:45:19.220593 22912 consensus_queue.cc:1035] T 8c8c4f25c5364a919da21f5fca365c45 P cab341909d314988b6893daac17235e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 15:45:19.225549 22555 raft_consensus.cc:1261] T 9b4926c9270e4754bac26cc4c3da4cf9 P 8586928e65c543c2b99cebf1dd7d700f [term 2 FOLLOWER]: Refusing update from remote peer cab341909d314988b6893daac17235e7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 15:45:19.227203 22912 consensus_queue.cc:1035] T 9b4926c9270e4754bac26cc4c3da4cf9 P cab341909d314988b6893daac17235e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:45:19.232935 22690 raft_consensus.cc:1261] T 8c8c4f25c5364a919da21f5fca365c45 P 27fddeff55ea476ebe7bd4881ae32035 [term 2 FOLLOWER]: Refusing update from remote peer cab341909d314988b6893daac17235e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 15:45:19.235749 22840 consensus_queue.cc:1035] T 8c8c4f25c5364a919da21f5fca365c45 P cab341909d314988b6893daac17235e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 15:45:19.379863 22690 raft_consensus.cc:1261] T 6b1f47a4af1b4480ab2e69adb8be3cc3 P 27fddeff55ea476ebe7bd4881ae32035 [term 2 FOLLOWER]: Refusing update from remote peer cab341909d314988b6893daac17235e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 15:45:19.380873 22840 consensus_queue.cc:1035] T 6b1f47a4af1b4480ab2e69adb8be3cc3 P cab341909d314988b6893daac17235e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 15:45:19.387194 22556 raft_consensus.cc:1261] T 6b1f47a4af1b4480ab2e69adb8be3cc3 P 8586928e65c543c2b99cebf1dd7d700f [term 2 FOLLOWER]: Refusing update from remote peer cab341909d314988b6893daac17235e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 15:45:19.389125 22914 consensus_queue.cc:1035] T 6b1f47a4af1b4480ab2e69adb8be3cc3 P cab341909d314988b6893daac17235e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 15:45:19.457628 22556 raft_consensus.cc:1261] T 4e0d833302ff456e96ab09aa12d4c5ba P 8586928e65c543c2b99cebf1dd7d700f [term 2 FOLLOWER]: Refusing update from remote peer cab341909d314988b6893daac17235e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 15:45:19.458653 22914 consensus_queue.cc:1035] T 4e0d833302ff456e96ab09aa12d4c5ba P cab341909d314988b6893daac17235e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 15:45:19.464787 22690 raft_consensus.cc:1261] T 4e0d833302ff456e96ab09aa12d4c5ba P 27fddeff55ea476ebe7bd4881ae32035 [term 2 FOLLOWER]: Refusing update from remote peer cab341909d314988b6893daac17235e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 15:45:19.466128 22844 consensus_queue.cc:1035] T 4e0d833302ff456e96ab09aa12d4c5ba P cab341909d314988b6893daac17235e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 15:45:21.549453 21289 auto_leader_rebalancer-test.cc:339] The leader distribution is
I20260809 15:45:21.550120 21289 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 15:45:21.565490 22436 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4e0d833302ff456e96ab09aa12d4c5ba"
dest_uuid: "cab341909d314988b6893daac17235e7"
mode: GRACEFUL
new_leader_uuid: "27fddeff55ea476ebe7bd4881ae32035"
from {username='slave'} at 127.0.0.1:40564
I20260809 15:45:21.565925 22436 raft_consensus.cc:595] T 4e0d833302ff456e96ab09aa12d4c5ba P cab341909d314988b6893daac17235e7 [term 2 LEADER]: Received request to transfer leadership to TS 27fddeff55ea476ebe7bd4881ae32035
I20260809 15:45:21.567764 22436 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6b1f47a4af1b4480ab2e69adb8be3cc3"
dest_uuid: "cab341909d314988b6893daac17235e7"
mode: GRACEFUL
new_leader_uuid: "27fddeff55ea476ebe7bd4881ae32035"
from {username='slave'} at 127.0.0.1:40564
I20260809 15:45:21.568152 22436 raft_consensus.cc:595] T 6b1f47a4af1b4480ab2e69adb8be3cc3 P cab341909d314988b6893daac17235e7 [term 2 LEADER]: Received request to transfer leadership to TS 27fddeff55ea476ebe7bd4881ae32035
I20260809 15:45:21.568926 21289 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260809 15:45:21.569303 21289 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 15:45:21.582194 22840 raft_consensus.cc:979] T 6b1f47a4af1b4480ab2e69adb8be3cc3 P cab341909d314988b6893daac17235e7: : Instructing follower 27fddeff55ea476ebe7bd4881ae32035 to start an election
I20260809 15:45:21.582571 22922 raft_consensus.cc:1067] T 6b1f47a4af1b4480ab2e69adb8be3cc3 P cab341909d314988b6893daac17235e7 [term 2 LEADER]: Signalling peer 27fddeff55ea476ebe7bd4881ae32035 to start an election
I20260809 15:45:21.583791 22690 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6b1f47a4af1b4480ab2e69adb8be3cc3"
dest_uuid: "27fddeff55ea476ebe7bd4881ae32035"
from {username='slave'} at 127.0.0.1:51580
I20260809 15:45:21.584182 22690 raft_consensus.cc:482] T 6b1f47a4af1b4480ab2e69adb8be3cc3 P 27fddeff55ea476ebe7bd4881ae32035 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 15:45:21.584431 22690 raft_consensus.cc:3037] T 6b1f47a4af1b4480ab2e69adb8be3cc3 P 27fddeff55ea476ebe7bd4881ae32035 [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:45:21.588691 22690 raft_consensus.cc:504] T 6b1f47a4af1b4480ab2e69adb8be3cc3 P 27fddeff55ea476ebe7bd4881ae32035 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } } peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } }
I20260809 15:45:21.590090 22690 leader_election.cc:288] T 6b1f47a4af1b4480ab2e69adb8be3cc3 P 27fddeff55ea476ebe7bd4881ae32035 [CANDIDATE]: Term 3 election: Requested vote from peers 8586928e65c543c2b99cebf1dd7d700f (127.20.202.66:45399), cab341909d314988b6893daac17235e7 (127.20.202.65:46511)
I20260809 15:45:21.590827 22556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b1f47a4af1b4480ab2e69adb8be3cc3" candidate_uuid: "27fddeff55ea476ebe7bd4881ae32035" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "8586928e65c543c2b99cebf1dd7d700f"
I20260809 15:45:21.590924 22436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b1f47a4af1b4480ab2e69adb8be3cc3" candidate_uuid: "27fddeff55ea476ebe7bd4881ae32035" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "cab341909d314988b6893daac17235e7"
I20260809 15:45:21.591364 22556 raft_consensus.cc:3037] T 6b1f47a4af1b4480ab2e69adb8be3cc3 P 8586928e65c543c2b99cebf1dd7d700f [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:45:21.591501 22436 raft_consensus.cc:3032] T 6b1f47a4af1b4480ab2e69adb8be3cc3 P cab341909d314988b6893daac17235e7 [term 2 LEADER]: Stepping down as leader of term 2
I20260809 15:45:21.591797 22436 raft_consensus.cc:729] T 6b1f47a4af1b4480ab2e69adb8be3cc3 P cab341909d314988b6893daac17235e7 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: cab341909d314988b6893daac17235e7, State: Running, Role: LEADER
I20260809 15:45:21.592350 22436 consensus_queue.cc:260] T 6b1f47a4af1b4480ab2e69adb8be3cc3 P cab341909d314988b6893daac17235e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } } peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } }
I20260809 15:45:21.593192 22436 raft_consensus.cc:3037] T 6b1f47a4af1b4480ab2e69adb8be3cc3 P cab341909d314988b6893daac17235e7 [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:45:21.595589 22556 raft_consensus.cc:2454] T 6b1f47a4af1b4480ab2e69adb8be3cc3 P 8586928e65c543c2b99cebf1dd7d700f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27fddeff55ea476ebe7bd4881ae32035 in term 3.
I20260809 15:45:21.596480 22635 leader_election.cc:302] T 6b1f47a4af1b4480ab2e69adb8be3cc3 P 27fddeff55ea476ebe7bd4881ae32035 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27fddeff55ea476ebe7bd4881ae32035, 8586928e65c543c2b99cebf1dd7d700f; no voters:
I20260809 15:45:21.597194 22772 raft_consensus.cc:2781] T 6b1f47a4af1b4480ab2e69adb8be3cc3 P 27fddeff55ea476ebe7bd4881ae32035 [term 3 FOLLOWER]: Leader election won for term 3
I20260809 15:45:21.597491 22436 raft_consensus.cc:2454] T 6b1f47a4af1b4480ab2e69adb8be3cc3 P cab341909d314988b6893daac17235e7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27fddeff55ea476ebe7bd4881ae32035 in term 3.
I20260809 15:45:21.598083 22772 raft_consensus.cc:686] T 6b1f47a4af1b4480ab2e69adb8be3cc3 P 27fddeff55ea476ebe7bd4881ae32035 [term 3 LEADER]: Becoming Leader. State: Replica: 27fddeff55ea476ebe7bd4881ae32035, State: Running, Role: LEADER
I20260809 15:45:21.598795 22772 consensus_queue.cc:237] T 6b1f47a4af1b4480ab2e69adb8be3cc3 P 27fddeff55ea476ebe7bd4881ae32035 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } } peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } }
I20260809 15:45:21.604902 22244 catalog_manager.cc:5495] T 6b1f47a4af1b4480ab2e69adb8be3cc3 P 27fddeff55ea476ebe7bd4881ae32035 reported cstate change: term changed from 2 to 3, leader changed from cab341909d314988b6893daac17235e7 (127.20.202.65) to 27fddeff55ea476ebe7bd4881ae32035 (127.20.202.67). New cstate: current_term: 3 leader_uuid: "27fddeff55ea476ebe7bd4881ae32035" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } health_report { overall_health: UNKNOWN } } }
I20260809 15:45:21.983994 22840 raft_consensus.cc:979] T 4e0d833302ff456e96ab09aa12d4c5ba P cab341909d314988b6893daac17235e7: : Instructing follower 27fddeff55ea476ebe7bd4881ae32035 to start an election
I20260809 15:45:21.984377 22922 raft_consensus.cc:1067] T 4e0d833302ff456e96ab09aa12d4c5ba P cab341909d314988b6893daac17235e7 [term 2 LEADER]: Signalling peer 27fddeff55ea476ebe7bd4881ae32035 to start an election
I20260809 15:45:21.985540 22689 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4e0d833302ff456e96ab09aa12d4c5ba"
dest_uuid: "27fddeff55ea476ebe7bd4881ae32035"
from {username='slave'} at 127.0.0.1:51580
I20260809 15:45:21.986032 22689 raft_consensus.cc:482] T 4e0d833302ff456e96ab09aa12d4c5ba P 27fddeff55ea476ebe7bd4881ae32035 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 15:45:21.986285 22689 raft_consensus.cc:3037] T 4e0d833302ff456e96ab09aa12d4c5ba P 27fddeff55ea476ebe7bd4881ae32035 [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:45:21.991068 22689 raft_consensus.cc:504] T 4e0d833302ff456e96ab09aa12d4c5ba P 27fddeff55ea476ebe7bd4881ae32035 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } } peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } }
I20260809 15:45:21.992504 22689 leader_election.cc:288] T 4e0d833302ff456e96ab09aa12d4c5ba P 27fddeff55ea476ebe7bd4881ae32035 [CANDIDATE]: Term 3 election: Requested vote from peers cab341909d314988b6893daac17235e7 (127.20.202.65:46511), 8586928e65c543c2b99cebf1dd7d700f (127.20.202.66:45399)
I20260809 15:45:21.993156 22436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e0d833302ff456e96ab09aa12d4c5ba" candidate_uuid: "27fddeff55ea476ebe7bd4881ae32035" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "cab341909d314988b6893daac17235e7"
I20260809 15:45:21.993491 22556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e0d833302ff456e96ab09aa12d4c5ba" candidate_uuid: "27fddeff55ea476ebe7bd4881ae32035" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "8586928e65c543c2b99cebf1dd7d700f"
I20260809 15:45:21.993716 22436 raft_consensus.cc:3032] T 4e0d833302ff456e96ab09aa12d4c5ba P cab341909d314988b6893daac17235e7 [term 2 LEADER]: Stepping down as leader of term 2
I20260809 15:45:21.994028 22556 raft_consensus.cc:3037] T 4e0d833302ff456e96ab09aa12d4c5ba P 8586928e65c543c2b99cebf1dd7d700f [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:45:21.994021 22436 raft_consensus.cc:729] T 4e0d833302ff456e96ab09aa12d4c5ba P cab341909d314988b6893daac17235e7 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: cab341909d314988b6893daac17235e7, State: Running, Role: LEADER
I20260809 15:45:21.994910 22436 consensus_queue.cc:260] T 4e0d833302ff456e96ab09aa12d4c5ba P cab341909d314988b6893daac17235e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } } peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } }
I20260809 15:45:21.996116 22436 raft_consensus.cc:3037] T 4e0d833302ff456e96ab09aa12d4c5ba P cab341909d314988b6893daac17235e7 [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:45:22.000183 22556 raft_consensus.cc:2454] T 4e0d833302ff456e96ab09aa12d4c5ba P 8586928e65c543c2b99cebf1dd7d700f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27fddeff55ea476ebe7bd4881ae32035 in term 3.
I20260809 15:45:22.001654 22635 leader_election.cc:302] T 4e0d833302ff456e96ab09aa12d4c5ba P 27fddeff55ea476ebe7bd4881ae32035 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27fddeff55ea476ebe7bd4881ae32035, 8586928e65c543c2b99cebf1dd7d700f; no voters:
I20260809 15:45:22.002391 22436 raft_consensus.cc:2454] T 4e0d833302ff456e96ab09aa12d4c5ba P cab341909d314988b6893daac17235e7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27fddeff55ea476ebe7bd4881ae32035 in term 3.
I20260809 15:45:22.002453 22772 raft_consensus.cc:2781] T 4e0d833302ff456e96ab09aa12d4c5ba P 27fddeff55ea476ebe7bd4881ae32035 [term 3 FOLLOWER]: Leader election won for term 3
I20260809 15:45:22.003095 22772 raft_consensus.cc:686] T 4e0d833302ff456e96ab09aa12d4c5ba P 27fddeff55ea476ebe7bd4881ae32035 [term 3 LEADER]: Becoming Leader. State: Replica: 27fddeff55ea476ebe7bd4881ae32035, State: Running, Role: LEADER
I20260809 15:45:22.003916 22772 consensus_queue.cc:237] T 4e0d833302ff456e96ab09aa12d4c5ba P 27fddeff55ea476ebe7bd4881ae32035 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } } peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } }
I20260809 15:45:22.011463 22244 catalog_manager.cc:5495] T 4e0d833302ff456e96ab09aa12d4c5ba P 27fddeff55ea476ebe7bd4881ae32035 reported cstate change: term changed from 2 to 3, leader changed from cab341909d314988b6893daac17235e7 (127.20.202.65) to 27fddeff55ea476ebe7bd4881ae32035 (127.20.202.67). New cstate: current_term: 3 leader_uuid: "27fddeff55ea476ebe7bd4881ae32035" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } health_report { overall_health: HEALTHY } } }
I20260809 15:45:22.204355 22436 raft_consensus.cc:1261] T 6b1f47a4af1b4480ab2e69adb8be3cc3 P cab341909d314988b6893daac17235e7 [term 3 FOLLOWER]: Refusing update from remote peer 27fddeff55ea476ebe7bd4881ae32035: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260809 15:45:22.205360 22772 consensus_queue.cc:1035] T 6b1f47a4af1b4480ab2e69adb8be3cc3 P 27fddeff55ea476ebe7bd4881ae32035 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 15:45:22.214130 22556 raft_consensus.cc:1261] T 6b1f47a4af1b4480ab2e69adb8be3cc3 P 8586928e65c543c2b99cebf1dd7d700f [term 3 FOLLOWER]: Refusing update from remote peer 27fddeff55ea476ebe7bd4881ae32035: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260809 15:45:22.216063 22924 consensus_queue.cc:1035] T 6b1f47a4af1b4480ab2e69adb8be3cc3 P 27fddeff55ea476ebe7bd4881ae32035 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 15:45:22.479465 22436 raft_consensus.cc:1261] T 4e0d833302ff456e96ab09aa12d4c5ba P cab341909d314988b6893daac17235e7 [term 3 FOLLOWER]: Refusing update from remote peer 27fddeff55ea476ebe7bd4881ae32035: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260809 15:45:22.480546 22936 consensus_queue.cc:1035] T 4e0d833302ff456e96ab09aa12d4c5ba P 27fddeff55ea476ebe7bd4881ae32035 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 15:45:22.488046 22556 raft_consensus.cc:1261] T 4e0d833302ff456e96ab09aa12d4c5ba P 8586928e65c543c2b99cebf1dd7d700f [term 3 FOLLOWER]: Refusing update from remote peer 27fddeff55ea476ebe7bd4881ae32035: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260809 15:45:22.489588 22924 consensus_queue.cc:1035] T 4e0d833302ff456e96ab09aa12d4c5ba P 27fddeff55ea476ebe7bd4881ae32035 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 15:45:22.569996 21289 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 15:45:22.575985 21289 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260809 15:45:22.576335 21289 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 15:45:23.577003 21289 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 15:45:23.581980 21289 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260809 15:45:23.582330 21289 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 15:45:24.583007 21289 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 15:45:24.587968 21289 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260809 15:45:24.588322 21289 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 15:45:25.589002 21289 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 15:45:25.594158 21289 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260809 15:45:25.594508 21289 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 15:45:26.595677 21289 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 15:45:26.601905 21289 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260809 15:45:26.602392 21289 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 15:45:27.603138 21289 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 15:45:27.607909 21289 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260809 15:45:27.608258 21289 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 15:45:28.608892 21289 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 15:45:28.613797 21289 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260809 15:45:28.614154 21289 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 15:45:29.614816 21289 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 15:45:29.620332 21289 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260809 15:45:29.620714 21289 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 15:45:30.621371 21289 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 15:45:30.626284 21289 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260809 15:45:30.626631 21289 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 15:45:31.654001 21289 auto_leader_rebalancer-test.cc:358] The leader distribution is
I20260809 15:45:31.657852 22690 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6b1f47a4af1b4480ab2e69adb8be3cc3"
dest_uuid: "27fddeff55ea476ebe7bd4881ae32035"
mode: GRACEFUL
new_leader_uuid: "cab341909d314988b6893daac17235e7"
from {username='slave'} at 127.0.0.1:51590
I20260809 15:45:31.658457 22690 raft_consensus.cc:595] T 6b1f47a4af1b4480ab2e69adb8be3cc3 P 27fddeff55ea476ebe7bd4881ae32035 [term 3 LEADER]: Received request to transfer leadership to TS cab341909d314988b6893daac17235e7
I20260809 15:45:31.668408 22690 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4e0d833302ff456e96ab09aa12d4c5ba"
dest_uuid: "27fddeff55ea476ebe7bd4881ae32035"
mode: GRACEFUL
new_leader_uuid: "cab341909d314988b6893daac17235e7"
from {username='slave'} at 127.0.0.1:51590
I20260809 15:45:31.668798 22690 raft_consensus.cc:595] T 4e0d833302ff456e96ab09aa12d4c5ba P 27fddeff55ea476ebe7bd4881ae32035 [term 3 LEADER]: Received request to transfer leadership to TS cab341909d314988b6893daac17235e7
I20260809 15:45:31.672597 22556 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ab0d7bb4b86540b28101d1c813ee214e"
dest_uuid: "8586928e65c543c2b99cebf1dd7d700f"
mode: GRACEFUL
new_leader_uuid: "cab341909d314988b6893daac17235e7"
from {username='slave'} at 127.0.0.1:56494
I20260809 15:45:31.672971 22556 raft_consensus.cc:595] T ab0d7bb4b86540b28101d1c813ee214e P 8586928e65c543c2b99cebf1dd7d700f [term 1 LEADER]: Received request to transfer leadership to TS cab341909d314988b6893daac17235e7
I20260809 15:45:31.676761 22556 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6d6e21b2d7b746469c42c6d80ad8e001"
dest_uuid: "8586928e65c543c2b99cebf1dd7d700f"
mode: GRACEFUL
new_leader_uuid: "cab341909d314988b6893daac17235e7"
from {username='slave'} at 127.0.0.1:56494
I20260809 15:45:31.677122 22556 raft_consensus.cc:595] T 6d6e21b2d7b746469c42c6d80ad8e001 P 8586928e65c543c2b99cebf1dd7d700f [term 1 LEADER]: Received request to transfer leadership to TS cab341909d314988b6893daac17235e7
I20260809 15:45:31.680747 22556 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "21aa1094e1cf4d40b51e5916cb6a2a96"
dest_uuid: "8586928e65c543c2b99cebf1dd7d700f"
mode: GRACEFUL
new_leader_uuid: "cab341909d314988b6893daac17235e7"
from {username='slave'} at 127.0.0.1:56494
I20260809 15:45:31.681118 22556 raft_consensus.cc:595] T 21aa1094e1cf4d40b51e5916cb6a2a96 P 8586928e65c543c2b99cebf1dd7d700f [term 1 LEADER]: Received request to transfer leadership to TS cab341909d314988b6893daac17235e7
I20260809 15:45:31.741437 22961 raft_consensus.cc:979] T 6d6e21b2d7b746469c42c6d80ad8e001 P 8586928e65c543c2b99cebf1dd7d700f: : Instructing follower cab341909d314988b6893daac17235e7 to start an election
I20260809 15:45:31.741981 22958 raft_consensus.cc:1067] T 6d6e21b2d7b746469c42c6d80ad8e001 P 8586928e65c543c2b99cebf1dd7d700f [term 1 LEADER]: Signalling peer cab341909d314988b6893daac17235e7 to start an election
I20260809 15:45:31.743384 22436 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6d6e21b2d7b746469c42c6d80ad8e001"
dest_uuid: "cab341909d314988b6893daac17235e7"
from {username='slave'} at 127.0.0.1:47180
I20260809 15:45:31.743764 22436 raft_consensus.cc:482] T 6d6e21b2d7b746469c42c6d80ad8e001 P cab341909d314988b6893daac17235e7 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 15:45:31.744045 22436 raft_consensus.cc:3037] T 6d6e21b2d7b746469c42c6d80ad8e001 P cab341909d314988b6893daac17235e7 [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:45:31.749334 22436 raft_consensus.cc:504] T 6d6e21b2d7b746469c42c6d80ad8e001 P cab341909d314988b6893daac17235e7 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } } peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } }
I20260809 15:45:31.751072 22436 leader_election.cc:288] T 6d6e21b2d7b746469c42c6d80ad8e001 P cab341909d314988b6893daac17235e7 [CANDIDATE]: Term 2 election: Requested vote from peers 8586928e65c543c2b99cebf1dd7d700f (127.20.202.66:45399), 27fddeff55ea476ebe7bd4881ae32035 (127.20.202.67:33583)
I20260809 15:45:31.751827 22556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d6e21b2d7b746469c42c6d80ad8e001" candidate_uuid: "cab341909d314988b6893daac17235e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8586928e65c543c2b99cebf1dd7d700f"
I20260809 15:45:31.752194 22690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d6e21b2d7b746469c42c6d80ad8e001" candidate_uuid: "cab341909d314988b6893daac17235e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "27fddeff55ea476ebe7bd4881ae32035"
I20260809 15:45:31.752462 22556 raft_consensus.cc:3032] T 6d6e21b2d7b746469c42c6d80ad8e001 P 8586928e65c543c2b99cebf1dd7d700f [term 1 LEADER]: Stepping down as leader of term 1
I20260809 15:45:31.752665 22690 raft_consensus.cc:3037] T 6d6e21b2d7b746469c42c6d80ad8e001 P 27fddeff55ea476ebe7bd4881ae32035 [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:45:31.752760 22556 raft_consensus.cc:729] T 6d6e21b2d7b746469c42c6d80ad8e001 P 8586928e65c543c2b99cebf1dd7d700f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8586928e65c543c2b99cebf1dd7d700f, State: Running, Role: LEADER
I20260809 15:45:31.753585 22556 consensus_queue.cc:260] T 6d6e21b2d7b746469c42c6d80ad8e001 P 8586928e65c543c2b99cebf1dd7d700f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } } peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } }
I20260809 15:45:31.754732 22556 raft_consensus.cc:3037] T 6d6e21b2d7b746469c42c6d80ad8e001 P 8586928e65c543c2b99cebf1dd7d700f [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:45:31.757226 22690 raft_consensus.cc:2454] T 6d6e21b2d7b746469c42c6d80ad8e001 P 27fddeff55ea476ebe7bd4881ae32035 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab341909d314988b6893daac17235e7 in term 2.
I20260809 15:45:31.758241 22360 leader_election.cc:302] T 6d6e21b2d7b746469c42c6d80ad8e001 P cab341909d314988b6893daac17235e7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27fddeff55ea476ebe7bd4881ae32035, cab341909d314988b6893daac17235e7; no voters:
I20260809 15:45:31.758977 22950 raft_consensus.cc:2781] T 6d6e21b2d7b746469c42c6d80ad8e001 P cab341909d314988b6893daac17235e7 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 15:45:31.759215 22556 raft_consensus.cc:2454] T 6d6e21b2d7b746469c42c6d80ad8e001 P 8586928e65c543c2b99cebf1dd7d700f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab341909d314988b6893daac17235e7 in term 2.
I20260809 15:45:31.759891 22950 raft_consensus.cc:686] T 6d6e21b2d7b746469c42c6d80ad8e001 P cab341909d314988b6893daac17235e7 [term 2 LEADER]: Becoming Leader. State: Replica: cab341909d314988b6893daac17235e7, State: Running, Role: LEADER
I20260809 15:45:31.760736 22950 consensus_queue.cc:237] T 6d6e21b2d7b746469c42c6d80ad8e001 P cab341909d314988b6893daac17235e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } } peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } }
I20260809 15:45:31.766630 22242 catalog_manager.cc:5495] T 6d6e21b2d7b746469c42c6d80ad8e001 P cab341909d314988b6893daac17235e7 reported cstate change: term changed from 1 to 2, leader changed from 8586928e65c543c2b99cebf1dd7d700f (127.20.202.66) to cab341909d314988b6893daac17235e7 (127.20.202.65). New cstate: current_term: 2 leader_uuid: "cab341909d314988b6893daac17235e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } health_report { overall_health: UNKNOWN } } }
I20260809 15:45:31.945297 22961 raft_consensus.cc:979] T 21aa1094e1cf4d40b51e5916cb6a2a96 P 8586928e65c543c2b99cebf1dd7d700f: : Instructing follower cab341909d314988b6893daac17235e7 to start an election
I20260809 15:45:31.945674 22961 raft_consensus.cc:1067] T 21aa1094e1cf4d40b51e5916cb6a2a96 P 8586928e65c543c2b99cebf1dd7d700f [term 1 LEADER]: Signalling peer cab341909d314988b6893daac17235e7 to start an election
I20260809 15:45:31.946998 22436 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "21aa1094e1cf4d40b51e5916cb6a2a96"
dest_uuid: "cab341909d314988b6893daac17235e7"
from {username='slave'} at 127.0.0.1:47180
I20260809 15:45:31.947474 22436 raft_consensus.cc:482] T 21aa1094e1cf4d40b51e5916cb6a2a96 P cab341909d314988b6893daac17235e7 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 15:45:31.947711 22436 raft_consensus.cc:3037] T 21aa1094e1cf4d40b51e5916cb6a2a96 P cab341909d314988b6893daac17235e7 [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:45:31.951994 22436 raft_consensus.cc:504] T 21aa1094e1cf4d40b51e5916cb6a2a96 P cab341909d314988b6893daac17235e7 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } } peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } }
I20260809 15:45:31.953178 22436 leader_election.cc:288] T 21aa1094e1cf4d40b51e5916cb6a2a96 P cab341909d314988b6893daac17235e7 [CANDIDATE]: Term 2 election: Requested vote from peers 8586928e65c543c2b99cebf1dd7d700f (127.20.202.66:45399), 27fddeff55ea476ebe7bd4881ae32035 (127.20.202.67:33583)
I20260809 15:45:31.953755 22556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21aa1094e1cf4d40b51e5916cb6a2a96" candidate_uuid: "cab341909d314988b6893daac17235e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8586928e65c543c2b99cebf1dd7d700f"
I20260809 15:45:31.953979 22690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21aa1094e1cf4d40b51e5916cb6a2a96" candidate_uuid: "cab341909d314988b6893daac17235e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "27fddeff55ea476ebe7bd4881ae32035"
I20260809 15:45:31.954277 22556 raft_consensus.cc:3032] T 21aa1094e1cf4d40b51e5916cb6a2a96 P 8586928e65c543c2b99cebf1dd7d700f [term 1 LEADER]: Stepping down as leader of term 1
I20260809 15:45:31.954452 22690 raft_consensus.cc:3037] T 21aa1094e1cf4d40b51e5916cb6a2a96 P 27fddeff55ea476ebe7bd4881ae32035 [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:45:31.954581 22556 raft_consensus.cc:729] T 21aa1094e1cf4d40b51e5916cb6a2a96 P 8586928e65c543c2b99cebf1dd7d700f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8586928e65c543c2b99cebf1dd7d700f, State: Running, Role: LEADER
I20260809 15:45:31.955296 22556 consensus_queue.cc:260] T 21aa1094e1cf4d40b51e5916cb6a2a96 P 8586928e65c543c2b99cebf1dd7d700f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } } peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } }
I20260809 15:45:31.956380 22556 raft_consensus.cc:3037] T 21aa1094e1cf4d40b51e5916cb6a2a96 P 8586928e65c543c2b99cebf1dd7d700f [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:45:31.960505 22690 raft_consensus.cc:2454] T 21aa1094e1cf4d40b51e5916cb6a2a96 P 27fddeff55ea476ebe7bd4881ae32035 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab341909d314988b6893daac17235e7 in term 2.
I20260809 15:45:31.961342 22360 leader_election.cc:302] T 21aa1094e1cf4d40b51e5916cb6a2a96 P cab341909d314988b6893daac17235e7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27fddeff55ea476ebe7bd4881ae32035, cab341909d314988b6893daac17235e7; no voters:
I20260809 15:45:31.961928 22950 raft_consensus.cc:2781] T 21aa1094e1cf4d40b51e5916cb6a2a96 P cab341909d314988b6893daac17235e7 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 15:45:31.961951 22556 raft_consensus.cc:2454] T 21aa1094e1cf4d40b51e5916cb6a2a96 P 8586928e65c543c2b99cebf1dd7d700f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab341909d314988b6893daac17235e7 in term 2.
I20260809 15:45:31.962500 22950 raft_consensus.cc:686] T 21aa1094e1cf4d40b51e5916cb6a2a96 P cab341909d314988b6893daac17235e7 [term 2 LEADER]: Becoming Leader. State: Replica: cab341909d314988b6893daac17235e7, State: Running, Role: LEADER
I20260809 15:45:31.963390 22950 consensus_queue.cc:237] T 21aa1094e1cf4d40b51e5916cb6a2a96 P cab341909d314988b6893daac17235e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } } peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } }
I20260809 15:45:31.969267 22242 catalog_manager.cc:5495] T 21aa1094e1cf4d40b51e5916cb6a2a96 P cab341909d314988b6893daac17235e7 reported cstate change: term changed from 1 to 2, leader changed from 8586928e65c543c2b99cebf1dd7d700f (127.20.202.66) to cab341909d314988b6893daac17235e7 (127.20.202.65). New cstate: current_term: 2 leader_uuid: "cab341909d314988b6893daac17235e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } health_report { overall_health: UNKNOWN } } }
I20260809 15:45:32.008952 22949 raft_consensus.cc:979] T 4e0d833302ff456e96ab09aa12d4c5ba P 27fddeff55ea476ebe7bd4881ae32035: : Instructing follower cab341909d314988b6893daac17235e7 to start an election
I20260809 15:45:32.009357 22972 raft_consensus.cc:1067] T 4e0d833302ff456e96ab09aa12d4c5ba P 27fddeff55ea476ebe7bd4881ae32035 [term 3 LEADER]: Signalling peer cab341909d314988b6893daac17235e7 to start an election
I20260809 15:45:32.010521 22436 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4e0d833302ff456e96ab09aa12d4c5ba"
dest_uuid: "cab341909d314988b6893daac17235e7"
from {username='slave'} at 127.0.0.1:47182
I20260809 15:45:32.010888 22436 raft_consensus.cc:482] T 4e0d833302ff456e96ab09aa12d4c5ba P cab341909d314988b6893daac17235e7 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 15:45:32.011180 22436 raft_consensus.cc:3037] T 4e0d833302ff456e96ab09aa12d4c5ba P cab341909d314988b6893daac17235e7 [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:45:32.014834 22436 raft_consensus.cc:504] T 4e0d833302ff456e96ab09aa12d4c5ba P cab341909d314988b6893daac17235e7 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } } peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } }
I20260809 15:45:32.016000 22436 leader_election.cc:288] T 4e0d833302ff456e96ab09aa12d4c5ba P cab341909d314988b6893daac17235e7 [CANDIDATE]: Term 4 election: Requested vote from peers 8586928e65c543c2b99cebf1dd7d700f (127.20.202.66:45399), 27fddeff55ea476ebe7bd4881ae32035 (127.20.202.67:33583)
I20260809 15:45:32.016692 22556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e0d833302ff456e96ab09aa12d4c5ba" candidate_uuid: "cab341909d314988b6893daac17235e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "8586928e65c543c2b99cebf1dd7d700f"
I20260809 15:45:32.016808 22690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e0d833302ff456e96ab09aa12d4c5ba" candidate_uuid: "cab341909d314988b6893daac17235e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "27fddeff55ea476ebe7bd4881ae32035"
I20260809 15:45:32.017151 22556 raft_consensus.cc:3037] T 4e0d833302ff456e96ab09aa12d4c5ba P 8586928e65c543c2b99cebf1dd7d700f [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:45:32.017241 22690 raft_consensus.cc:3032] T 4e0d833302ff456e96ab09aa12d4c5ba P 27fddeff55ea476ebe7bd4881ae32035 [term 3 LEADER]: Stepping down as leader of term 3
I20260809 15:45:32.017498 22690 raft_consensus.cc:729] T 4e0d833302ff456e96ab09aa12d4c5ba P 27fddeff55ea476ebe7bd4881ae32035 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 27fddeff55ea476ebe7bd4881ae32035, State: Running, Role: LEADER
I20260809 15:45:32.018069 22690 consensus_queue.cc:260] T 4e0d833302ff456e96ab09aa12d4c5ba P 27fddeff55ea476ebe7bd4881ae32035 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } } peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } }
I20260809 15:45:32.018959 22690 raft_consensus.cc:3037] T 4e0d833302ff456e96ab09aa12d4c5ba P 27fddeff55ea476ebe7bd4881ae32035 [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:45:32.021354 22556 raft_consensus.cc:2454] T 4e0d833302ff456e96ab09aa12d4c5ba P 8586928e65c543c2b99cebf1dd7d700f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab341909d314988b6893daac17235e7 in term 4.
I20260809 15:45:32.022184 22361 leader_election.cc:302] T 4e0d833302ff456e96ab09aa12d4c5ba P cab341909d314988b6893daac17235e7 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8586928e65c543c2b99cebf1dd7d700f, cab341909d314988b6893daac17235e7; no voters:
I20260809 15:45:32.022876 22950 raft_consensus.cc:2781] T 4e0d833302ff456e96ab09aa12d4c5ba P cab341909d314988b6893daac17235e7 [term 4 FOLLOWER]: Leader election won for term 4
I20260809 15:45:32.023240 22950 raft_consensus.cc:686] T 4e0d833302ff456e96ab09aa12d4c5ba P cab341909d314988b6893daac17235e7 [term 4 LEADER]: Becoming Leader. State: Replica: cab341909d314988b6893daac17235e7, State: Running, Role: LEADER
I20260809 15:45:32.023680 22690 raft_consensus.cc:2454] T 4e0d833302ff456e96ab09aa12d4c5ba P 27fddeff55ea476ebe7bd4881ae32035 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab341909d314988b6893daac17235e7 in term 4.
I20260809 15:45:32.023947 22950 consensus_queue.cc:237] T 4e0d833302ff456e96ab09aa12d4c5ba P cab341909d314988b6893daac17235e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } } peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } }
I20260809 15:45:32.030099 22242 catalog_manager.cc:5495] T 4e0d833302ff456e96ab09aa12d4c5ba P cab341909d314988b6893daac17235e7 reported cstate change: term changed from 3 to 4, leader changed from 27fddeff55ea476ebe7bd4881ae32035 (127.20.202.67) to cab341909d314988b6893daac17235e7 (127.20.202.65). New cstate: current_term: 4 leader_uuid: "cab341909d314988b6893daac17235e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } health_report { overall_health: UNKNOWN } } }
I20260809 15:45:32.048733 22949 raft_consensus.cc:979] T 6b1f47a4af1b4480ab2e69adb8be3cc3 P 27fddeff55ea476ebe7bd4881ae32035: : Instructing follower cab341909d314988b6893daac17235e7 to start an election
I20260809 15:45:32.049149 22949 raft_consensus.cc:1067] T 6b1f47a4af1b4480ab2e69adb8be3cc3 P 27fddeff55ea476ebe7bd4881ae32035 [term 3 LEADER]: Signalling peer cab341909d314988b6893daac17235e7 to start an election
I20260809 15:45:32.050479 22436 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6b1f47a4af1b4480ab2e69adb8be3cc3"
dest_uuid: "cab341909d314988b6893daac17235e7"
from {username='slave'} at 127.0.0.1:47182
I20260809 15:45:32.050849 22436 raft_consensus.cc:482] T 6b1f47a4af1b4480ab2e69adb8be3cc3 P cab341909d314988b6893daac17235e7 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 15:45:32.051075 22436 raft_consensus.cc:3037] T 6b1f47a4af1b4480ab2e69adb8be3cc3 P cab341909d314988b6893daac17235e7 [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:45:32.056679 22436 raft_consensus.cc:504] T 6b1f47a4af1b4480ab2e69adb8be3cc3 P cab341909d314988b6893daac17235e7 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } } peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } }
I20260809 15:45:32.058235 22436 leader_election.cc:288] T 6b1f47a4af1b4480ab2e69adb8be3cc3 P cab341909d314988b6893daac17235e7 [CANDIDATE]: Term 4 election: Requested vote from peers 8586928e65c543c2b99cebf1dd7d700f (127.20.202.66:45399), 27fddeff55ea476ebe7bd4881ae32035 (127.20.202.67:33583)
I20260809 15:45:32.059074 22556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b1f47a4af1b4480ab2e69adb8be3cc3" candidate_uuid: "cab341909d314988b6893daac17235e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "8586928e65c543c2b99cebf1dd7d700f"
I20260809 15:45:32.059049 22690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b1f47a4af1b4480ab2e69adb8be3cc3" candidate_uuid: "cab341909d314988b6893daac17235e7" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "27fddeff55ea476ebe7bd4881ae32035"
I20260809 15:45:32.059674 22556 raft_consensus.cc:3037] T 6b1f47a4af1b4480ab2e69adb8be3cc3 P 8586928e65c543c2b99cebf1dd7d700f [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:45:32.059732 22690 raft_consensus.cc:3032] T 6b1f47a4af1b4480ab2e69adb8be3cc3 P 27fddeff55ea476ebe7bd4881ae32035 [term 3 LEADER]: Stepping down as leader of term 3
I20260809 15:45:32.060076 22690 raft_consensus.cc:729] T 6b1f47a4af1b4480ab2e69adb8be3cc3 P 27fddeff55ea476ebe7bd4881ae32035 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 27fddeff55ea476ebe7bd4881ae32035, State: Running, Role: LEADER
I20260809 15:45:32.060616 22690 consensus_queue.cc:260] T 6b1f47a4af1b4480ab2e69adb8be3cc3 P 27fddeff55ea476ebe7bd4881ae32035 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } } peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } }
I20260809 15:45:32.061362 22690 raft_consensus.cc:3037] T 6b1f47a4af1b4480ab2e69adb8be3cc3 P 27fddeff55ea476ebe7bd4881ae32035 [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:45:32.063225 22556 raft_consensus.cc:2454] T 6b1f47a4af1b4480ab2e69adb8be3cc3 P 8586928e65c543c2b99cebf1dd7d700f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab341909d314988b6893daac17235e7 in term 4.
I20260809 15:45:32.064006 22361 leader_election.cc:302] T 6b1f47a4af1b4480ab2e69adb8be3cc3 P cab341909d314988b6893daac17235e7 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8586928e65c543c2b99cebf1dd7d700f, cab341909d314988b6893daac17235e7; no voters:
I20260809 15:45:32.064824 22950 raft_consensus.cc:2781] T 6b1f47a4af1b4480ab2e69adb8be3cc3 P cab341909d314988b6893daac17235e7 [term 4 FOLLOWER]: Leader election won for term 4
I20260809 15:45:32.065239 22950 raft_consensus.cc:686] T 6b1f47a4af1b4480ab2e69adb8be3cc3 P cab341909d314988b6893daac17235e7 [term 4 LEADER]: Becoming Leader. State: Replica: cab341909d314988b6893daac17235e7, State: Running, Role: LEADER
I20260809 15:45:32.065428 22690 raft_consensus.cc:2454] T 6b1f47a4af1b4480ab2e69adb8be3cc3 P 27fddeff55ea476ebe7bd4881ae32035 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab341909d314988b6893daac17235e7 in term 4.
I20260809 15:45:32.066035 22950 consensus_queue.cc:237] T 6b1f47a4af1b4480ab2e69adb8be3cc3 P cab341909d314988b6893daac17235e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } } peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } }
I20260809 15:45:32.072893 22242 catalog_manager.cc:5495] T 6b1f47a4af1b4480ab2e69adb8be3cc3 P cab341909d314988b6893daac17235e7 reported cstate change: term changed from 3 to 4, leader changed from 27fddeff55ea476ebe7bd4881ae32035 (127.20.202.67) to cab341909d314988b6893daac17235e7 (127.20.202.65). New cstate: current_term: 4 leader_uuid: "cab341909d314988b6893daac17235e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } health_report { overall_health: HEALTHY } } }
I20260809 15:45:32.141103 22961 raft_consensus.cc:979] T ab0d7bb4b86540b28101d1c813ee214e P 8586928e65c543c2b99cebf1dd7d700f: : Instructing follower cab341909d314988b6893daac17235e7 to start an election
I20260809 15:45:32.141484 22958 raft_consensus.cc:1067] T ab0d7bb4b86540b28101d1c813ee214e P 8586928e65c543c2b99cebf1dd7d700f [term 1 LEADER]: Signalling peer cab341909d314988b6893daac17235e7 to start an election
I20260809 15:45:32.142617 22436 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ab0d7bb4b86540b28101d1c813ee214e"
dest_uuid: "cab341909d314988b6893daac17235e7"
from {username='slave'} at 127.0.0.1:47180
I20260809 15:45:32.142982 22436 raft_consensus.cc:482] T ab0d7bb4b86540b28101d1c813ee214e P cab341909d314988b6893daac17235e7 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 15:45:32.143187 22436 raft_consensus.cc:3037] T ab0d7bb4b86540b28101d1c813ee214e P cab341909d314988b6893daac17235e7 [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:45:32.146889 22436 raft_consensus.cc:504] T ab0d7bb4b86540b28101d1c813ee214e P cab341909d314988b6893daac17235e7 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } } peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } }
I20260809 15:45:32.148131 22436 leader_election.cc:288] T ab0d7bb4b86540b28101d1c813ee214e P cab341909d314988b6893daac17235e7 [CANDIDATE]: Term 2 election: Requested vote from peers 8586928e65c543c2b99cebf1dd7d700f (127.20.202.66:45399), 27fddeff55ea476ebe7bd4881ae32035 (127.20.202.67:33583)
I20260809 15:45:32.148797 22556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab0d7bb4b86540b28101d1c813ee214e" candidate_uuid: "cab341909d314988b6893daac17235e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8586928e65c543c2b99cebf1dd7d700f"
I20260809 15:45:32.149184 22556 raft_consensus.cc:3032] T ab0d7bb4b86540b28101d1c813ee214e P 8586928e65c543c2b99cebf1dd7d700f [term 1 LEADER]: Stepping down as leader of term 1
I20260809 15:45:32.149089 22690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab0d7bb4b86540b28101d1c813ee214e" candidate_uuid: "cab341909d314988b6893daac17235e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "27fddeff55ea476ebe7bd4881ae32035"
I20260809 15:45:32.149396 22556 raft_consensus.cc:729] T ab0d7bb4b86540b28101d1c813ee214e P 8586928e65c543c2b99cebf1dd7d700f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8586928e65c543c2b99cebf1dd7d700f, State: Running, Role: LEADER
I20260809 15:45:32.149554 22690 raft_consensus.cc:3037] T ab0d7bb4b86540b28101d1c813ee214e P 27fddeff55ea476ebe7bd4881ae32035 [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:45:32.150008 22556 consensus_queue.cc:260] T ab0d7bb4b86540b28101d1c813ee214e P 8586928e65c543c2b99cebf1dd7d700f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } } peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } }
I20260809 15:45:32.150826 22556 raft_consensus.cc:3037] T ab0d7bb4b86540b28101d1c813ee214e P 8586928e65c543c2b99cebf1dd7d700f [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:45:32.153307 22690 raft_consensus.cc:2454] T ab0d7bb4b86540b28101d1c813ee214e P 27fddeff55ea476ebe7bd4881ae32035 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab341909d314988b6893daac17235e7 in term 2.
I20260809 15:45:32.153983 22360 leader_election.cc:302] T ab0d7bb4b86540b28101d1c813ee214e P cab341909d314988b6893daac17235e7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27fddeff55ea476ebe7bd4881ae32035, cab341909d314988b6893daac17235e7; no voters:
I20260809 15:45:32.154510 22950 raft_consensus.cc:2781] T ab0d7bb4b86540b28101d1c813ee214e P cab341909d314988b6893daac17235e7 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 15:45:32.154677 22556 raft_consensus.cc:2454] T ab0d7bb4b86540b28101d1c813ee214e P 8586928e65c543c2b99cebf1dd7d700f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab341909d314988b6893daac17235e7 in term 2.
I20260809 15:45:32.154867 22950 raft_consensus.cc:686] T ab0d7bb4b86540b28101d1c813ee214e P cab341909d314988b6893daac17235e7 [term 2 LEADER]: Becoming Leader. State: Replica: cab341909d314988b6893daac17235e7, State: Running, Role: LEADER
I20260809 15:45:32.155683 22950 consensus_queue.cc:237] T ab0d7bb4b86540b28101d1c813ee214e P cab341909d314988b6893daac17235e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } } peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } }
I20260809 15:45:32.161239 22242 catalog_manager.cc:5495] T ab0d7bb4b86540b28101d1c813ee214e P cab341909d314988b6893daac17235e7 reported cstate change: term changed from 1 to 2, leader changed from 8586928e65c543c2b99cebf1dd7d700f (127.20.202.66) to cab341909d314988b6893daac17235e7 (127.20.202.65). New cstate: current_term: 2 leader_uuid: "cab341909d314988b6893daac17235e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } health_report { overall_health: UNKNOWN } } }
I20260809 15:45:32.211304 22556 raft_consensus.cc:1261] T 6d6e21b2d7b746469c42c6d80ad8e001 P 8586928e65c543c2b99cebf1dd7d700f [term 2 FOLLOWER]: Refusing update from remote peer cab341909d314988b6893daac17235e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 15:45:32.212302 22950 consensus_queue.cc:1035] T 6d6e21b2d7b746469c42c6d80ad8e001 P cab341909d314988b6893daac17235e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 15:45:32.222011 22690 raft_consensus.cc:1261] T 6d6e21b2d7b746469c42c6d80ad8e001 P 27fddeff55ea476ebe7bd4881ae32035 [term 2 FOLLOWER]: Refusing update from remote peer cab341909d314988b6893daac17235e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 15:45:32.223333 22963 consensus_queue.cc:1035] T 6d6e21b2d7b746469c42c6d80ad8e001 P cab341909d314988b6893daac17235e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 15:45:32.518841 22556 raft_consensus.cc:1261] T 21aa1094e1cf4d40b51e5916cb6a2a96 P 8586928e65c543c2b99cebf1dd7d700f [term 2 FOLLOWER]: Refusing update from remote peer cab341909d314988b6893daac17235e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 15:45:32.519855 22963 consensus_queue.cc:1035] T 21aa1094e1cf4d40b51e5916cb6a2a96 P cab341909d314988b6893daac17235e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 15:45:32.528085 22690 raft_consensus.cc:1261] T 21aa1094e1cf4d40b51e5916cb6a2a96 P 27fddeff55ea476ebe7bd4881ae32035 [term 2 FOLLOWER]: Refusing update from remote peer cab341909d314988b6893daac17235e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 15:45:32.529155 22990 consensus_queue.cc:1035] T 21aa1094e1cf4d40b51e5916cb6a2a96 P cab341909d314988b6893daac17235e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 15:45:32.545917 22690 raft_consensus.cc:1261] T 6b1f47a4af1b4480ab2e69adb8be3cc3 P 27fddeff55ea476ebe7bd4881ae32035 [term 4 FOLLOWER]: Refusing update from remote peer cab341909d314988b6893daac17235e7: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260809 15:45:32.546842 22990 consensus_queue.cc:1035] T 6b1f47a4af1b4480ab2e69adb8be3cc3 P cab341909d314988b6893daac17235e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260809 15:45:32.555387 22556 raft_consensus.cc:1261] T 6b1f47a4af1b4480ab2e69adb8be3cc3 P 8586928e65c543c2b99cebf1dd7d700f [term 4 FOLLOWER]: Refusing update from remote peer cab341909d314988b6893daac17235e7: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260809 15:45:32.556411 22950 consensus_queue.cc:1035] T 6b1f47a4af1b4480ab2e69adb8be3cc3 P cab341909d314988b6893daac17235e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260809 15:45:32.596750 22556 raft_consensus.cc:1261] T 4e0d833302ff456e96ab09aa12d4c5ba P 8586928e65c543c2b99cebf1dd7d700f [term 4 FOLLOWER]: Refusing update from remote peer cab341909d314988b6893daac17235e7: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260809 15:45:32.597626 22950 consensus_queue.cc:1035] T 4e0d833302ff456e96ab09aa12d4c5ba P cab341909d314988b6893daac17235e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260809 15:45:32.606920 22690 raft_consensus.cc:1261] T 4e0d833302ff456e96ab09aa12d4c5ba P 27fddeff55ea476ebe7bd4881ae32035 [term 4 FOLLOWER]: Refusing update from remote peer cab341909d314988b6893daac17235e7: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260809 15:45:32.608038 22999 consensus_queue.cc:1035] T 4e0d833302ff456e96ab09aa12d4c5ba P cab341909d314988b6893daac17235e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260809 15:45:32.753293 22690 raft_consensus.cc:1261] T ab0d7bb4b86540b28101d1c813ee214e P 27fddeff55ea476ebe7bd4881ae32035 [term 2 FOLLOWER]: Refusing update from remote peer cab341909d314988b6893daac17235e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 15:45:32.754333 22999 consensus_queue.cc:1035] T ab0d7bb4b86540b28101d1c813ee214e P cab341909d314988b6893daac17235e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 15:45:32.758913 22556 raft_consensus.cc:1261] T ab0d7bb4b86540b28101d1c813ee214e P 8586928e65c543c2b99cebf1dd7d700f [term 2 FOLLOWER]: Refusing update from remote peer cab341909d314988b6893daac17235e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 15:45:32.760303 22990 consensus_queue.cc:1035] T ab0d7bb4b86540b28101d1c813ee214e P cab341909d314988b6893daac17235e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 15:45:34.710289 21289 auto_leader_rebalancer-test.cc:373] The leader distribution is
I20260809 15:45:34.710893 21289 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 15:45:34.717224 22436 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1154606dbfd849c7833c096541e8c58f"
dest_uuid: "cab341909d314988b6893daac17235e7"
mode: GRACEFUL
new_leader_uuid: "27fddeff55ea476ebe7bd4881ae32035"
from {username='slave'} at 127.0.0.1:40564
I20260809 15:45:34.717725 22436 raft_consensus.cc:595] T 1154606dbfd849c7833c096541e8c58f P cab341909d314988b6893daac17235e7 [term 1 LEADER]: Received request to transfer leadership to TS 27fddeff55ea476ebe7bd4881ae32035
I20260809 15:45:34.719759 22436 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "21aa1094e1cf4d40b51e5916cb6a2a96"
dest_uuid: "cab341909d314988b6893daac17235e7"
mode: GRACEFUL
new_leader_uuid: "27fddeff55ea476ebe7bd4881ae32035"
from {username='slave'} at 127.0.0.1:40564
I20260809 15:45:34.720146 22436 raft_consensus.cc:595] T 21aa1094e1cf4d40b51e5916cb6a2a96 P cab341909d314988b6893daac17235e7 [term 2 LEADER]: Received request to transfer leadership to TS 27fddeff55ea476ebe7bd4881ae32035
I20260809 15:45:34.721817 22436 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4e0d833302ff456e96ab09aa12d4c5ba"
dest_uuid: "cab341909d314988b6893daac17235e7"
mode: GRACEFUL
new_leader_uuid: "8586928e65c543c2b99cebf1dd7d700f"
from {username='slave'} at 127.0.0.1:40564
I20260809 15:45:34.722182 22436 raft_consensus.cc:595] T 4e0d833302ff456e96ab09aa12d4c5ba P cab341909d314988b6893daac17235e7 [term 4 LEADER]: Received request to transfer leadership to TS 8586928e65c543c2b99cebf1dd7d700f
I20260809 15:45:34.723855 22436 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6d6e21b2d7b746469c42c6d80ad8e001"
dest_uuid: "cab341909d314988b6893daac17235e7"
mode: GRACEFUL
new_leader_uuid: "8586928e65c543c2b99cebf1dd7d700f"
from {username='slave'} at 127.0.0.1:40564
I20260809 15:45:34.724205 22436 raft_consensus.cc:595] T 6d6e21b2d7b746469c42c6d80ad8e001 P cab341909d314988b6893daac17235e7 [term 2 LEADER]: Received request to transfer leadership to TS 8586928e65c543c2b99cebf1dd7d700f
I20260809 15:45:34.724962 21289 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 4
I20260809 15:45:34.725311 21289 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 15:45:34.872498 22999 raft_consensus.cc:979] T 1154606dbfd849c7833c096541e8c58f P cab341909d314988b6893daac17235e7: : Instructing follower 27fddeff55ea476ebe7bd4881ae32035 to start an election
I20260809 15:45:34.872876 23006 raft_consensus.cc:1067] T 1154606dbfd849c7833c096541e8c58f P cab341909d314988b6893daac17235e7 [term 1 LEADER]: Signalling peer 27fddeff55ea476ebe7bd4881ae32035 to start an election
I20260809 15:45:34.874082 22690 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1154606dbfd849c7833c096541e8c58f"
dest_uuid: "27fddeff55ea476ebe7bd4881ae32035"
from {username='slave'} at 127.0.0.1:51580
I20260809 15:45:34.874482 22690 raft_consensus.cc:482] T 1154606dbfd849c7833c096541e8c58f P 27fddeff55ea476ebe7bd4881ae32035 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 15:45:34.874724 22690 raft_consensus.cc:3037] T 1154606dbfd849c7833c096541e8c58f P 27fddeff55ea476ebe7bd4881ae32035 [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:45:34.878624 22690 raft_consensus.cc:504] T 1154606dbfd849c7833c096541e8c58f P 27fddeff55ea476ebe7bd4881ae32035 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } } peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } }
I20260809 15:45:34.880163 22690 leader_election.cc:288] T 1154606dbfd849c7833c096541e8c58f P 27fddeff55ea476ebe7bd4881ae32035 [CANDIDATE]: Term 2 election: Requested vote from peers cab341909d314988b6893daac17235e7 (127.20.202.65:46511), 8586928e65c543c2b99cebf1dd7d700f (127.20.202.66:45399)
I20260809 15:45:34.880883 22436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1154606dbfd849c7833c096541e8c58f" candidate_uuid: "27fddeff55ea476ebe7bd4881ae32035" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "cab341909d314988b6893daac17235e7"
I20260809 15:45:34.880977 22556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1154606dbfd849c7833c096541e8c58f" candidate_uuid: "27fddeff55ea476ebe7bd4881ae32035" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8586928e65c543c2b99cebf1dd7d700f"
I20260809 15:45:34.881364 22436 raft_consensus.cc:3032] T 1154606dbfd849c7833c096541e8c58f P cab341909d314988b6893daac17235e7 [term 1 LEADER]: Stepping down as leader of term 1
I20260809 15:45:34.881505 22556 raft_consensus.cc:3037] T 1154606dbfd849c7833c096541e8c58f P 8586928e65c543c2b99cebf1dd7d700f [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:45:34.881623 22436 raft_consensus.cc:729] T 1154606dbfd849c7833c096541e8c58f P cab341909d314988b6893daac17235e7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cab341909d314988b6893daac17235e7, State: Running, Role: LEADER
I20260809 15:45:34.882220 22436 consensus_queue.cc:260] T 1154606dbfd849c7833c096541e8c58f P cab341909d314988b6893daac17235e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } } peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } }
I20260809 15:45:34.883016 22436 raft_consensus.cc:3037] T 1154606dbfd849c7833c096541e8c58f P cab341909d314988b6893daac17235e7 [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:45:34.885576 22556 raft_consensus.cc:2454] T 1154606dbfd849c7833c096541e8c58f P 8586928e65c543c2b99cebf1dd7d700f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27fddeff55ea476ebe7bd4881ae32035 in term 2.
I20260809 15:45:34.886734 22635 leader_election.cc:302] T 1154606dbfd849c7833c096541e8c58f P 27fddeff55ea476ebe7bd4881ae32035 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27fddeff55ea476ebe7bd4881ae32035, 8586928e65c543c2b99cebf1dd7d700f; no voters:
I20260809 15:45:34.887426 22949 raft_consensus.cc:2781] T 1154606dbfd849c7833c096541e8c58f P 27fddeff55ea476ebe7bd4881ae32035 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 15:45:34.887800 22999 raft_consensus.cc:979] T 4e0d833302ff456e96ab09aa12d4c5ba P cab341909d314988b6893daac17235e7: : Instructing follower 8586928e65c543c2b99cebf1dd7d700f to start an election
I20260809 15:45:34.887825 22436 raft_consensus.cc:2454] T 1154606dbfd849c7833c096541e8c58f P cab341909d314988b6893daac17235e7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27fddeff55ea476ebe7bd4881ae32035 in term 2.
I20260809 15:45:34.888275 22999 raft_consensus.cc:1067] T 4e0d833302ff456e96ab09aa12d4c5ba P cab341909d314988b6893daac17235e7 [term 4 LEADER]: Signalling peer 8586928e65c543c2b99cebf1dd7d700f to start an election
I20260809 15:45:34.888494 22949 raft_consensus.cc:686] T 1154606dbfd849c7833c096541e8c58f P 27fddeff55ea476ebe7bd4881ae32035 [term 2 LEADER]: Becoming Leader. State: Replica: 27fddeff55ea476ebe7bd4881ae32035, State: Running, Role: LEADER
I20260809 15:45:34.889400 22949 consensus_queue.cc:237] T 1154606dbfd849c7833c096541e8c58f P 27fddeff55ea476ebe7bd4881ae32035 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } } peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } }
I20260809 15:45:34.889869 22555 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4e0d833302ff456e96ab09aa12d4c5ba"
dest_uuid: "8586928e65c543c2b99cebf1dd7d700f"
from {username='slave'} at 127.0.0.1:56476
I20260809 15:45:34.890337 22555 raft_consensus.cc:482] T 4e0d833302ff456e96ab09aa12d4c5ba P 8586928e65c543c2b99cebf1dd7d700f [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 15:45:34.890633 22555 raft_consensus.cc:3037] T 4e0d833302ff456e96ab09aa12d4c5ba P 8586928e65c543c2b99cebf1dd7d700f [term 4 FOLLOWER]: Advancing to term 5
I20260809 15:45:34.896205 22555 raft_consensus.cc:504] T 4e0d833302ff456e96ab09aa12d4c5ba P 8586928e65c543c2b99cebf1dd7d700f [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } } peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } }
I20260809 15:45:34.896723 22242 catalog_manager.cc:5495] T 1154606dbfd849c7833c096541e8c58f P 27fddeff55ea476ebe7bd4881ae32035 reported cstate change: term changed from 1 to 2, leader changed from cab341909d314988b6893daac17235e7 (127.20.202.65) to 27fddeff55ea476ebe7bd4881ae32035 (127.20.202.67). New cstate: current_term: 2 leader_uuid: "27fddeff55ea476ebe7bd4881ae32035" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } health_report { overall_health: HEALTHY } } }
I20260809 15:45:34.898906 22555 leader_election.cc:288] T 4e0d833302ff456e96ab09aa12d4c5ba P 8586928e65c543c2b99cebf1dd7d700f [CANDIDATE]: Term 5 election: Requested vote from peers cab341909d314988b6893daac17235e7 (127.20.202.65:46511), 27fddeff55ea476ebe7bd4881ae32035 (127.20.202.67:33583)
I20260809 15:45:34.899005 22436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e0d833302ff456e96ab09aa12d4c5ba" candidate_uuid: "8586928e65c543c2b99cebf1dd7d700f" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "cab341909d314988b6893daac17235e7"
I20260809 15:45:34.899412 22690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e0d833302ff456e96ab09aa12d4c5ba" candidate_uuid: "8586928e65c543c2b99cebf1dd7d700f" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "27fddeff55ea476ebe7bd4881ae32035"
I20260809 15:45:34.899531 22436 raft_consensus.cc:3032] T 4e0d833302ff456e96ab09aa12d4c5ba P cab341909d314988b6893daac17235e7 [term 4 LEADER]: Stepping down as leader of term 4
I20260809 15:45:34.899884 22690 raft_consensus.cc:3037] T 4e0d833302ff456e96ab09aa12d4c5ba P 27fddeff55ea476ebe7bd4881ae32035 [term 4 FOLLOWER]: Advancing to term 5
I20260809 15:45:34.899919 22436 raft_consensus.cc:729] T 4e0d833302ff456e96ab09aa12d4c5ba P cab341909d314988b6893daac17235e7 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: cab341909d314988b6893daac17235e7, State: Running, Role: LEADER
I20260809 15:45:34.900651 22436 consensus_queue.cc:260] T 4e0d833302ff456e96ab09aa12d4c5ba P cab341909d314988b6893daac17235e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } } peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } }
I20260809 15:45:34.901711 22436 raft_consensus.cc:3037] T 4e0d833302ff456e96ab09aa12d4c5ba P cab341909d314988b6893daac17235e7 [term 4 FOLLOWER]: Advancing to term 5
I20260809 15:45:34.905440 22690 raft_consensus.cc:2454] T 4e0d833302ff456e96ab09aa12d4c5ba P 27fddeff55ea476ebe7bd4881ae32035 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8586928e65c543c2b99cebf1dd7d700f in term 5.
I20260809 15:45:34.906664 22493 leader_election.cc:302] T 4e0d833302ff456e96ab09aa12d4c5ba P 8586928e65c543c2b99cebf1dd7d700f [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27fddeff55ea476ebe7bd4881ae32035, 8586928e65c543c2b99cebf1dd7d700f; no voters:
I20260809 15:45:34.907353 23005 raft_consensus.cc:2781] T 4e0d833302ff456e96ab09aa12d4c5ba P 8586928e65c543c2b99cebf1dd7d700f [term 5 FOLLOWER]: Leader election won for term 5
I20260809 15:45:34.907622 22436 raft_consensus.cc:2454] T 4e0d833302ff456e96ab09aa12d4c5ba P cab341909d314988b6893daac17235e7 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8586928e65c543c2b99cebf1dd7d700f in term 5.
I20260809 15:45:34.908476 23005 raft_consensus.cc:686] T 4e0d833302ff456e96ab09aa12d4c5ba P 8586928e65c543c2b99cebf1dd7d700f [term 5 LEADER]: Becoming Leader. State: Replica: 8586928e65c543c2b99cebf1dd7d700f, State: Running, Role: LEADER
I20260809 15:45:34.909309 23005 consensus_queue.cc:237] T 4e0d833302ff456e96ab09aa12d4c5ba P 8586928e65c543c2b99cebf1dd7d700f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } } peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } }
I20260809 15:45:34.916090 22242 catalog_manager.cc:5495] T 4e0d833302ff456e96ab09aa12d4c5ba P 8586928e65c543c2b99cebf1dd7d700f reported cstate change: term changed from 4 to 5, leader changed from cab341909d314988b6893daac17235e7 (127.20.202.65) to 8586928e65c543c2b99cebf1dd7d700f (127.20.202.66). New cstate: current_term: 5 leader_uuid: "8586928e65c543c2b99cebf1dd7d700f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } health_report { overall_health: UNKNOWN } } }
I20260809 15:45:35.018977 23006 raft_consensus.cc:979] T 6d6e21b2d7b746469c42c6d80ad8e001 P cab341909d314988b6893daac17235e7: : Instructing follower 8586928e65c543c2b99cebf1dd7d700f to start an election
I20260809 15:45:35.019397 22999 raft_consensus.cc:1067] T 6d6e21b2d7b746469c42c6d80ad8e001 P cab341909d314988b6893daac17235e7 [term 2 LEADER]: Signalling peer 8586928e65c543c2b99cebf1dd7d700f to start an election
I20260809 15:45:35.020563 22555 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6d6e21b2d7b746469c42c6d80ad8e001"
dest_uuid: "8586928e65c543c2b99cebf1dd7d700f"
from {username='slave'} at 127.0.0.1:56476
I20260809 15:45:35.020993 22555 raft_consensus.cc:482] T 6d6e21b2d7b746469c42c6d80ad8e001 P 8586928e65c543c2b99cebf1dd7d700f [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 15:45:35.021237 22555 raft_consensus.cc:3037] T 6d6e21b2d7b746469c42c6d80ad8e001 P 8586928e65c543c2b99cebf1dd7d700f [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:45:35.025367 22555 raft_consensus.cc:504] T 6d6e21b2d7b746469c42c6d80ad8e001 P 8586928e65c543c2b99cebf1dd7d700f [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } } peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } }
I20260809 15:45:35.026758 22555 leader_election.cc:288] T 6d6e21b2d7b746469c42c6d80ad8e001 P 8586928e65c543c2b99cebf1dd7d700f [CANDIDATE]: Term 3 election: Requested vote from peers cab341909d314988b6893daac17235e7 (127.20.202.65:46511), 27fddeff55ea476ebe7bd4881ae32035 (127.20.202.67:33583)
I20260809 15:45:35.027335 22436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d6e21b2d7b746469c42c6d80ad8e001" candidate_uuid: "8586928e65c543c2b99cebf1dd7d700f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "cab341909d314988b6893daac17235e7"
I20260809 15:45:35.027652 22690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d6e21b2d7b746469c42c6d80ad8e001" candidate_uuid: "8586928e65c543c2b99cebf1dd7d700f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "27fddeff55ea476ebe7bd4881ae32035"
I20260809 15:45:35.027776 22436 raft_consensus.cc:3032] T 6d6e21b2d7b746469c42c6d80ad8e001 P cab341909d314988b6893daac17235e7 [term 2 LEADER]: Stepping down as leader of term 2
I20260809 15:45:35.028066 22436 raft_consensus.cc:729] T 6d6e21b2d7b746469c42c6d80ad8e001 P cab341909d314988b6893daac17235e7 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: cab341909d314988b6893daac17235e7, State: Running, Role: LEADER
I20260809 15:45:35.028084 22690 raft_consensus.cc:3037] T 6d6e21b2d7b746469c42c6d80ad8e001 P 27fddeff55ea476ebe7bd4881ae32035 [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:45:35.028734 22436 consensus_queue.cc:260] T 6d6e21b2d7b746469c42c6d80ad8e001 P cab341909d314988b6893daac17235e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } } peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } }
I20260809 15:45:35.029516 22436 raft_consensus.cc:3037] T 6d6e21b2d7b746469c42c6d80ad8e001 P cab341909d314988b6893daac17235e7 [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:45:35.031819 22690 raft_consensus.cc:2454] T 6d6e21b2d7b746469c42c6d80ad8e001 P 27fddeff55ea476ebe7bd4881ae32035 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8586928e65c543c2b99cebf1dd7d700f in term 3.
I20260809 15:45:35.032519 22493 leader_election.cc:302] T 6d6e21b2d7b746469c42c6d80ad8e001 P 8586928e65c543c2b99cebf1dd7d700f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27fddeff55ea476ebe7bd4881ae32035, 8586928e65c543c2b99cebf1dd7d700f; no voters:
I20260809 15:45:35.033082 23005 raft_consensus.cc:2781] T 6d6e21b2d7b746469c42c6d80ad8e001 P 8586928e65c543c2b99cebf1dd7d700f [term 3 FOLLOWER]: Leader election won for term 3
I20260809 15:45:35.033275 22436 raft_consensus.cc:2454] T 6d6e21b2d7b746469c42c6d80ad8e001 P cab341909d314988b6893daac17235e7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8586928e65c543c2b99cebf1dd7d700f in term 3.
I20260809 15:45:35.033408 23005 raft_consensus.cc:686] T 6d6e21b2d7b746469c42c6d80ad8e001 P 8586928e65c543c2b99cebf1dd7d700f [term 3 LEADER]: Becoming Leader. State: Replica: 8586928e65c543c2b99cebf1dd7d700f, State: Running, Role: LEADER
I20260809 15:45:35.034129 23005 consensus_queue.cc:237] T 6d6e21b2d7b746469c42c6d80ad8e001 P 8586928e65c543c2b99cebf1dd7d700f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } } peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } }
I20260809 15:45:35.040026 22242 catalog_manager.cc:5495] T 6d6e21b2d7b746469c42c6d80ad8e001 P 8586928e65c543c2b99cebf1dd7d700f reported cstate change: term changed from 2 to 3, leader changed from cab341909d314988b6893daac17235e7 (127.20.202.65) to 8586928e65c543c2b99cebf1dd7d700f (127.20.202.66). New cstate: current_term: 3 leader_uuid: "8586928e65c543c2b99cebf1dd7d700f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } health_report { overall_health: UNKNOWN } } }
I20260809 15:45:35.064816 23006 raft_consensus.cc:979] T 21aa1094e1cf4d40b51e5916cb6a2a96 P cab341909d314988b6893daac17235e7: : Instructing follower 27fddeff55ea476ebe7bd4881ae32035 to start an election
I20260809 15:45:35.065140 22999 raft_consensus.cc:1067] T 21aa1094e1cf4d40b51e5916cb6a2a96 P cab341909d314988b6893daac17235e7 [term 2 LEADER]: Signalling peer 27fddeff55ea476ebe7bd4881ae32035 to start an election
I20260809 15:45:35.066187 22690 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "21aa1094e1cf4d40b51e5916cb6a2a96"
dest_uuid: "27fddeff55ea476ebe7bd4881ae32035"
from {username='slave'} at 127.0.0.1:51580
I20260809 15:45:35.066530 22690 raft_consensus.cc:482] T 21aa1094e1cf4d40b51e5916cb6a2a96 P 27fddeff55ea476ebe7bd4881ae32035 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 15:45:35.066752 22690 raft_consensus.cc:3037] T 21aa1094e1cf4d40b51e5916cb6a2a96 P 27fddeff55ea476ebe7bd4881ae32035 [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:45:35.070092 22690 raft_consensus.cc:504] T 21aa1094e1cf4d40b51e5916cb6a2a96 P 27fddeff55ea476ebe7bd4881ae32035 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } } peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } }
I20260809 15:45:35.071308 22690 leader_election.cc:288] T 21aa1094e1cf4d40b51e5916cb6a2a96 P 27fddeff55ea476ebe7bd4881ae32035 [CANDIDATE]: Term 3 election: Requested vote from peers cab341909d314988b6893daac17235e7 (127.20.202.65:46511), 8586928e65c543c2b99cebf1dd7d700f (127.20.202.66:45399)
I20260809 15:45:35.071873 22436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21aa1094e1cf4d40b51e5916cb6a2a96" candidate_uuid: "27fddeff55ea476ebe7bd4881ae32035" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "cab341909d314988b6893daac17235e7"
I20260809 15:45:35.072117 22555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21aa1094e1cf4d40b51e5916cb6a2a96" candidate_uuid: "27fddeff55ea476ebe7bd4881ae32035" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "8586928e65c543c2b99cebf1dd7d700f"
I20260809 15:45:35.072377 22436 raft_consensus.cc:3032] T 21aa1094e1cf4d40b51e5916cb6a2a96 P cab341909d314988b6893daac17235e7 [term 2 LEADER]: Stepping down as leader of term 2
I20260809 15:45:35.072623 22555 raft_consensus.cc:3037] T 21aa1094e1cf4d40b51e5916cb6a2a96 P 8586928e65c543c2b99cebf1dd7d700f [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:45:35.072656 22436 raft_consensus.cc:729] T 21aa1094e1cf4d40b51e5916cb6a2a96 P cab341909d314988b6893daac17235e7 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: cab341909d314988b6893daac17235e7, State: Running, Role: LEADER
I20260809 15:45:35.074157 22436 consensus_queue.cc:260] T 21aa1094e1cf4d40b51e5916cb6a2a96 P cab341909d314988b6893daac17235e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } } peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } }
I20260809 15:45:35.075217 22436 raft_consensus.cc:3037] T 21aa1094e1cf4d40b51e5916cb6a2a96 P cab341909d314988b6893daac17235e7 [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:45:35.077515 22555 raft_consensus.cc:2454] T 21aa1094e1cf4d40b51e5916cb6a2a96 P 8586928e65c543c2b99cebf1dd7d700f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27fddeff55ea476ebe7bd4881ae32035 in term 3.
I20260809 15:45:35.078261 22635 leader_election.cc:302] T 21aa1094e1cf4d40b51e5916cb6a2a96 P 27fddeff55ea476ebe7bd4881ae32035 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27fddeff55ea476ebe7bd4881ae32035, 8586928e65c543c2b99cebf1dd7d700f; no voters:
I20260809 15:45:35.078789 22949 raft_consensus.cc:2781] T 21aa1094e1cf4d40b51e5916cb6a2a96 P 27fddeff55ea476ebe7bd4881ae32035 [term 3 FOLLOWER]: Leader election won for term 3
I20260809 15:45:35.079303 22949 raft_consensus.cc:686] T 21aa1094e1cf4d40b51e5916cb6a2a96 P 27fddeff55ea476ebe7bd4881ae32035 [term 3 LEADER]: Becoming Leader. State: Replica: 27fddeff55ea476ebe7bd4881ae32035, State: Running, Role: LEADER
I20260809 15:45:35.080039 22949 consensus_queue.cc:237] T 21aa1094e1cf4d40b51e5916cb6a2a96 P 27fddeff55ea476ebe7bd4881ae32035 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } } peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } }
I20260809 15:45:35.080646 22436 raft_consensus.cc:2454] T 21aa1094e1cf4d40b51e5916cb6a2a96 P cab341909d314988b6893daac17235e7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27fddeff55ea476ebe7bd4881ae32035 in term 3.
I20260809 15:45:35.086125 22242 catalog_manager.cc:5495] T 21aa1094e1cf4d40b51e5916cb6a2a96 P 27fddeff55ea476ebe7bd4881ae32035 reported cstate change: term changed from 2 to 3, leader changed from cab341909d314988b6893daac17235e7 (127.20.202.65) to 27fddeff55ea476ebe7bd4881ae32035 (127.20.202.67). New cstate: current_term: 3 leader_uuid: "27fddeff55ea476ebe7bd4881ae32035" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 } health_report { overall_health: HEALTHY } } }
I20260809 15:45:35.269447 22555 raft_consensus.cc:1261] T 1154606dbfd849c7833c096541e8c58f P 8586928e65c543c2b99cebf1dd7d700f [term 2 FOLLOWER]: Refusing update from remote peer 27fddeff55ea476ebe7bd4881ae32035: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 15:45:35.270370 22949 consensus_queue.cc:1035] T 1154606dbfd849c7833c096541e8c58f P 27fddeff55ea476ebe7bd4881ae32035 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 15:45:35.279381 22436 raft_consensus.cc:1261] T 1154606dbfd849c7833c096541e8c58f P cab341909d314988b6893daac17235e7 [term 2 FOLLOWER]: Refusing update from remote peer 27fddeff55ea476ebe7bd4881ae32035: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 15:45:35.280601 22949 consensus_queue.cc:1035] T 1154606dbfd849c7833c096541e8c58f P 27fddeff55ea476ebe7bd4881ae32035 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 15:45:35.304579 22690 raft_consensus.cc:1261] T 4e0d833302ff456e96ab09aa12d4c5ba P 27fddeff55ea476ebe7bd4881ae32035 [term 5 FOLLOWER]: Refusing update from remote peer 8586928e65c543c2b99cebf1dd7d700f: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260809 15:45:35.305483 23005 consensus_queue.cc:1035] T 4e0d833302ff456e96ab09aa12d4c5ba P 8586928e65c543c2b99cebf1dd7d700f [LEADER]: Connected to new peer: Peer: permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260809 15:45:35.312366 22436 raft_consensus.cc:1261] T 4e0d833302ff456e96ab09aa12d4c5ba P cab341909d314988b6893daac17235e7 [term 5 FOLLOWER]: Refusing update from remote peer 8586928e65c543c2b99cebf1dd7d700f: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260809 15:45:35.313807 23029 consensus_queue.cc:1035] T 4e0d833302ff456e96ab09aa12d4c5ba P 8586928e65c543c2b99cebf1dd7d700f [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260809 15:45:35.517745 22436 raft_consensus.cc:1261] T 21aa1094e1cf4d40b51e5916cb6a2a96 P cab341909d314988b6893daac17235e7 [term 3 FOLLOWER]: Refusing update from remote peer 27fddeff55ea476ebe7bd4881ae32035: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260809 15:45:35.518721 22949 consensus_queue.cc:1035] T 21aa1094e1cf4d40b51e5916cb6a2a96 P 27fddeff55ea476ebe7bd4881ae32035 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 15:45:35.525509 22555 raft_consensus.cc:1261] T 21aa1094e1cf4d40b51e5916cb6a2a96 P 8586928e65c543c2b99cebf1dd7d700f [term 3 FOLLOWER]: Refusing update from remote peer 27fddeff55ea476ebe7bd4881ae32035: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260809 15:45:35.527570 23007 consensus_queue.cc:1035] T 21aa1094e1cf4d40b51e5916cb6a2a96 P 27fddeff55ea476ebe7bd4881ae32035 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8586928e65c543c2b99cebf1dd7d700f" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 45399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 15:45:35.589394 22436 raft_consensus.cc:1261] T 6d6e21b2d7b746469c42c6d80ad8e001 P cab341909d314988b6893daac17235e7 [term 3 FOLLOWER]: Refusing update from remote peer 8586928e65c543c2b99cebf1dd7d700f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260809 15:45:35.590351 23029 consensus_queue.cc:1035] T 6d6e21b2d7b746469c42c6d80ad8e001 P 8586928e65c543c2b99cebf1dd7d700f [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab341909d314988b6893daac17235e7" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 46511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 15:45:35.599346 22690 raft_consensus.cc:1261] T 6d6e21b2d7b746469c42c6d80ad8e001 P 27fddeff55ea476ebe7bd4881ae32035 [term 3 FOLLOWER]: Refusing update from remote peer 8586928e65c543c2b99cebf1dd7d700f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260809 15:45:35.600834 23029 consensus_queue.cc:1035] T 6d6e21b2d7b746469c42c6d80ad8e001 P 8586928e65c543c2b99cebf1dd7d700f [LEADER]: Connected to new peer: Peer: permanent_uuid: "27fddeff55ea476ebe7bd4881ae32035" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 33583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 15:45:35.725937 21289 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 15:45:35.730716 21289 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260809 15:45:35.731021 21289 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 15:45:36.731861 21289 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 15:45:36.738866 21289 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260809 15:45:36.739233 21289 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 15:45:37.739920 21289 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 15:45:37.744740 21289 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260809 15:45:37.745093 21289 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 15:45:38.745771 21289 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 15:45:38.750540 21289 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260809 15:45:38.750895 21289 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 15:45:39.751587 21289 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 15:45:39.757025 21289 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260809 15:45:39.757377 21289 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 15:45:40.758028 21289 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 15:45:40.763039 21289 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260809 15:45:40.763415 21289 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 15:45:41.764097 21289 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 15:45:41.768980 21289 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260809 15:45:41.769332 21289 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 15:45:42.769996 21289 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 15:45:42.775163 21289 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260809 15:45:42.775525 21289 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 15:45:43.776307 21289 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 15:45:43.781116 21289 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260809 15:45:43.781459 21289 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 15:45:44.806782 21289 auto_leader_rebalancer-test.cc:384] The leader distribution is
I20260809 15:45:44.807288 21289 tablet_server.cc:178] TabletServer@127.20.202.65:0 shutting down...
W20260809 15:45:44.820835 22633 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 6 similar messages]
W20260809 15:45:44.831393 22633 consensus_peers.cc:487] T 36cb01872dee4485b764d163e0bf4e20 P 27fddeff55ea476ebe7bd4881ae32035 -> Peer cab341909d314988b6893daac17235e7 (127.20.202.65:46511): Couldn't send request to peer cab341909d314988b6893daac17235e7. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260809 15:45:44.842254 22633 consensus_peers.cc:487] T 21aa1094e1cf4d40b51e5916cb6a2a96 P 27fddeff55ea476ebe7bd4881ae32035 -> Peer cab341909d314988b6893daac17235e7 (127.20.202.65:46511): Couldn't send request to peer cab341909d314988b6893daac17235e7. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260809 15:45:44.843166 21289 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260809 15:45:44.843772 21289 tablet_replica.cc:330] T ab0d7bb4b86540b28101d1c813ee214e P cab341909d314988b6893daac17235e7: stopping tablet replica
I20260809 15:45:44.844291 21289 raft_consensus.cc:2229] T ab0d7bb4b86540b28101d1c813ee214e P cab341909d314988b6893daac17235e7 [term 2 LEADER]: Raft consensus shutting down.
I20260809 15:45:44.845057 21289 raft_consensus.cc:2258] T ab0d7bb4b86540b28101d1c813ee214e P cab341909d314988b6893daac17235e7 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260809 15:45:44.846709 21289 tablet_replica.cc:330] T 21aa1094e1cf4d40b51e5916cb6a2a96 P cab341909d314988b6893daac17235e7: stopping tablet replica
I20260809 15:45:44.847182 21289 raft_consensus.cc:2229] T 21aa1094e1cf4d40b51e5916cb6a2a96 P cab341909d314988b6893daac17235e7 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260809 15:45:44.847610 21289 raft_consensus.cc:2258] T 21aa1094e1cf4d40b51e5916cb6a2a96 P cab341909d314988b6893daac17235e7 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260809 15:45:44.849021 21289 tablet_replica.cc:330] T 36cb01872dee4485b764d163e0bf4e20 P cab341909d314988b6893daac17235e7: stopping tablet replica
I20260809 15:45:44.849450 21289 raft_consensus.cc:2229] T 36cb01872dee4485b764d163e0bf4e20 P cab341909d314988b6893daac17235e7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 15:45:44.849759 21289 raft_consensus.cc:2258] T 36cb01872dee4485b764d163e0bf4e20 P cab341909d314988b6893daac17235e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 15:45:44.851336 21289 tablet_replica.cc:330] T 4e0d833302ff456e96ab09aa12d4c5ba P cab341909d314988b6893daac17235e7: stopping tablet replica
I20260809 15:45:44.851778 21289 raft_consensus.cc:2229] T 4e0d833302ff456e96ab09aa12d4c5ba P cab341909d314988b6893daac17235e7 [term 5 FOLLOWER]: Raft consensus shutting down.
I20260809 15:45:44.852155 21289 raft_consensus.cc:2258] T 4e0d833302ff456e96ab09aa12d4c5ba P cab341909d314988b6893daac17235e7 [term 5 FOLLOWER]: Raft consensus is shut down!
I20260809 15:45:44.853569 21289 tablet_replica.cc:330] T 6eb2494806b34a2bad3215770bcb4c7c P cab341909d314988b6893daac17235e7: stopping tablet replica
I20260809 15:45:44.854020 21289 raft_consensus.cc:2229] T 6eb2494806b34a2bad3215770bcb4c7c P cab341909d314988b6893daac17235e7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 15:45:44.854372 21289 raft_consensus.cc:2258] T 6eb2494806b34a2bad3215770bcb4c7c P cab341909d314988b6893daac17235e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 15:45:44.855836 21289 tablet_replica.cc:330] T 6d6e21b2d7b746469c42c6d80ad8e001 P cab341909d314988b6893daac17235e7: stopping tablet replica
I20260809 15:45:44.856262 21289 raft_consensus.cc:2229] T 6d6e21b2d7b746469c42c6d80ad8e001 P cab341909d314988b6893daac17235e7 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260809 15:45:44.856635 21289 raft_consensus.cc:2258] T 6d6e21b2d7b746469c42c6d80ad8e001 P cab341909d314988b6893daac17235e7 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260809 15:45:44.858091 21289 tablet_replica.cc:330] T 6b1f47a4af1b4480ab2e69adb8be3cc3 P cab341909d314988b6893daac17235e7: stopping tablet replica
I20260809 15:45:44.858505 21289 raft_consensus.cc:2229] T 6b1f47a4af1b4480ab2e69adb8be3cc3 P cab341909d314988b6893daac17235e7 [term 4 LEADER]: Raft consensus shutting down.
I20260809 15:45:44.859241 21289 raft_consensus.cc:2258] T 6b1f47a4af1b4480ab2e69adb8be3cc3 P cab341909d314988b6893daac17235e7 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260809 15:45:44.860857 21289 tablet_replica.cc:330] T 9b4926c9270e4754bac26cc4c3da4cf9 P cab341909d314988b6893daac17235e7: stopping tablet replica
I20260809 15:45:44.861253 21289 raft_consensus.cc:2229] T 9b4926c9270e4754bac26cc4c3da4cf9 P cab341909d314988b6893daac17235e7 [term 2 LEADER]: Raft consensus shutting down.
I20260809 15:45:44.861922 21289 raft_consensus.cc:2258] T 9b4926c9270e4754bac26cc4c3da4cf9 P cab341909d314988b6893daac17235e7 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260809 15:45:44.863435 21289 tablet_replica.cc:330] T 8c8c4f25c5364a919da21f5fca365c45 P cab341909d314988b6893daac17235e7: stopping tablet replica
I20260809 15:45:44.863838 21289 raft_consensus.cc:2229] T 8c8c4f25c5364a919da21f5fca365c45 P cab341909d314988b6893daac17235e7 [term 2 LEADER]: Raft consensus shutting down.
I20260809 15:45:44.864476 21289 raft_consensus.cc:2258] T 8c8c4f25c5364a919da21f5fca365c45 P cab341909d314988b6893daac17235e7 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260809 15:45:44.865959 21289 tablet_replica.cc:330] T 1154606dbfd849c7833c096541e8c58f P cab341909d314988b6893daac17235e7: stopping tablet replica
I20260809 15:45:44.866433 21289 raft_consensus.cc:2229] T 1154606dbfd849c7833c096541e8c58f P cab341909d314988b6893daac17235e7 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260809 15:45:44.866791 21289 raft_consensus.cc:2258] T 1154606dbfd849c7833c096541e8c58f P cab341909d314988b6893daac17235e7 [term 2 FOLLOWER]: Raft consensus is shut down!
W20260809 15:45:44.891633 22492 consensus_peers.cc:487] T 6d6e21b2d7b746469c42c6d80ad8e001 P 8586928e65c543c2b99cebf1dd7d700f -> Peer cab341909d314988b6893daac17235e7 (127.20.202.65:46511): Couldn't send request to peer cab341909d314988b6893daac17235e7. Status: Network error: Client connection negotiation failed: client connection to 127.20.202.65:46511: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260809 15:45:44.894359 21289 tablet_server.cc:194] TabletServer@127.20.202.65:0 shutdown complete.
I20260809 15:45:44.911531 21289 tablet_server.cc:178] TabletServer@127.20.202.66:0 shutting down...
W20260809 15:45:44.935415 22633 consensus_peers.cc:487] T 1154606dbfd849c7833c096541e8c58f P 27fddeff55ea476ebe7bd4881ae32035 -> Peer cab341909d314988b6893daac17235e7 (127.20.202.65:46511): Couldn't send request to peer cab341909d314988b6893daac17235e7. Status: Network error: Client connection negotiation failed: client connection to 127.20.202.65:46511: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260809 15:45:44.935891 21289 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260809 15:45:44.936424 21289 tablet_replica.cc:330] T 8c8c4f25c5364a919da21f5fca365c45 P 8586928e65c543c2b99cebf1dd7d700f: stopping tablet replica
I20260809 15:45:44.936997 21289 raft_consensus.cc:2229] T 8c8c4f25c5364a919da21f5fca365c45 P 8586928e65c543c2b99cebf1dd7d700f [term 2 FOLLOWER]: Raft consensus shutting down.
I20260809 15:45:44.937400 21289 raft_consensus.cc:2258] T 8c8c4f25c5364a919da21f5fca365c45 P 8586928e65c543c2b99cebf1dd7d700f [term 2 FOLLOWER]: Raft consensus is shut down!
I20260809 15:45:44.938978 21289 tablet_replica.cc:330] T 1154606dbfd849c7833c096541e8c58f P 8586928e65c543c2b99cebf1dd7d700f: stopping tablet replica
I20260809 15:45:44.939476 21289 raft_consensus.cc:2229] T 1154606dbfd849c7833c096541e8c58f P 8586928e65c543c2b99cebf1dd7d700f [term 2 FOLLOWER]: Raft consensus shutting down.
I20260809 15:45:44.939859 21289 raft_consensus.cc:2258] T 1154606dbfd849c7833c096541e8c58f P 8586928e65c543c2b99cebf1dd7d700f [term 2 FOLLOWER]: Raft consensus is shut down!
I20260809 15:45:44.941346 21289 tablet_replica.cc:330] T 36cb01872dee4485b764d163e0bf4e20 P 8586928e65c543c2b99cebf1dd7d700f: stopping tablet replica
I20260809 15:45:44.941789 21289 raft_consensus.cc:2229] T 36cb01872dee4485b764d163e0bf4e20 P 8586928e65c543c2b99cebf1dd7d700f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 15:45:44.942129 21289 raft_consensus.cc:2258] T 36cb01872dee4485b764d163e0bf4e20 P 8586928e65c543c2b99cebf1dd7d700f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 15:45:44.943634 21289 tablet_replica.cc:330] T 4e0d833302ff456e96ab09aa12d4c5ba P 8586928e65c543c2b99cebf1dd7d700f: stopping tablet replica
I20260809 15:45:44.944051 21289 raft_consensus.cc:2229] T 4e0d833302ff456e96ab09aa12d4c5ba P 8586928e65c543c2b99cebf1dd7d700f [term 5 LEADER]: Raft consensus shutting down.
I20260809 15:45:44.944706 21289 raft_consensus.cc:2258] T 4e0d833302ff456e96ab09aa12d4c5ba P 8586928e65c543c2b99cebf1dd7d700f [term 5 FOLLOWER]: Raft consensus is shut down!
I20260809 15:45:44.946149 21289 tablet_replica.cc:330] T 9b4926c9270e4754bac26cc4c3da4cf9 P 8586928e65c543c2b99cebf1dd7d700f: stopping tablet replica
I20260809 15:45:44.946581 21289 raft_consensus.cc:2229] T 9b4926c9270e4754bac26cc4c3da4cf9 P 8586928e65c543c2b99cebf1dd7d700f [term 2 FOLLOWER]: Raft consensus shutting down.
I20260809 15:45:44.946955 21289 raft_consensus.cc:2258] T 9b4926c9270e4754bac26cc4c3da4cf9 P 8586928e65c543c2b99cebf1dd7d700f [term 2 FOLLOWER]: Raft consensus is shut down!
I20260809 15:45:44.948483 21289 tablet_replica.cc:330] T 6b1f47a4af1b4480ab2e69adb8be3cc3 P 8586928e65c543c2b99cebf1dd7d700f: stopping tablet replica
I20260809 15:45:44.948899 21289 raft_consensus.cc:2229] T 6b1f47a4af1b4480ab2e69adb8be3cc3 P 8586928e65c543c2b99cebf1dd7d700f [term 4 FOLLOWER]: Raft consensus shutting down.
I20260809 15:45:44.949290 21289 raft_consensus.cc:2258] T 6b1f47a4af1b4480ab2e69adb8be3cc3 P 8586928e65c543c2b99cebf1dd7d700f [term 4 FOLLOWER]: Raft consensus is shut down!
I20260809 15:45:44.950672 21289 tablet_replica.cc:330] T 6eb2494806b34a2bad3215770bcb4c7c P 8586928e65c543c2b99cebf1dd7d700f: stopping tablet replica
I20260809 15:45:44.951082 21289 raft_consensus.cc:2229] T 6eb2494806b34a2bad3215770bcb4c7c P 8586928e65c543c2b99cebf1dd7d700f [term 1 LEADER]: Raft consensus shutting down.
I20260809 15:45:44.951786 21289 raft_consensus.cc:2258] T 6eb2494806b34a2bad3215770bcb4c7c P 8586928e65c543c2b99cebf1dd7d700f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 15:45:44.953191 21289 tablet_replica.cc:330] T 6d6e21b2d7b746469c42c6d80ad8e001 P 8586928e65c543c2b99cebf1dd7d700f: stopping tablet replica
I20260809 15:45:44.953609 21289 raft_consensus.cc:2229] T 6d6e21b2d7b746469c42c6d80ad8e001 P 8586928e65c543c2b99cebf1dd7d700f [term 3 LEADER]: Raft consensus shutting down.
I20260809 15:45:44.954285 21289 raft_consensus.cc:2258] T 6d6e21b2d7b746469c42c6d80ad8e001 P 8586928e65c543c2b99cebf1dd7d700f [term 3 FOLLOWER]: Raft consensus is shut down!
I20260809 15:45:44.955744 21289 tablet_replica.cc:330] T ab0d7bb4b86540b28101d1c813ee214e P 8586928e65c543c2b99cebf1dd7d700f: stopping tablet replica
I20260809 15:45:44.956162 21289 raft_consensus.cc:2229] T ab0d7bb4b86540b28101d1c813ee214e P 8586928e65c543c2b99cebf1dd7d700f [term 2 FOLLOWER]: Raft consensus shutting down.
I20260809 15:45:44.956512 21289 raft_consensus.cc:2258] T ab0d7bb4b86540b28101d1c813ee214e P 8586928e65c543c2b99cebf1dd7d700f [term 2 FOLLOWER]: Raft consensus is shut down!
I20260809 15:45:44.957971 21289 tablet_replica.cc:330] T 21aa1094e1cf4d40b51e5916cb6a2a96 P 8586928e65c543c2b99cebf1dd7d700f: stopping tablet replica
I20260809 15:45:44.958376 21289 raft_consensus.cc:2229] T 21aa1094e1cf4d40b51e5916cb6a2a96 P 8586928e65c543c2b99cebf1dd7d700f [term 3 FOLLOWER]: Raft consensus shutting down.
I20260809 15:45:44.958758 21289 raft_consensus.cc:2258] T 21aa1094e1cf4d40b51e5916cb6a2a96 P 8586928e65c543c2b99cebf1dd7d700f [term 3 FOLLOWER]: Raft consensus is shut down!
W20260809 15:45:44.973039 22635 consensus_peers.cc:487] T 1154606dbfd849c7833c096541e8c58f P 27fddeff55ea476ebe7bd4881ae32035 -> Peer 8586928e65c543c2b99cebf1dd7d700f (127.20.202.66:45399): Couldn't send request to peer 8586928e65c543c2b99cebf1dd7d700f. Status: Network error: Client connection negotiation failed: client connection to 127.20.202.66:45399: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260809 15:45:44.986148 21289 tablet_server.cc:194] TabletServer@127.20.202.66:0 shutdown complete.
I20260809 15:45:45.003379 21289 tablet_server.cc:178] TabletServer@127.20.202.67:0 shutting down...
I20260809 15:45:45.026144 21289 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260809 15:45:45.026664 21289 tablet_replica.cc:330] T 6b1f47a4af1b4480ab2e69adb8be3cc3 P 27fddeff55ea476ebe7bd4881ae32035: stopping tablet replica
I20260809 15:45:45.027195 21289 raft_consensus.cc:2229] T 6b1f47a4af1b4480ab2e69adb8be3cc3 P 27fddeff55ea476ebe7bd4881ae32035 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260809 15:45:45.027643 21289 raft_consensus.cc:2258] T 6b1f47a4af1b4480ab2e69adb8be3cc3 P 27fddeff55ea476ebe7bd4881ae32035 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260809 15:45:45.029274 21289 tablet_replica.cc:330] T 9b4926c9270e4754bac26cc4c3da4cf9 P 27fddeff55ea476ebe7bd4881ae32035: stopping tablet replica
I20260809 15:45:45.029737 21289 raft_consensus.cc:2229] T 9b4926c9270e4754bac26cc4c3da4cf9 P 27fddeff55ea476ebe7bd4881ae32035 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260809 15:45:45.030113 21289 raft_consensus.cc:2258] T 9b4926c9270e4754bac26cc4c3da4cf9 P 27fddeff55ea476ebe7bd4881ae32035 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260809 15:45:45.031596 21289 tablet_replica.cc:330] T 36cb01872dee4485b764d163e0bf4e20 P 27fddeff55ea476ebe7bd4881ae32035: stopping tablet replica
I20260809 15:45:45.032007 21289 raft_consensus.cc:2229] T 36cb01872dee4485b764d163e0bf4e20 P 27fddeff55ea476ebe7bd4881ae32035 [term 1 LEADER]: Raft consensus shutting down.
I20260809 15:45:45.032688 21289 raft_consensus.cc:2258] T 36cb01872dee4485b764d163e0bf4e20 P 27fddeff55ea476ebe7bd4881ae32035 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 15:45:45.034085 21289 tablet_replica.cc:330] T 4e0d833302ff456e96ab09aa12d4c5ba P 27fddeff55ea476ebe7bd4881ae32035: stopping tablet replica
I20260809 15:45:45.034514 21289 raft_consensus.cc:2229] T 4e0d833302ff456e96ab09aa12d4c5ba P 27fddeff55ea476ebe7bd4881ae32035 [term 5 FOLLOWER]: Raft consensus shutting down.
I20260809 15:45:45.034876 21289 raft_consensus.cc:2258] T 4e0d833302ff456e96ab09aa12d4c5ba P 27fddeff55ea476ebe7bd4881ae32035 [term 5 FOLLOWER]: Raft consensus is shut down!
I20260809 15:45:45.036322 21289 tablet_replica.cc:330] T 21aa1094e1cf4d40b51e5916cb6a2a96 P 27fddeff55ea476ebe7bd4881ae32035: stopping tablet replica
I20260809 15:45:45.036747 21289 raft_consensus.cc:2229] T 21aa1094e1cf4d40b51e5916cb6a2a96 P 27fddeff55ea476ebe7bd4881ae32035 [term 3 LEADER]: Raft consensus shutting down.
I20260809 15:45:45.037411 21289 raft_consensus.cc:2258] T 21aa1094e1cf4d40b51e5916cb6a2a96 P 27fddeff55ea476ebe7bd4881ae32035 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260809 15:45:45.038798 21289 tablet_replica.cc:330] T ab0d7bb4b86540b28101d1c813ee214e P 27fddeff55ea476ebe7bd4881ae32035: stopping tablet replica
I20260809 15:45:45.039209 21289 raft_consensus.cc:2229] T ab0d7bb4b86540b28101d1c813ee214e P 27fddeff55ea476ebe7bd4881ae32035 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260809 15:45:45.039645 21289 raft_consensus.cc:2258] T ab0d7bb4b86540b28101d1c813ee214e P 27fddeff55ea476ebe7bd4881ae32035 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260809 15:45:45.041051 21289 tablet_replica.cc:330] T 6eb2494806b34a2bad3215770bcb4c7c P 27fddeff55ea476ebe7bd4881ae32035: stopping tablet replica
I20260809 15:45:45.041460 21289 raft_consensus.cc:2229] T 6eb2494806b34a2bad3215770bcb4c7c P 27fddeff55ea476ebe7bd4881ae32035 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 15:45:45.041807 21289 raft_consensus.cc:2258] T 6eb2494806b34a2bad3215770bcb4c7c P 27fddeff55ea476ebe7bd4881ae32035 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 15:45:45.043162 21289 tablet_replica.cc:330] T 1154606dbfd849c7833c096541e8c58f P 27fddeff55ea476ebe7bd4881ae32035: stopping tablet replica
I20260809 15:45:45.043591 21289 raft_consensus.cc:2229] T 1154606dbfd849c7833c096541e8c58f P 27fddeff55ea476ebe7bd4881ae32035 [term 2 LEADER]: Raft consensus shutting down.
I20260809 15:45:45.044229 21289 raft_consensus.cc:2258] T 1154606dbfd849c7833c096541e8c58f P 27fddeff55ea476ebe7bd4881ae32035 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260809 15:45:45.045734 21289 tablet_replica.cc:330] T 8c8c4f25c5364a919da21f5fca365c45 P 27fddeff55ea476ebe7bd4881ae32035: stopping tablet replica
I20260809 15:45:45.046214 21289 raft_consensus.cc:2229] T 8c8c4f25c5364a919da21f5fca365c45 P 27fddeff55ea476ebe7bd4881ae32035 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260809 15:45:45.046633 21289 raft_consensus.cc:2258] T 8c8c4f25c5364a919da21f5fca365c45 P 27fddeff55ea476ebe7bd4881ae32035 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260809 15:45:45.048064 21289 tablet_replica.cc:330] T 6d6e21b2d7b746469c42c6d80ad8e001 P 27fddeff55ea476ebe7bd4881ae32035: stopping tablet replica
I20260809 15:45:45.048447 21289 raft_consensus.cc:2229] T 6d6e21b2d7b746469c42c6d80ad8e001 P 27fddeff55ea476ebe7bd4881ae32035 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260809 15:45:45.048820 21289 raft_consensus.cc:2258] T 6d6e21b2d7b746469c42c6d80ad8e001 P 27fddeff55ea476ebe7bd4881ae32035 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260809 15:45:45.075528 21289 tablet_server.cc:194] TabletServer@127.20.202.67:0 shutdown complete.
I20260809 15:45:45.092160 21289 master.cc:537] Master@127.20.202.126:44859 shutting down...
I20260809 15:45:45.110059 21289 raft_consensus.cc:2229] T 00000000000000000000000000000000 P b6998a43cbdd4d0eba73e55cb87cf709 [term 1 LEADER]: Raft consensus shutting down.
I20260809 15:45:45.110551 21289 raft_consensus.cc:2258] T 00000000000000000000000000000000 P b6998a43cbdd4d0eba73e55cb87cf709 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 15:45:45.110852 21289 tablet_replica.cc:330] T 00000000000000000000000000000000 P b6998a43cbdd4d0eba73e55cb87cf709: stopping tablet replica
I20260809 15:45:45.128620 21289 master.cc:552] Master@127.20.202.126:44859 shutdown complete.
27fddeff55ea476ebe7bd4881ae32035 1
8586928e65c543c2b99cebf1dd7d700f 4
cab341909d314988b6893daac17235e7 5
27fddeff55ea476ebe7bd4881ae32035 3
8586928e65c543c2b99cebf1dd7d700f 4
cab341909d314988b6893daac17235e7 3
27fddeff55ea476ebe7bd4881ae32035 1
8586928e65c543c2b99cebf1dd7d700f 1
cab341909d314988b6893daac17235e7 8
27fddeff55ea476ebe7bd4881ae32035 3
8586928e65c543c2b99cebf1dd7d700f 3
cab341909d314988b6893daac17235e7 4
[ OK ] LeaderRebalancerTest.FunctionalTestForNotDivided (28367 ms)
[ RUN ] LeaderRebalancerTest.AddTserver
I20260809 15:45:45.164405 21289 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.20.202.126:43895
I20260809 15:45:45.165326 21289 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:45:45.170853 23076 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:45:45.170868 23077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:45:45.172232 23081 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:45:45.172456 21289 server_base.cc:1034] running on GCE node
I20260809 15:45:45.173389 21289 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260809 15:45:45.173560 21289 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260809 15:45:45.173722 21289 hybrid_clock.cc:648] HybridClock initialized: now 1786290345173710 us; error 0 us; skew 500 ppm
I20260809 15:45:45.174214 21289 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 15:45:45.176452 21289 webserver.cc:460] Webserver started at http://127.20.202.126:35405/ using document root <none> and password file <none>
I20260809 15:45:45.176877 21289 fs_manager.cc:359] Metadata directory not provided
I20260809 15:45:45.177038 21289 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:45:45.177299 21289 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 15:45:45.178400 21289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj2AJ_N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786290285951082-21289-0/minicluster-data/master-0-root/instance:
uuid: "a86a2513b8024a869d80303dcbf44163"
format_stamp: "Formatted at 2026-08-09 15:45:45 on dist-test-slave-gjrz"
I20260809 15:45:45.182534 21289 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260809 15:45:45.185562 23089 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:45:45.186187 21289 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 15:45:45.186414 21289 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj2AJ_N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786290285951082-21289-0/minicluster-data/master-0-root
uuid: "a86a2513b8024a869d80303dcbf44163"
format_stamp: "Formatted at 2026-08-09 15:45:45 on dist-test-slave-gjrz"
I20260809 15:45:45.186664 21289 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj2AJ_N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786290285951082-21289-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskj2AJ_N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786290285951082-21289-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskj2AJ_N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786290285951082-21289-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:45:45.208698 21289 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:45:45.209739 21289 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:45:45.244727 21289 rpc_server.cc:307] RPC server started. Bound to: 127.20.202.126:43895
I20260809 15:45:45.244832 23176 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.202.126:43895 every 8 connection(s)
I20260809 15:45:45.248185 23177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:45.257213 23177 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a86a2513b8024a869d80303dcbf44163: Bootstrap starting.
I20260809 15:45:45.261242 23177 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a86a2513b8024a869d80303dcbf44163: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:45.265110 23177 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a86a2513b8024a869d80303dcbf44163: No bootstrap required, opened a new log
I20260809 15:45:45.267020 23177 raft_consensus.cc:348] T 00000000000000000000000000000000 P a86a2513b8024a869d80303dcbf44163 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a86a2513b8024a869d80303dcbf44163" member_type: VOTER }
I20260809 15:45:45.267485 23177 raft_consensus.cc:374] T 00000000000000000000000000000000 P a86a2513b8024a869d80303dcbf44163 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:45.267696 23177 raft_consensus.cc:729] T 00000000000000000000000000000000 P a86a2513b8024a869d80303dcbf44163 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a86a2513b8024a869d80303dcbf44163, State: Initialized, Role: FOLLOWER
I20260809 15:45:45.268250 23177 consensus_queue.cc:260] T 00000000000000000000000000000000 P a86a2513b8024a869d80303dcbf44163 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a86a2513b8024a869d80303dcbf44163" member_type: VOTER }
I20260809 15:45:45.268684 23177 raft_consensus.cc:388] T 00000000000000000000000000000000 P a86a2513b8024a869d80303dcbf44163 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 15:45:45.268898 23177 raft_consensus.cc:482] T 00000000000000000000000000000000 P a86a2513b8024a869d80303dcbf44163 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 15:45:45.269168 23177 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a86a2513b8024a869d80303dcbf44163 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:45.272567 23177 raft_consensus.cc:504] T 00000000000000000000000000000000 P a86a2513b8024a869d80303dcbf44163 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a86a2513b8024a869d80303dcbf44163" member_type: VOTER }
I20260809 15:45:45.273027 23177 leader_election.cc:302] T 00000000000000000000000000000000 P a86a2513b8024a869d80303dcbf44163 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a86a2513b8024a869d80303dcbf44163; no voters:
I20260809 15:45:45.274029 23177 leader_election.cc:288] T 00000000000000000000000000000000 P a86a2513b8024a869d80303dcbf44163 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 15:45:45.274367 23180 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a86a2513b8024a869d80303dcbf44163 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:45:45.275588 23180 raft_consensus.cc:686] T 00000000000000000000000000000000 P a86a2513b8024a869d80303dcbf44163 [term 1 LEADER]: Becoming Leader. State: Replica: a86a2513b8024a869d80303dcbf44163, State: Running, Role: LEADER
I20260809 15:45:45.276196 23180 consensus_queue.cc:237] T 00000000000000000000000000000000 P a86a2513b8024a869d80303dcbf44163 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a86a2513b8024a869d80303dcbf44163" member_type: VOTER }
I20260809 15:45:45.276686 23177 sys_catalog.cc:564] T 00000000000000000000000000000000 P a86a2513b8024a869d80303dcbf44163 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 15:45:45.278939 23182 sys_catalog.cc:455] T 00000000000000000000000000000000 P a86a2513b8024a869d80303dcbf44163 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a86a2513b8024a869d80303dcbf44163. Latest consensus state: current_term: 1 leader_uuid: "a86a2513b8024a869d80303dcbf44163" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a86a2513b8024a869d80303dcbf44163" member_type: VOTER } }
I20260809 15:45:45.279008 23181 sys_catalog.cc:455] T 00000000000000000000000000000000 P a86a2513b8024a869d80303dcbf44163 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a86a2513b8024a869d80303dcbf44163" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a86a2513b8024a869d80303dcbf44163" member_type: VOTER } }
I20260809 15:45:45.279659 23181 sys_catalog.cc:458] T 00000000000000000000000000000000 P a86a2513b8024a869d80303dcbf44163 [sys.catalog]: This master's current role is: LEADER
I20260809 15:45:45.279654 23182 sys_catalog.cc:458] T 00000000000000000000000000000000 P a86a2513b8024a869d80303dcbf44163 [sys.catalog]: This master's current role is: LEADER
I20260809 15:45:45.284307 23187 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 15:45:45.291141 23187 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 15:45:45.292085 21289 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260809 15:45:45.298950 23187 catalog_manager.cc:1347] Generated new cluster ID: 1d1be1c1135b489cae3e341f2b2b772b
I20260809 15:45:45.299149 23187 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 15:45:45.316289 23187 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 15:45:45.317688 23187 catalog_manager.cc:1504] Loading token signing keys...
I20260809 15:45:45.328604 23187 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a86a2513b8024a869d80303dcbf44163: Generated new TSK 0
I20260809 15:45:45.329108 23187 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 15:45:45.358098 21289 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:45:45.363324 23203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:45:45.364342 23208 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:45:45.365921 23214 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:45:45.366119 21289 server_base.cc:1034] running on GCE node
I20260809 15:45:45.367092 21289 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260809 15:45:45.367280 21289 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260809 15:45:45.367435 21289 hybrid_clock.cc:648] HybridClock initialized: now 1786290345367420 us; error 0 us; skew 500 ppm
I20260809 15:45:45.367939 21289 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 15:45:45.370048 21289 webserver.cc:460] Webserver started at http://127.20.202.65:38255/ using document root <none> and password file <none>
I20260809 15:45:45.370481 21289 fs_manager.cc:359] Metadata directory not provided
I20260809 15:45:45.370656 21289 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:45:45.370888 21289 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 15:45:45.371945 21289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj2AJ_N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786290285951082-21289-0/minicluster-data/ts-0-root/instance:
uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
format_stamp: "Formatted at 2026-08-09 15:45:45 on dist-test-slave-gjrz"
I20260809 15:45:45.376012 21289 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260809 15:45:45.379009 23222 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:45:45.379686 21289 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260809 15:45:45.379930 21289 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj2AJ_N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786290285951082-21289-0/minicluster-data/ts-0-root
uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
format_stamp: "Formatted at 2026-08-09 15:45:45 on dist-test-slave-gjrz"
I20260809 15:45:45.380173 21289 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj2AJ_N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786290285951082-21289-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskj2AJ_N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786290285951082-21289-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskj2AJ_N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786290285951082-21289-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:45:45.394246 21289 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:45:45.395238 21289 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:45:45.396579 21289 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 15:45:45.398829 21289 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 15:45:45.399013 21289 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 15:45:45.399189 21289 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 15:45:45.399354 21289 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260809 15:45:45.436648 21289 rpc_server.cc:307] RPC server started. Bound to: 127.20.202.65:35459
I20260809 15:45:45.436771 23320 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.202.65:35459 every 8 connection(s)
I20260809 15:45:45.440795 21289 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:45:45.447633 23328 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:45:45.448248 23330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:45:45.450999 23333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:45:45.451046 21289 server_base.cc:1034] running on GCE node
I20260809 15:45:45.452641 21289 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260809 15:45:45.452883 21289 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260809 15:45:45.453058 21289 hybrid_clock.cc:648] HybridClock initialized: now 1786290345453044 us; error 0 us; skew 500 ppm
I20260809 15:45:45.453724 21289 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 15:45:45.454859 23322 heartbeater.cc:347] Connected to a master server at 127.20.202.126:43895
I20260809 15:45:45.455183 23322 heartbeater.cc:464] Registering TS with master...
I20260809 15:45:45.455838 23322 heartbeater.cc:511] Master 127.20.202.126:43895 requested a full tablet report, sending...
I20260809 15:45:45.457896 23118 ts_manager.cc:194] Registered new tserver with Master: 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459)
I20260809 15:45:45.458000 21289 webserver.cc:460] Webserver started at http://127.20.202.66:46225/ using document root <none> and password file <none>
I20260809 15:45:45.458595 21289 fs_manager.cc:359] Metadata directory not provided
I20260809 15:45:45.458827 21289 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:45:45.459124 21289 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 15:45:45.460340 23118 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:56216
I20260809 15:45:45.460582 21289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj2AJ_N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786290285951082-21289-0/minicluster-data/ts-1-root/instance:
uuid: "c0f38c1908414566aa9115bb2cbdd85d"
format_stamp: "Formatted at 2026-08-09 15:45:45 on dist-test-slave-gjrz"
I20260809 15:45:45.465176 21289 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260809 15:45:45.468118 23343 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:45:45.468762 21289 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 15:45:45.469003 21289 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj2AJ_N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786290285951082-21289-0/minicluster-data/ts-1-root
uuid: "c0f38c1908414566aa9115bb2cbdd85d"
format_stamp: "Formatted at 2026-08-09 15:45:45 on dist-test-slave-gjrz"
I20260809 15:45:45.469245 21289 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj2AJ_N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786290285951082-21289-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskj2AJ_N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786290285951082-21289-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskj2AJ_N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786290285951082-21289-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:45:45.484197 21289 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:45:45.485209 21289 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:45:45.486584 21289 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 15:45:45.488704 21289 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 15:45:45.488885 21289 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 15:45:45.489090 21289 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 15:45:45.489228 21289 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 15:45:45.526410 21289 rpc_server.cc:307] RPC server started. Bound to: 127.20.202.66:36263
I20260809 15:45:45.526513 23443 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.202.66:36263 every 8 connection(s)
I20260809 15:45:45.530682 21289 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:45:45.537606 23454 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:45:45.537657 23451 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:45:45.542052 23462 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:45:45.542376 21289 server_base.cc:1034] running on GCE node
I20260809 15:45:45.543449 23444 heartbeater.cc:347] Connected to a master server at 127.20.202.126:43895
I20260809 15:45:45.543800 23444 heartbeater.cc:464] Registering TS with master...
I20260809 15:45:45.543825 21289 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260809 15:45:45.544154 21289 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260809 15:45:45.544341 21289 hybrid_clock.cc:648] HybridClock initialized: now 1786290345544326 us; error 0 us; skew 500 ppm
I20260809 15:45:45.544636 23444 heartbeater.cc:511] Master 127.20.202.126:43895 requested a full tablet report, sending...
I20260809 15:45:45.545047 21289 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 15:45:45.546686 23118 ts_manager.cc:194] Registered new tserver with Master: c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:45:45.547658 21289 webserver.cc:460] Webserver started at http://127.20.202.67:45759/ using document root <none> and password file <none>
I20260809 15:45:45.548252 21289 fs_manager.cc:359] Metadata directory not provided
I20260809 15:45:45.548463 21289 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:45:45.548763 21289 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 15:45:45.548731 23118 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:56218
I20260809 15:45:45.550215 21289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj2AJ_N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786290285951082-21289-0/minicluster-data/ts-2-root/instance:
uuid: "65c36d164b0540f099d592f38307ee6c"
format_stamp: "Formatted at 2026-08-09 15:45:45 on dist-test-slave-gjrz"
I20260809 15:45:45.554600 21289 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.007s
I20260809 15:45:45.557528 23467 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:45:45.558171 21289 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 15:45:45.558416 21289 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj2AJ_N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786290285951082-21289-0/minicluster-data/ts-2-root
uuid: "65c36d164b0540f099d592f38307ee6c"
format_stamp: "Formatted at 2026-08-09 15:45:45 on dist-test-slave-gjrz"
I20260809 15:45:45.558666 21289 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj2AJ_N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786290285951082-21289-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskj2AJ_N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786290285951082-21289-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskj2AJ_N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786290285951082-21289-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:45:45.569525 21289 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:45:45.570530 21289 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:45:45.571861 21289 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 15:45:45.573825 21289 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 15:45:45.574003 21289 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 15:45:45.574210 21289 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 15:45:45.574349 21289 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 15:45:45.611164 21289 rpc_server.cc:307] RPC server started. Bound to: 127.20.202.67:40553
I20260809 15:45:45.611292 23567 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.202.67:40553 every 8 connection(s)
I20260809 15:45:45.625137 23568 heartbeater.cc:347] Connected to a master server at 127.20.202.126:43895
I20260809 15:45:45.625448 23568 heartbeater.cc:464] Registering TS with master...
I20260809 15:45:45.626076 23568 heartbeater.cc:511] Master 127.20.202.126:43895 requested a full tablet report, sending...
I20260809 15:45:45.627732 23118 ts_manager.cc:194] Registered new tserver with Master: 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:45:45.628134 21289 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.014022437s
I20260809 15:45:45.629156 23118 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:56222
I20260809 15:45:45.630155 21289 test_util.cc:268] Using random seed: 562321851
I20260809 15:45:45.652143 23118 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56230:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}""\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 15:45:45.657896 23118 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 15:45:45.853826 23269 tablet_service.cc:1467] Processing CreateTablet for tablet cef3e7277d7d4760824a3436ed4a847d (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260809 15:45:45.855221 23269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cef3e7277d7d4760824a3436ed4a847d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:45.856314 23266 tablet_service.cc:1467] Processing CreateTablet for tablet 0c48a3049897452383257aa757834fb5 (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260809 15:45:45.857573 23266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c48a3049897452383257aa757834fb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:45.860278 23268 tablet_service.cc:1467] Processing CreateTablet for tablet a3622f6e3e3641cea1960e3a31498bb7 (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260809 15:45:45.861510 23268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3622f6e3e3641cea1960e3a31498bb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:45.861346 23265 tablet_service.cc:1467] Processing CreateTablet for tablet f7715d71dbac447b9351fc5eb36c5e80 (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260809 15:45:45.862525 23265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7715d71dbac447b9351fc5eb36c5e80. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:45.863405 23267 tablet_service.cc:1467] Processing CreateTablet for tablet 5ad8a3b25d7b49e5a2142d5709edd0c7 (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260809 15:45:45.864615 23267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ad8a3b25d7b49e5a2142d5709edd0c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:45.865688 23257 tablet_service.cc:1467] Processing CreateTablet for tablet 16ca412a0dae4feebe85dd01bf1a9173 (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260809 15:45:45.866885 23257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16ca412a0dae4feebe85dd01bf1a9173. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:45.860284 23260 tablet_service.cc:1467] Processing CreateTablet for tablet 953eea268e17491ca76f3a555860a512 (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260809 15:45:45.868604 23260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 953eea268e17491ca76f3a555860a512. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:45.869246 23515 tablet_service.cc:1467] Processing CreateTablet for tablet cef3e7277d7d4760824a3436ed4a847d (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260809 15:45:45.870484 23515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cef3e7277d7d4760824a3436ed4a847d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:45.872752 23258 tablet_service.cc:1467] Processing CreateTablet for tablet c0741ab44d99439abc3aa3bdd27ee160 (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260809 15:45:45.873982 23258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0741ab44d99439abc3aa3bdd27ee160. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:45.875319 23514 tablet_service.cc:1467] Processing CreateTablet for tablet a3622f6e3e3641cea1960e3a31498bb7 (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260809 15:45:45.876590 23514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3622f6e3e3641cea1960e3a31498bb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:45.876358 23256 tablet_service.cc:1467] Processing CreateTablet for tablet 2db057fd10e149a9895dff6ef8c9f038 (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260809 15:45:45.877621 23256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2db057fd10e149a9895dff6ef8c9f038. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:45.878420 23510 tablet_service.cc:1467] Processing CreateTablet for tablet 5ad8a3b25d7b49e5a2142d5709edd0c7 (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260809 15:45:45.879755 23510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ad8a3b25d7b49e5a2142d5709edd0c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:45.879827 23254 tablet_service.cc:1467] Processing CreateTablet for tablet 2a823d35f4f348d1b52ba9e6d80b00d2 (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260809 15:45:45.881057 23254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a823d35f4f348d1b52ba9e6d80b00d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:45.881584 23504 tablet_service.cc:1467] Processing CreateTablet for tablet 0c48a3049897452383257aa757834fb5 (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260809 15:45:45.882885 23504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c48a3049897452383257aa757834fb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:45.884299 23503 tablet_service.cc:1467] Processing CreateTablet for tablet f7715d71dbac447b9351fc5eb36c5e80 (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260809 15:45:45.885504 23503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7715d71dbac447b9351fc5eb36c5e80. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:45.886827 23499 tablet_service.cc:1467] Processing CreateTablet for tablet 2db057fd10e149a9895dff6ef8c9f038 (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260809 15:45:45.888063 23499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2db057fd10e149a9895dff6ef8c9f038. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:45.888367 23502 tablet_service.cc:1467] Processing CreateTablet for tablet 953eea268e17491ca76f3a555860a512 (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260809 15:45:45.888448 23389 tablet_service.cc:1467] Processing CreateTablet for tablet a3622f6e3e3641cea1960e3a31498bb7 (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260809 15:45:45.889577 23502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 953eea268e17491ca76f3a555860a512. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:45.889732 23389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3622f6e3e3641cea1960e3a31498bb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:45.890033 23381 tablet_service.cc:1467] Processing CreateTablet for tablet 16ca412a0dae4feebe85dd01bf1a9173 (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260809 15:45:45.891249 23381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16ca412a0dae4feebe85dd01bf1a9173. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:45.892091 23388 tablet_service.cc:1467] Processing CreateTablet for tablet 5ad8a3b25d7b49e5a2142d5709edd0c7 (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260809 15:45:45.892359 23390 tablet_service.cc:1467] Processing CreateTablet for tablet cef3e7277d7d4760824a3436ed4a847d (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260809 15:45:45.893302 23388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ad8a3b25d7b49e5a2142d5709edd0c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:45.893513 23500 tablet_service.cc:1467] Processing CreateTablet for tablet 16ca412a0dae4feebe85dd01bf1a9173 (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260809 15:45:45.894665 23500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16ca412a0dae4feebe85dd01bf1a9173. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:45.895576 23390 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cef3e7277d7d4760824a3436ed4a847d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:45.896359 23380 tablet_service.cc:1467] Processing CreateTablet for tablet 2db057fd10e149a9895dff6ef8c9f038 (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260809 15:45:45.897473 23384 tablet_service.cc:1467] Processing CreateTablet for tablet f7715d71dbac447b9351fc5eb36c5e80 (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260809 15:45:45.892295 23501 tablet_service.cc:1467] Processing CreateTablet for tablet c0741ab44d99439abc3aa3bdd27ee160 (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260809 15:45:45.896377 23387 tablet_service.cc:1467] Processing CreateTablet for tablet 0c48a3049897452383257aa757834fb5 (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260809 15:45:45.898701 23501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0741ab44d99439abc3aa3bdd27ee160. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:45.900236 23378 tablet_service.cc:1467] Processing CreateTablet for tablet 2a823d35f4f348d1b52ba9e6d80b00d2 (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260809 15:45:45.900367 23382 tablet_service.cc:1467] Processing CreateTablet for tablet c0741ab44d99439abc3aa3bdd27ee160 (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260809 15:45:45.901465 23383 tablet_service.cc:1467] Processing CreateTablet for tablet 953eea268e17491ca76f3a555860a512 (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260809 15:45:45.897589 23380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2db057fd10e149a9895dff6ef8c9f038. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:45.909739 23384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7715d71dbac447b9351fc5eb36c5e80. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:45.910310 23382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0741ab44d99439abc3aa3bdd27ee160. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:45.910866 23383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 953eea268e17491ca76f3a555860a512. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:45.911478 23378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a823d35f4f348d1b52ba9e6d80b00d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:45.915603 23387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c48a3049897452383257aa757834fb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:45.978982 23495 tablet_service.cc:1467] Processing CreateTablet for tablet 2a823d35f4f348d1b52ba9e6d80b00d2 (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260809 15:45:45.980273 23495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a823d35f4f348d1b52ba9e6d80b00d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:45.983752 23384 tablet_service.cc:1467] Processing CreateTablet for tablet 05ad7d8fdca64e3da527280ca5be9280 (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260809 15:45:45.984952 23384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05ad7d8fdca64e3da527280ca5be9280. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:45.996727 23500 tablet_service.cc:1467] Processing CreateTablet for tablet 998c6cee72cf40228c6dba064c0db2da (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260809 15:45:45.996858 23383 tablet_service.cc:1467] Processing CreateTablet for tablet 998c6cee72cf40228c6dba064c0db2da (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260809 15:45:45.997888 23500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 998c6cee72cf40228c6dba064c0db2da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:45.998039 23383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 998c6cee72cf40228c6dba064c0db2da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:46.001039 23389 tablet_service.cc:1467] Processing CreateTablet for tablet 5712f6b6fffc4cc5a8ddfbe63e4097df (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260809 15:45:46.003837 23501 tablet_service.cc:1467] Processing CreateTablet for tablet 2424fd9bd7f14e508fd106cbfda786a4 (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260809 15:45:46.012751 23591 tablet_bootstrap.cc:492] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P c0f38c1908414566aa9115bb2cbdd85d: Bootstrap starting.
I20260809 15:45:46.009100 23390 tablet_service.cc:1467] Processing CreateTablet for tablet ba56725e42f74a1baa4c04d9660727f9 (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260809 15:45:46.014169 23390 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba56725e42f74a1baa4c04d9660727f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:46.013535 23501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2424fd9bd7f14e508fd106cbfda786a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:46.010257 23256 tablet_service.cc:1467] Processing CreateTablet for tablet 05ad7d8fdca64e3da527280ca5be9280 (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260809 15:45:46.016645 23503 tablet_service.cc:1467] Processing CreateTablet for tablet 05ad7d8fdca64e3da527280ca5be9280 (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260809 15:45:46.016793 23502 tablet_service.cc:1467] Processing CreateTablet for tablet 472816994d814cb198b69e5248697440 (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260809 15:45:46.017848 23503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05ad7d8fdca64e3da527280ca5be9280. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:46.021283 23388 tablet_service.cc:1467] Processing CreateTablet for tablet 472816994d814cb198b69e5248697440 (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260809 15:45:46.022441 23388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 472816994d814cb198b69e5248697440. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:46.023396 23257 tablet_service.cc:1467] Processing CreateTablet for tablet 998c6cee72cf40228c6dba064c0db2da (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260809 15:45:45.999119 23499 tablet_service.cc:1467] Processing CreateTablet for tablet 5712f6b6fffc4cc5a8ddfbe63e4097df (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260809 15:45:46.010977 23510 tablet_service.cc:1467] Processing CreateTablet for tablet ba56725e42f74a1baa4c04d9660727f9 (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260809 15:45:46.026722 23258 tablet_service.cc:1467] Processing CreateTablet for tablet 5712f6b6fffc4cc5a8ddfbe63e4097df (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260809 15:45:46.030099 23504 tablet_service.cc:1467] Processing CreateTablet for tablet 8e4ffcf83c234ae7910ddef3ec0b4bda (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260809 15:45:46.030805 23256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05ad7d8fdca64e3da527280ca5be9280. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:46.036154 23257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 998c6cee72cf40228c6dba064c0db2da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:46.035832 23269 tablet_service.cc:1467] Processing CreateTablet for tablet 3d880de657e544468c82157769622a26 (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260809 15:45:46.037142 23269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d880de657e544468c82157769622a26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:46.039354 23267 tablet_service.cc:1467] Processing CreateTablet for tablet e14d118b2eaa42aa8a7cd4025aba7128 (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260809 15:45:46.039978 23389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5712f6b6fffc4cc5a8ddfbe63e4097df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:46.039791 23265 tablet_service.cc:1467] Processing CreateTablet for tablet ba56725e42f74a1baa4c04d9660727f9 (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260809 15:45:46.040669 23267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e14d118b2eaa42aa8a7cd4025aba7128. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:46.041147 23381 tablet_service.cc:1467] Processing CreateTablet for tablet 3d880de657e544468c82157769622a26 (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260809 15:45:46.038144 23268 tablet_service.cc:1467] Processing CreateTablet for tablet 6ced718c1a5c4da79cb8c1ebd1435b87 (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260809 15:45:46.042379 23381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d880de657e544468c82157769622a26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:46.043766 23502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 472816994d814cb198b69e5248697440. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:46.042601 23268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ced718c1a5c4da79cb8c1ebd1435b87. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:46.069021 23515 tablet_service.cc:1467] Processing CreateTablet for tablet 0c29937a80724ddb9d07a27534a146f3 (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260809 15:45:46.070226 23515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c29937a80724ddb9d07a27534a146f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:46.075997 23254 tablet_service.cc:1467] Processing CreateTablet for tablet 472816994d814cb198b69e5248697440 (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260809 15:45:46.077169 23254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 472816994d814cb198b69e5248697440. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:46.078133 23510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba56725e42f74a1baa4c04d9660727f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:46.080416 23380 tablet_service.cc:1467] Processing CreateTablet for tablet 2424fd9bd7f14e508fd106cbfda786a4 (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260809 15:45:46.081622 23380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2424fd9bd7f14e508fd106cbfda786a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:46.083508 23591 tablet_bootstrap.cc:654] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P c0f38c1908414566aa9115bb2cbdd85d: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:46.083676 23504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e4ffcf83c234ae7910ddef3ec0b4bda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:46.086699 23266 tablet_service.cc:1467] Processing CreateTablet for tablet 8e4ffcf83c234ae7910ddef3ec0b4bda (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260809 15:45:46.094628 23499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5712f6b6fffc4cc5a8ddfbe63e4097df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:46.094920 23593 tablet_bootstrap.cc:492] T 0c48a3049897452383257aa757834fb5 P 65c36d164b0540f099d592f38307ee6c: Bootstrap starting.
I20260809 15:45:46.097410 23594 tablet_bootstrap.cc:492] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 67b3aeadfa7a432abe8af4d292e04ab9: Bootstrap starting.
I20260809 15:45:46.044879 23514 tablet_service.cc:1467] Processing CreateTablet for tablet 3d880de657e544468c82157769622a26 (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260809 15:45:46.099392 23514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d880de657e544468c82157769622a26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:46.093765 23382 tablet_service.cc:1467] Processing CreateTablet for tablet e14d118b2eaa42aa8a7cd4025aba7128 (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260809 15:45:46.104384 23388 tablet_service.cc:1467] Processing CreateTablet for tablet 8e4ffcf83c234ae7910ddef3ec0b4bda (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260809 15:45:46.106571 23265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba56725e42f74a1baa4c04d9660727f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:46.107178 23266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e4ffcf83c234ae7910ddef3ec0b4bda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:46.109549 23260 tablet_service.cc:1467] Processing CreateTablet for tablet 2424fd9bd7f14e508fd106cbfda786a4 (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260809 15:45:46.110698 23260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2424fd9bd7f14e508fd106cbfda786a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:46.104761 23382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e14d118b2eaa42aa8a7cd4025aba7128. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:46.111301 23593 tablet_bootstrap.cc:654] T 0c48a3049897452383257aa757834fb5 P 65c36d164b0540f099d592f38307ee6c: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:46.115533 23258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5712f6b6fffc4cc5a8ddfbe63e4097df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:46.116322 23495 tablet_service.cc:1467] Processing CreateTablet for tablet 6ced718c1a5c4da79cb8c1ebd1435b87 (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260809 15:45:46.117497 23495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ced718c1a5c4da79cb8c1ebd1435b87. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:46.118343 23378 tablet_service.cc:1467] Processing CreateTablet for tablet 6ced718c1a5c4da79cb8c1ebd1435b87 (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260809 15:45:46.118804 23267 tablet_service.cc:1467] Processing CreateTablet for tablet 7baa2492df0340c4a5ac241959ce1ac3 (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260809 15:45:46.123553 23388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e4ffcf83c234ae7910ddef3ec0b4bda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:46.132140 23378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ced718c1a5c4da79cb8c1ebd1435b87. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:46.126969 23387 tablet_service.cc:1467] Processing CreateTablet for tablet 7baa2492df0340c4a5ac241959ce1ac3 (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260809 15:45:46.136232 23387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7baa2492df0340c4a5ac241959ce1ac3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:46.138157 23510 tablet_service.cc:1467] Processing CreateTablet for tablet e14d118b2eaa42aa8a7cd4025aba7128 (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260809 15:45:46.139420 23267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7baa2492df0340c4a5ac241959ce1ac3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:46.140522 23594 tablet_bootstrap.cc:654] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 67b3aeadfa7a432abe8af4d292e04ab9: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:46.154667 23504 tablet_service.cc:1467] Processing CreateTablet for tablet 7baa2492df0340c4a5ac241959ce1ac3 (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260809 15:45:46.157925 23502 tablet_service.cc:1467] Processing CreateTablet for tablet ae6d2c08a7a94f58ac8dd698e3fd456c (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260809 15:45:46.158121 23268 tablet_service.cc:1467] Processing CreateTablet for tablet ae6d2c08a7a94f58ac8dd698e3fd456c (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260809 15:45:46.159229 23268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae6d2c08a7a94f58ac8dd698e3fd456c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:46.165979 23390 tablet_service.cc:1467] Processing CreateTablet for tablet ae6d2c08a7a94f58ac8dd698e3fd456c (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260809 15:45:46.167431 23510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e14d118b2eaa42aa8a7cd4025aba7128. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:46.167136 23390 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae6d2c08a7a94f58ac8dd698e3fd456c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:46.176236 23499 tablet_service.cc:1467] Processing CreateTablet for tablet 73f6f44c259547f1b05a3a434616fa54 (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260809 15:45:46.176667 23593 tablet_bootstrap.cc:492] T 0c48a3049897452383257aa757834fb5 P 65c36d164b0540f099d592f38307ee6c: No bootstrap required, opened a new log
I20260809 15:45:46.177206 23593 ts_tablet_manager.cc:1397] T 0c48a3049897452383257aa757834fb5 P 65c36d164b0540f099d592f38307ee6c: Time spent bootstrapping tablet: real 0.083s user 0.011s sys 0.011s
I20260809 15:45:46.179886 23593 raft_consensus.cc:348] T 0c48a3049897452383257aa757834fb5 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:46.180693 23593 raft_consensus.cc:374] T 0c48a3049897452383257aa757834fb5 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:46.181037 23593 raft_consensus.cc:729] T 0c48a3049897452383257aa757834fb5 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Initialized, Role: FOLLOWER
I20260809 15:45:46.181888 23593 consensus_queue.cc:260] T 0c48a3049897452383257aa757834fb5 P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:46.204211 23380 tablet_service.cc:1467] Processing CreateTablet for tablet 73f6f44c259547f1b05a3a434616fa54 (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260809 15:45:46.211441 23502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae6d2c08a7a94f58ac8dd698e3fd456c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:46.211835 23387 tablet_service.cc:1467] Processing CreateTablet for tablet e71a05ba1d4d42bd95c3f85f9313399e (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260809 15:45:46.212970 23499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73f6f44c259547f1b05a3a434616fa54. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:46.214468 23504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7baa2492df0340c4a5ac241959ce1ac3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:46.218804 23380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73f6f44c259547f1b05a3a434616fa54. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:46.219192 23258 tablet_service.cc:1467] Processing CreateTablet for tablet 73f6f44c259547f1b05a3a434616fa54 (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260809 15:45:46.220270 23387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e71a05ba1d4d42bd95c3f85f9313399e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:46.220395 23258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73f6f44c259547f1b05a3a434616fa54. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:46.229702 23256 tablet_service.cc:1467] Processing CreateTablet for tablet 0c29937a80724ddb9d07a27534a146f3 (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260809 15:45:46.245496 23591 tablet_bootstrap.cc:492] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P c0f38c1908414566aa9115bb2cbdd85d: No bootstrap required, opened a new log
I20260809 15:45:46.245934 23591 ts_tablet_manager.cc:1397] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P c0f38c1908414566aa9115bb2cbdd85d: Time spent bootstrapping tablet: real 0.233s user 0.023s sys 0.027s
I20260809 15:45:46.248406 23591 raft_consensus.cc:348] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:46.249068 23591 raft_consensus.cc:374] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:46.249333 23591 raft_consensus.cc:729] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Initialized, Role: FOLLOWER
I20260809 15:45:46.250167 23591 consensus_queue.cc:260] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P c0f38c1908414566aa9115bb2cbdd85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:46.253176 23503 tablet_service.cc:1467] Processing CreateTablet for tablet 3296ef50be254fc29cfc77d22a8f7a23 (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260809 15:45:46.254348 23503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3296ef50be254fc29cfc77d22a8f7a23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:46.256695 23266 tablet_service.cc:1467] Processing CreateTablet for tablet e71a05ba1d4d42bd95c3f85f9313399e (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260809 15:45:46.257184 23510 tablet_service.cc:1467] Processing CreateTablet for tablet 6ae14a74a26f434db08c652e097385c8 (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260809 15:45:46.258354 23510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ae14a74a26f434db08c652e097385c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:46.260725 23256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c29937a80724ddb9d07a27534a146f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:46.263890 23266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e71a05ba1d4d42bd95c3f85f9313399e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:46.266901 23260 tablet_service.cc:1467] Processing CreateTablet for tablet 6ae14a74a26f434db08c652e097385c8 (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260809 15:45:46.269727 23260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ae14a74a26f434db08c652e097385c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:46.269567 23257 tablet_service.cc:1467] Processing CreateTablet for tablet d2d4f749fcfd4fb3bfb8115af3471598 (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260809 15:45:46.267833 23381 tablet_service.cc:1467] Processing CreateTablet for tablet 3296ef50be254fc29cfc77d22a8f7a23 (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260809 15:45:46.258701 23500 tablet_service.cc:1467] Processing CreateTablet for tablet e71a05ba1d4d42bd95c3f85f9313399e (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260809 15:45:46.270789 23500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e71a05ba1d4d42bd95c3f85f9313399e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:46.271018 23384 tablet_service.cc:1467] Processing CreateTablet for tablet 6ae14a74a26f434db08c652e097385c8 (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260809 15:45:46.258683 23501 tablet_service.cc:1467] Processing CreateTablet for tablet d2d4f749fcfd4fb3bfb8115af3471598 (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260809 15:45:46.272195 23384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ae14a74a26f434db08c652e097385c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:46.272653 23501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2d4f749fcfd4fb3bfb8115af3471598. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:46.274878 23381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3296ef50be254fc29cfc77d22a8f7a23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:46.268615 23515 tablet_service.cc:1467] Processing CreateTablet for tablet 99bdb93503db4de4a1cfb0161ea0c07c (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260809 15:45:46.283975 23378 tablet_service.cc:1467] Processing CreateTablet for tablet d2d4f749fcfd4fb3bfb8115af3471598 (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260809 15:45:46.284911 23257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2d4f749fcfd4fb3bfb8115af3471598. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:46.284010 23515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99bdb93503db4de4a1cfb0161ea0c07c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:46.285925 23265 tablet_service.cc:1467] Processing CreateTablet for tablet 99bdb93503db4de4a1cfb0161ea0c07c (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
W20260809 15:45:46.287534 23193 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260809 15:45:46.300278 23267 tablet_service.cc:1467] Processing CreateTablet for tablet 0a51c0baaa6e432997c9d3552af4f254 (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260809 15:45:46.300473 23265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99bdb93503db4de4a1cfb0161ea0c07c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:46.271570 23269 tablet_service.cc:1467] Processing CreateTablet for tablet 3296ef50be254fc29cfc77d22a8f7a23 (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260809 15:45:46.306267 23514 tablet_service.cc:1467] Processing CreateTablet for tablet 0a51c0baaa6e432997c9d3552af4f254 (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260809 15:45:46.307433 23514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a51c0baaa6e432997c9d3552af4f254. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:46.317013 23378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2d4f749fcfd4fb3bfb8115af3471598. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:46.317358 23269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3296ef50be254fc29cfc77d22a8f7a23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:46.322726 23383 tablet_service.cc:1467] Processing CreateTablet for tablet 0c29937a80724ddb9d07a27534a146f3 (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260809 15:45:46.324280 23594 tablet_bootstrap.cc:492] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 67b3aeadfa7a432abe8af4d292e04ab9: No bootstrap required, opened a new log
I20260809 15:45:46.324721 23594 ts_tablet_manager.cc:1397] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent bootstrapping tablet: real 0.228s user 0.007s sys 0.006s
I20260809 15:45:46.326937 23268 tablet_service.cc:1467] Processing CreateTablet for tablet 0da4d5283c8a46878bd39b85e5c71f66 (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260809 15:45:46.329267 23267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a51c0baaa6e432997c9d3552af4f254. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:46.332276 23593 ts_tablet_manager.cc:1428] T 0c48a3049897452383257aa757834fb5 P 65c36d164b0540f099d592f38307ee6c: Time spent starting tablet: real 0.155s user 0.010s sys 0.041s
I20260809 15:45:46.333000 23568 heartbeater.cc:503] Master 127.20.202.126:43895 was elected leader, sending a full tablet report...
I20260809 15:45:46.333168 23593 tablet_bootstrap.cc:492] T f7715d71dbac447b9351fc5eb36c5e80 P 65c36d164b0540f099d592f38307ee6c: Bootstrap starting.
I20260809 15:45:46.337895 23444 heartbeater.cc:503] Master 127.20.202.126:43895 was elected leader, sending a full tablet report...
I20260809 15:45:46.339871 23254 tablet_service.cc:1467] Processing CreateTablet for tablet a88421dba80d4d3388aa5eb62e23853b (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260809 15:45:46.343632 23268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0da4d5283c8a46878bd39b85e5c71f66. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:46.356874 23591 ts_tablet_manager.cc:1428] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P c0f38c1908414566aa9115bb2cbdd85d: Time spent starting tablet: real 0.111s user 0.011s sys 0.012s
I20260809 15:45:46.327231 23594 raft_consensus.cc:348] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:46.357633 23269 tablet_service.cc:1467] Processing CreateTablet for tablet 763e58d6ba644eea830a6c1664d39c9b (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260809 15:45:46.358034 23594 raft_consensus.cc:374] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:46.358372 23594 raft_consensus.cc:729] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Initialized, Role: FOLLOWER
I20260809 15:45:46.359119 23594 consensus_queue.cc:260] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 67b3aeadfa7a432abe8af4d292e04ab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:46.367388 23383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c29937a80724ddb9d07a27534a146f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:46.373072 23266 tablet_service.cc:1467] Processing CreateTablet for tablet d1a134ba33ea4076bbaac0ae460c17ba (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260809 15:45:46.378080 23495 tablet_service.cc:1467] Processing CreateTablet for tablet 0da4d5283c8a46878bd39b85e5c71f66 (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260809 15:45:46.381709 23382 tablet_service.cc:1467] Processing CreateTablet for tablet 99bdb93503db4de4a1cfb0161ea0c07c (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
W20260809 15:45:46.368932 23569 tablet.cc:2369] T 0c48a3049897452383257aa757834fb5 P 65c36d164b0540f099d592f38307ee6c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 15:45:46.383553 23382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99bdb93503db4de4a1cfb0161ea0c07c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:46.391865 23387 tablet_service.cc:1467] Processing CreateTablet for tablet 0a51c0baaa6e432997c9d3552af4f254 (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260809 15:45:46.392459 23380 tablet_service.cc:1467] Processing CreateTablet for tablet 0da4d5283c8a46878bd39b85e5c71f66 (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260809 15:45:46.393004 23387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a51c0baaa6e432997c9d3552af4f254. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:46.393563 23380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0da4d5283c8a46878bd39b85e5c71f66. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:46.375530 23591 tablet_bootstrap.cc:492] T f7715d71dbac447b9351fc5eb36c5e80 P c0f38c1908414566aa9115bb2cbdd85d: Bootstrap starting.
I20260809 15:45:46.394955 23269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 763e58d6ba644eea830a6c1664d39c9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:46.399906 23495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0da4d5283c8a46878bd39b85e5c71f66. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:46.400280 23390 tablet_service.cc:1467] Processing CreateTablet for tablet 763e58d6ba644eea830a6c1664d39c9b (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260809 15:45:46.400631 23500 tablet_service.cc:1467] Processing CreateTablet for tablet a88421dba80d4d3388aa5eb62e23853b (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260809 15:45:46.401816 23500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a88421dba80d4d3388aa5eb62e23853b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:46.408839 23502 tablet_service.cc:1467] Processing CreateTablet for tablet d1a134ba33ea4076bbaac0ae460c17ba (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260809 15:45:46.409984 23502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1a134ba33ea4076bbaac0ae460c17ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:46.408725 23515 tablet_service.cc:1467] Processing CreateTablet for tablet 763e58d6ba644eea830a6c1664d39c9b (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260809 15:45:46.409828 23499 tablet_service.cc:1467] Processing CreateTablet for tablet 7660dfa2ef694927b8eda4100a2bcd9b (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260809 15:45:46.412025 23594 ts_tablet_manager.cc:1428] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent starting tablet: real 0.087s user 0.009s sys 0.006s
I20260809 15:45:46.412734 23322 heartbeater.cc:503] Master 127.20.202.126:43895 was elected leader, sending a full tablet report...
I20260809 15:45:46.412935 23594 tablet_bootstrap.cc:492] T 2db057fd10e149a9895dff6ef8c9f038 P 67b3aeadfa7a432abe8af4d292e04ab9: Bootstrap starting.
I20260809 15:45:46.415855 23495 tablet_service.cc:1467] Processing CreateTablet for tablet c5c2e425a74b4e64ae70b5def750434c (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260809 15:45:46.420601 23503 tablet_service.cc:1467] Processing CreateTablet for tablet 538a228ddfb54bc6a36528d5ff07597b (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260809 15:45:46.411168 23499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7660dfa2ef694927b8eda4100a2bcd9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:46.426626 23266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1a134ba33ea4076bbaac0ae460c17ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:46.430727 23390 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 763e58d6ba644eea830a6c1664d39c9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:46.434894 23594 tablet_bootstrap.cc:654] T 2db057fd10e149a9895dff6ef8c9f038 P 67b3aeadfa7a432abe8af4d292e04ab9: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:46.435921 23510 tablet_service.cc:1467] Processing CreateTablet for tablet 0acc47ce21c3422698d1ef7277c3ed63 (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260809 15:45:46.436854 23504 tablet_service.cc:1467] Processing CreateTablet for tablet 30c1740ef6874df0a332fbd6fd4b03ce (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260809 15:45:46.445094 23501 tablet_service.cc:1467] Processing CreateTablet for tablet 1f9d6565aa4c45a3b3f01210d0eb71f7 (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260809 15:45:46.445748 23593 tablet_bootstrap.cc:654] T f7715d71dbac447b9351fc5eb36c5e80 P 65c36d164b0540f099d592f38307ee6c: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:46.446385 23503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 538a228ddfb54bc6a36528d5ff07597b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:46.436997 23381 tablet_service.cc:1467] Processing CreateTablet for tablet d1a134ba33ea4076bbaac0ae460c17ba (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260809 15:45:46.447784 23591 tablet_bootstrap.cc:654] T f7715d71dbac447b9351fc5eb36c5e80 P c0f38c1908414566aa9115bb2cbdd85d: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:46.447995 23381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1a134ba33ea4076bbaac0ae460c17ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:46.447873 23389 tablet_service.cc:1467] Processing CreateTablet for tablet c5c2e425a74b4e64ae70b5def750434c (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260809 15:45:46.451548 23515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 763e58d6ba644eea830a6c1664d39c9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:46.451874 23389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5c2e425a74b4e64ae70b5def750434c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:46.455549 23504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30c1740ef6874df0a332fbd6fd4b03ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:46.460940 23388 tablet_service.cc:1467] Processing CreateTablet for tablet a88421dba80d4d3388aa5eb62e23853b (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260809 15:45:46.462575 23510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0acc47ce21c3422698d1ef7277c3ed63. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:46.467499 23501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f9d6565aa4c45a3b3f01210d0eb71f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:46.470345 23495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5c2e425a74b4e64ae70b5def750434c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:46.441083 23384 tablet_service.cc:1467] Processing CreateTablet for tablet 7660dfa2ef694927b8eda4100a2bcd9b (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260809 15:45:46.462088 23388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a88421dba80d4d3388aa5eb62e23853b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:46.475960 23384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7660dfa2ef694927b8eda4100a2bcd9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:46.484144 23500 tablet_service.cc:1467] Processing CreateTablet for tablet 4bcac4b24e7a4448abdc934bba5a624e (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260809 15:45:46.490402 23593 tablet_bootstrap.cc:492] T f7715d71dbac447b9351fc5eb36c5e80 P 65c36d164b0540f099d592f38307ee6c: No bootstrap required, opened a new log
I20260809 15:45:46.490465 23378 tablet_service.cc:1467] Processing CreateTablet for tablet 538a228ddfb54bc6a36528d5ff07597b (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260809 15:45:46.491086 23593 ts_tablet_manager.cc:1397] T f7715d71dbac447b9351fc5eb36c5e80 P 65c36d164b0540f099d592f38307ee6c: Time spent bootstrapping tablet: real 0.158s user 0.015s sys 0.000s
I20260809 15:45:46.491684 23378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 538a228ddfb54bc6a36528d5ff07597b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:46.493801 23593 raft_consensus.cc:348] T f7715d71dbac447b9351fc5eb36c5e80 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:46.494539 23593 raft_consensus.cc:374] T f7715d71dbac447b9351fc5eb36c5e80 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:46.494820 23593 raft_consensus.cc:729] T f7715d71dbac447b9351fc5eb36c5e80 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Initialized, Role: FOLLOWER
I20260809 15:45:46.495070 23382 tablet_service.cc:1467] Processing CreateTablet for tablet 0acc47ce21c3422698d1ef7277c3ed63 (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260809 15:45:46.496232 23382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0acc47ce21c3422698d1ef7277c3ed63. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:46.496707 23383 tablet_service.cc:1467] Processing CreateTablet for tablet 30c1740ef6874df0a332fbd6fd4b03ce (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260809 15:45:46.497828 23383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30c1740ef6874df0a332fbd6fd4b03ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:46.498682 23593 consensus_queue.cc:260] T f7715d71dbac447b9351fc5eb36c5e80 P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:46.509619 23388 tablet_service.cc:1467] Processing CreateTablet for tablet 4bcac4b24e7a4448abdc934bba5a624e (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260809 15:45:46.510900 23388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bcac4b24e7a4448abdc934bba5a624e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:46.515506 23594 tablet_bootstrap.cc:492] T 2db057fd10e149a9895dff6ef8c9f038 P 67b3aeadfa7a432abe8af4d292e04ab9: No bootstrap required, opened a new log
I20260809 15:45:46.515944 23594 ts_tablet_manager.cc:1397] T 2db057fd10e149a9895dff6ef8c9f038 P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent bootstrapping tablet: real 0.103s user 0.007s sys 0.013s
I20260809 15:45:46.518431 23594 raft_consensus.cc:348] T 2db057fd10e149a9895dff6ef8c9f038 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:46.500270 23500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bcac4b24e7a4448abdc934bba5a624e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:46.520026 23495 tablet_service.cc:1467] Processing CreateTablet for tablet 297c82ed906d4b48b184985eed3de601 (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260809 15:45:46.521119 23495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 297c82ed906d4b48b184985eed3de601. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:46.523710 23387 tablet_service.cc:1467] Processing CreateTablet for tablet 1f9d6565aa4c45a3b3f01210d0eb71f7 (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260809 15:45:46.524842 23387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f9d6565aa4c45a3b3f01210d0eb71f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:46.526515 23388 tablet_service.cc:1467] Processing CreateTablet for tablet 297c82ed906d4b48b184985eed3de601 (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260809 15:45:46.526823 23601 raft_consensus.cc:482] T 0c48a3049897452383257aa757834fb5 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:45:46.527395 23601 raft_consensus.cc:504] T 0c48a3049897452383257aa757834fb5 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:46.527868 23388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 297c82ed906d4b48b184985eed3de601. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:46.529829 23601 leader_election.cc:288] T 0c48a3049897452383257aa757834fb5 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:45:46.519094 23594 raft_consensus.cc:374] T 2db057fd10e149a9895dff6ef8c9f038 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:46.541478 23501 tablet_service.cc:1467] Processing CreateTablet for tablet d0721a4403a440f383318a3cea9d1442 (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260809 15:45:46.537114 23594 raft_consensus.cc:729] T 2db057fd10e149a9895dff6ef8c9f038 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Initialized, Role: FOLLOWER
I20260809 15:45:46.542397 23593 ts_tablet_manager.cc:1428] T f7715d71dbac447b9351fc5eb36c5e80 P 65c36d164b0540f099d592f38307ee6c: Time spent starting tablet: real 0.051s user 0.006s sys 0.000s
I20260809 15:45:46.542654 23501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0721a4403a440f383318a3cea9d1442. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:46.548017 23593 tablet_bootstrap.cc:492] T 16ca412a0dae4feebe85dd01bf1a9173 P 65c36d164b0540f099d592f38307ee6c: Bootstrap starting.
I20260809 15:45:46.541478 23514 tablet_service.cc:1467] Processing CreateTablet for tablet b3aa6a9a52804df0ba9b0ca3c82f38c3 (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260809 15:45:46.544691 23502 tablet_service.cc:1467] Processing CreateTablet for tablet 563daf42d7784db195ea99c3cb21b370 (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260809 15:45:46.552255 23502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 563daf42d7784db195ea99c3cb21b370. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:46.553277 23593 tablet_bootstrap.cc:654] T 16ca412a0dae4feebe85dd01bf1a9173 P 65c36d164b0540f099d592f38307ee6c: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:46.554958 23591 tablet_bootstrap.cc:492] T f7715d71dbac447b9351fc5eb36c5e80 P c0f38c1908414566aa9115bb2cbdd85d: No bootstrap required, opened a new log
I20260809 15:45:46.555491 23591 ts_tablet_manager.cc:1397] T f7715d71dbac447b9351fc5eb36c5e80 P c0f38c1908414566aa9115bb2cbdd85d: Time spent bootstrapping tablet: real 0.186s user 0.035s sys 0.017s
I20260809 15:45:46.555768 23514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3aa6a9a52804df0ba9b0ca3c82f38c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:46.555658 23594 consensus_queue.cc:260] T 2db057fd10e149a9895dff6ef8c9f038 P 67b3aeadfa7a432abe8af4d292e04ab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:46.557750 23594 ts_tablet_manager.cc:1428] T 2db057fd10e149a9895dff6ef8c9f038 P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent starting tablet: real 0.042s user 0.004s sys 0.000s
I20260809 15:45:46.558625 23594 tablet_bootstrap.cc:492] T 16ca412a0dae4feebe85dd01bf1a9173 P 67b3aeadfa7a432abe8af4d292e04ab9: Bootstrap starting.
I20260809 15:45:46.562614 23594 tablet_bootstrap.cc:654] T 16ca412a0dae4feebe85dd01bf1a9173 P 67b3aeadfa7a432abe8af4d292e04ab9: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:46.459086 23266 tablet_service.cc:1467] Processing CreateTablet for tablet 7660dfa2ef694927b8eda4100a2bcd9b (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260809 15:45:46.584857 23266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7660dfa2ef694927b8eda4100a2bcd9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:46.589372 23254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a88421dba80d4d3388aa5eb62e23853b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:46.597930 23267 tablet_service.cc:1467] Processing CreateTablet for tablet 538a228ddfb54bc6a36528d5ff07597b (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260809 15:45:46.599488 23267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 538a228ddfb54bc6a36528d5ff07597b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:46.601447 23257 tablet_service.cc:1467] Processing CreateTablet for tablet c5c2e425a74b4e64ae70b5def750434c (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260809 15:45:46.602988 23257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5c2e425a74b4e64ae70b5def750434c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:46.557541 23591 raft_consensus.cc:348] T f7715d71dbac447b9351fc5eb36c5e80 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:46.619628 23604 raft_consensus.cc:482] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:45:46.647722 23604 raft_consensus.cc:504] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:46.598986 23258 tablet_service.cc:1467] Processing CreateTablet for tablet 0acc47ce21c3422698d1ef7277c3ed63 (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260809 15:45:46.640054 23602 raft_consensus.cc:482] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:45:46.657456 23602 raft_consensus.cc:504] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:46.660634 23602 leader_election.cc:288] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:45:46.684075 23604 leader_election.cc:288] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:45:46.699731 23591 raft_consensus.cc:374] T f7715d71dbac447b9351fc5eb36c5e80 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:46.696812 23258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0acc47ce21c3422698d1ef7277c3ed63. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:46.700217 23591 raft_consensus.cc:729] T f7715d71dbac447b9351fc5eb36c5e80 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Initialized, Role: FOLLOWER
I20260809 15:45:46.701150 23591 consensus_queue.cc:260] T f7715d71dbac447b9351fc5eb36c5e80 P c0f38c1908414566aa9115bb2cbdd85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:46.704241 23381 tablet_service.cc:1467] Processing CreateTablet for tablet b3aa6a9a52804df0ba9b0ca3c82f38c3 (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260809 15:45:46.704355 23268 tablet_service.cc:1467] Processing CreateTablet for tablet 30c1740ef6874df0a332fbd6fd4b03ce (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260809 15:45:46.705583 23381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3aa6a9a52804df0ba9b0ca3c82f38c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:46.705693 23268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30c1740ef6874df0a332fbd6fd4b03ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:46.705991 23269 tablet_service.cc:1467] Processing CreateTablet for tablet 297c82ed906d4b48b184985eed3de601 (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260809 15:45:46.707685 23503 tablet_service.cc:1467] Processing CreateTablet for tablet 00a1881267a14ba8b78c81040f17a014 (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260809 15:45:46.709103 23503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00a1881267a14ba8b78c81040f17a014. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:46.699695 23504 tablet_service.cc:1467] Processing CreateTablet for tablet 4564dfc6d4094c1b9240b367ccecf3b1 (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260809 15:45:46.712402 23504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4564dfc6d4094c1b9240b367ccecf3b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:46.707121 23269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 297c82ed906d4b48b184985eed3de601. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:46.685115 23260 tablet_service.cc:1467] Processing CreateTablet for tablet 1f9d6565aa4c45a3b3f01210d0eb71f7 (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260809 15:45:46.726300 23380 tablet_service.cc:1467] Processing CreateTablet for tablet d0721a4403a440f383318a3cea9d1442 (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260809 15:45:46.740216 23591 ts_tablet_manager.cc:1428] T f7715d71dbac447b9351fc5eb36c5e80 P c0f38c1908414566aa9115bb2cbdd85d: Time spent starting tablet: real 0.184s user 0.005s sys 0.002s
I20260809 15:45:46.741482 23591 tablet_bootstrap.cc:492] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d: Bootstrap starting.
I20260809 15:45:46.743006 23256 tablet_service.cc:1467] Processing CreateTablet for tablet 4bcac4b24e7a4448abdc934bba5a624e (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260809 15:45:46.744689 23256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bcac4b24e7a4448abdc934bba5a624e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:46.769228 23269 tablet_service.cc:1467] Processing CreateTablet for tablet b3aa6a9a52804df0ba9b0ca3c82f38c3 (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260809 15:45:46.770663 23269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3aa6a9a52804df0ba9b0ca3c82f38c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:46.780468 23256 tablet_service.cc:1467] Processing CreateTablet for tablet d0721a4403a440f383318a3cea9d1442 (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260809 15:45:46.784725 23256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0721a4403a440f383318a3cea9d1442. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:46.797722 23601 raft_consensus.cc:482] T f7715d71dbac447b9351fc5eb36c5e80 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:45:46.798156 23601 raft_consensus.cc:504] T f7715d71dbac447b9351fc5eb36c5e80 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:46.800190 23601 leader_election.cc:288] T f7715d71dbac447b9351fc5eb36c5e80 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:45:46.815388 23500 tablet_service.cc:1467] Processing CreateTablet for tablet 213543cdbe814c89a17a963469922c1b (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260809 15:45:46.816604 23500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 213543cdbe814c89a17a963469922c1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:46.817714 23269 tablet_service.cc:1467] Processing CreateTablet for tablet 563daf42d7784db195ea99c3cb21b370 (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260809 15:45:46.822188 23381 tablet_service.cc:1467] Processing CreateTablet for tablet 563daf42d7784db195ea99c3cb21b370 (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260809 15:45:46.826326 23389 tablet_service.cc:1467] Processing CreateTablet for tablet 4564dfc6d4094c1b9240b367ccecf3b1 (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260809 15:45:46.826520 23501 tablet_service.cc:1467] Processing CreateTablet for tablet f9e7cadcd24d45fcbfa84209225e8f2d (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260809 15:45:46.827785 23501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9e7cadcd24d45fcbfa84209225e8f2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:46.828775 23593 tablet_bootstrap.cc:492] T 16ca412a0dae4feebe85dd01bf1a9173 P 65c36d164b0540f099d592f38307ee6c: No bootstrap required, opened a new log
I20260809 15:45:46.829671 23593 ts_tablet_manager.cc:1397] T 16ca412a0dae4feebe85dd01bf1a9173 P 65c36d164b0540f099d592f38307ee6c: Time spent bootstrapping tablet: real 0.282s user 0.016s sys 0.000s
I20260809 15:45:46.812964 23380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0721a4403a440f383318a3cea9d1442. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:46.845412 23514 tablet_service.cc:1467] Processing CreateTablet for tablet df20671a7b874dbb98c0ef28371c1308 (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260809 15:45:46.847754 23515 tablet_service.cc:1467] Processing CreateTablet for tablet 3924b648461e463b9d7925e5671d68ae (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260809 15:45:46.848217 23510 tablet_service.cc:1467] Processing CreateTablet for tablet 294d4bcc1e7a477cb56455581277a60f (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260809 15:45:46.847095 23381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 563daf42d7784db195ea99c3cb21b370. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:46.853559 23510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 294d4bcc1e7a477cb56455581277a60f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:46.849593 23260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f9d6565aa4c45a3b3f01210d0eb71f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:46.861075 23514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df20671a7b874dbb98c0ef28371c1308. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:46.862442 23594 tablet_bootstrap.cc:492] T 16ca412a0dae4feebe85dd01bf1a9173 P 67b3aeadfa7a432abe8af4d292e04ab9: No bootstrap required, opened a new log
I20260809 15:45:46.863027 23266 tablet_service.cc:1467] Processing CreateTablet for tablet 4564dfc6d4094c1b9240b367ccecf3b1 (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260809 15:45:46.868347 23502 tablet_service.cc:1467] Processing CreateTablet for tablet 07fcabb569414bddbd39df6cedd24e64 (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260809 15:45:46.869467 23502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07fcabb569414bddbd39df6cedd24e64. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:46.871619 23515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3924b648461e463b9d7925e5671d68ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:46.879432 23389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4564dfc6d4094c1b9240b367ccecf3b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:46.879833 23500 tablet_service.cc:1467] Processing CreateTablet for tablet d3e94b7e50754a81b3d72a07476cb7b9 (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260809 15:45:46.881278 23500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3e94b7e50754a81b3d72a07476cb7b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:46.884244 23266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4564dfc6d4094c1b9240b367ccecf3b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:46.883965 23387 tablet_service.cc:1467] Processing CreateTablet for tablet 213543cdbe814c89a17a963469922c1b (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260809 15:45:46.885076 23387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 213543cdbe814c89a17a963469922c1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:46.886983 23390 tablet_service.cc:1467] Processing CreateTablet for tablet f9e7cadcd24d45fcbfa84209225e8f2d (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260809 15:45:46.888185 23390 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9e7cadcd24d45fcbfa84209225e8f2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:46.888142 23383 tablet_service.cc:1467] Processing CreateTablet for tablet 294d4bcc1e7a477cb56455581277a60f (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260809 15:45:46.889487 23383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 294d4bcc1e7a477cb56455581277a60f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:46.891901 23256 tablet_service.cc:1467] Processing CreateTablet for tablet 00a1881267a14ba8b78c81040f17a014 (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260809 15:45:46.897071 23269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 563daf42d7784db195ea99c3cb21b370. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:46.905764 23594 ts_tablet_manager.cc:1397] T 16ca412a0dae4feebe85dd01bf1a9173 P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent bootstrapping tablet: real 0.347s user 0.012s sys 0.006s
I20260809 15:45:46.908842 23594 raft_consensus.cc:348] T 16ca412a0dae4feebe85dd01bf1a9173 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:46.909531 23594 raft_consensus.cc:374] T 16ca412a0dae4feebe85dd01bf1a9173 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:46.909790 23594 raft_consensus.cc:729] T 16ca412a0dae4feebe85dd01bf1a9173 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Initialized, Role: FOLLOWER
I20260809 15:45:46.914002 23510 tablet_service.cc:1467] Processing CreateTablet for tablet 4b30f28311964a0b9149a50495e83d8f (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260809 15:45:46.917246 23381 tablet_service.cc:1467] Processing CreateTablet for tablet 07fcabb569414bddbd39df6cedd24e64 (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260809 15:45:46.917354 23495 tablet_service.cc:1467] Processing CreateTablet for tablet d3be1e356b254d22b995be32f1c12f6f (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260809 15:45:46.918566 23381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07fcabb569414bddbd39df6cedd24e64. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:46.919447 23499 tablet_service.cc:1467] Processing CreateTablet for tablet cee02fff2c9c48a19f43eb358a89e7b0 (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260809 15:45:46.915144 23510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b30f28311964a0b9149a50495e83d8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:46.922003 23286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c48a3049897452383257aa757834fb5" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:45:46.923388 23495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3be1e356b254d22b995be32f1c12f6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:46.923705 23285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7715d71dbac447b9351fc5eb36c5e80" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:45:46.924058 23499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cee02fff2c9c48a19f43eb358a89e7b0. 1 dirs total, 0 dirs full, 0 dirs failed
W20260809 15:45:46.925298 23470 leader_election.cc:341] T 0c48a3049897452383257aa757834fb5 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459): Illegal state: must be running to vote when last-logged opid is not known
W20260809 15:45:46.926059 23470 leader_election.cc:341] T f7715d71dbac447b9351fc5eb36c5e80 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:46.927489 23268 tablet_service.cc:1467] Processing CreateTablet for tablet 213543cdbe814c89a17a963469922c1b (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260809 15:45:46.929010 23504 tablet_service.cc:1467] Processing CreateTablet for tablet 28e17dcb5efe45b8b6a8c42da03b4819 (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260809 15:45:46.947844 23504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28e17dcb5efe45b8b6a8c42da03b4819. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:46.930671 23257 tablet_service.cc:1467] Processing CreateTablet for tablet f9e7cadcd24d45fcbfa84209225e8f2d (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260809 15:45:46.934188 23258 tablet_service.cc:1467] Processing CreateTablet for tablet 07fcabb569414bddbd39df6cedd24e64 (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260809 15:45:46.942559 23388 tablet_service.cc:1467] Processing CreateTablet for tablet d3e94b7e50754a81b3d72a07476cb7b9 (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260809 15:45:46.953536 23388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3e94b7e50754a81b3d72a07476cb7b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:46.929711 23593 raft_consensus.cc:348] T 16ca412a0dae4feebe85dd01bf1a9173 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:46.953550 23529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad8a3b25d7b49e5a2142d5709edd0c7" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:45:46.954056 23593 raft_consensus.cc:374] T 16ca412a0dae4feebe85dd01bf1a9173 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:46.954507 23593 raft_consensus.cc:729] T 16ca412a0dae4feebe85dd01bf1a9173 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Initialized, Role: FOLLOWER
I20260809 15:45:46.944860 23591 tablet_bootstrap.cc:654] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d: Neither blocks nor log segments found. Creating new log.
W20260809 15:45:46.955228 23353 leader_election.cc:341] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:46.956254 23404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c48a3049897452383257aa757834fb5" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:45:46.956853 23594 consensus_queue.cc:260] T 16ca412a0dae4feebe85dd01bf1a9173 P 67b3aeadfa7a432abe8af4d292e04ab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:46.957760 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7715d71dbac447b9351fc5eb36c5e80" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:45:46.958740 23402 raft_consensus.cc:2454] T f7715d71dbac447b9351fc5eb36c5e80 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 0.
I20260809 15:45:46.958405 23593 consensus_queue.cc:260] T 16ca412a0dae4feebe85dd01bf1a9173 P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:45:46.959887 23472 leader_election.cc:341] T 0c48a3049897452383257aa757834fb5 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:46.927816 23256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00a1881267a14ba8b78c81040f17a014. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:46.963716 23382 tablet_service.cc:1467] Processing CreateTablet for tablet 00a1881267a14ba8b78c81040f17a014 (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260809 15:45:46.964880 23382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00a1881267a14ba8b78c81040f17a014. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:46.967648 23268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 213543cdbe814c89a17a963469922c1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:46.974517 23593 ts_tablet_manager.cc:1428] T 16ca412a0dae4feebe85dd01bf1a9173 P 65c36d164b0540f099d592f38307ee6c: Time spent starting tablet: real 0.144s user 0.007s sys 0.000s
I20260809 15:45:46.975569 23593 tablet_bootstrap.cc:492] T 2db057fd10e149a9895dff6ef8c9f038 P 65c36d164b0540f099d592f38307ee6c: Bootstrap starting.
I20260809 15:45:46.960345 23472 leader_election.cc:302] T 0c48a3049897452383257aa757834fb5 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:45:46.976998 23472 leader_election.cc:302] T f7715d71dbac447b9351fc5eb36c5e80 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters: 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:45:46.982424 23601 raft_consensus.cc:2726] T 0c48a3049897452383257aa757834fb5 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 15:45:46.984302 23601 raft_consensus.cc:2781] T f7715d71dbac447b9351fc5eb36c5e80 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 15:45:46.984650 23601 raft_consensus.cc:482] T f7715d71dbac447b9351fc5eb36c5e80 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:45:46.984951 23601 raft_consensus.cc:3037] T f7715d71dbac447b9351fc5eb36c5e80 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:46.985669 23604 raft_consensus.cc:482] T 2db057fd10e149a9895dff6ef8c9f038 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:45:46.986145 23604 raft_consensus.cc:504] T 2db057fd10e149a9895dff6ef8c9f038 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:46.988056 23604 leader_election.cc:288] T 2db057fd10e149a9895dff6ef8c9f038 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:45:46.983732 23257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9e7cadcd24d45fcbfa84209225e8f2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:47.007764 23594 ts_tablet_manager.cc:1428] T 16ca412a0dae4feebe85dd01bf1a9173 P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent starting tablet: real 0.102s user 0.006s sys 0.001s
I20260809 15:45:47.008773 23594 tablet_bootstrap.cc:492] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 67b3aeadfa7a432abe8af4d292e04ab9: Bootstrap starting.
I20260809 15:45:47.013002 23254 tablet_service.cc:1467] Processing CreateTablet for tablet d3e94b7e50754a81b3d72a07476cb7b9 (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260809 15:45:47.014138 23254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3e94b7e50754a81b3d72a07476cb7b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:47.023516 23258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07fcabb569414bddbd39df6cedd24e64. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:47.029611 23265 tablet_service.cc:1467] Processing CreateTablet for tablet df20671a7b874dbb98c0ef28371c1308 (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260809 15:45:47.030900 23265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df20671a7b874dbb98c0ef28371c1308. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:47.032759 23501 tablet_service.cc:1467] Processing CreateTablet for tablet 907bacfa79d04a55a68c8dbe60056462 (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260809 15:45:47.038478 23378 tablet_service.cc:1467] Processing CreateTablet for tablet 3924b648461e463b9d7925e5671d68ae (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260809 15:45:47.038949 23285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad8a3b25d7b49e5a2142d5709edd0c7" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:45:47.039633 23285 raft_consensus.cc:2454] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 0.
I20260809 15:45:47.042371 23351 leader_election.cc:302] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c
I20260809 15:45:47.043572 23602 raft_consensus.cc:2781] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 15:45:47.043916 23602 raft_consensus.cc:482] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:45:47.044229 23602 raft_consensus.cc:3037] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:47.045368 23384 tablet_service.cc:1467] Processing CreateTablet for tablet df20671a7b874dbb98c0ef28371c1308 (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260809 15:45:47.046494 23384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df20671a7b874dbb98c0ef28371c1308. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:47.058218 23501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 907bacfa79d04a55a68c8dbe60056462. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:47.063195 23404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2db057fd10e149a9895dff6ef8c9f038" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:45:47.063226 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad8a3b25d7b49e5a2142d5709edd0c7" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
W20260809 15:45:47.064716 23228 leader_election.cc:341] T 2db057fd10e149a9895dff6ef8c9f038 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:47.023586 23601 raft_consensus.cc:504] T f7715d71dbac447b9351fc5eb36c5e80 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:47.067978 23601 leader_election.cc:288] T f7715d71dbac447b9351fc5eb36c5e80 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 1 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:45:47.068660 23285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7715d71dbac447b9351fc5eb36c5e80" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
W20260809 15:45:47.070144 23470 leader_election.cc:341] T f7715d71dbac447b9351fc5eb36c5e80 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:47.073434 23267 tablet_service.cc:1467] Processing CreateTablet for tablet 294d4bcc1e7a477cb56455581277a60f (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260809 15:45:47.074752 23267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 294d4bcc1e7a477cb56455581277a60f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:47.050906 23602 raft_consensus.cc:504] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:47.080629 23402 raft_consensus.cc:2379] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 in current term 1: Already voted for candidate c0f38c1908414566aa9115bb2cbdd85d in this term.
I20260809 15:45:47.084748 23285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad8a3b25d7b49e5a2142d5709edd0c7" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:45:47.085265 23285 raft_consensus.cc:3037] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:47.085611 23602 leader_election.cc:288] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:45:47.086122 23529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad8a3b25d7b49e5a2142d5709edd0c7" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
W20260809 15:45:47.087468 23353 leader_election.cc:341] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:47.099890 23269 tablet_service.cc:1467] Processing CreateTablet for tablet 3924b648461e463b9d7925e5671d68ae (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260809 15:45:47.105834 23260 tablet_service.cc:1467] Processing CreateTablet for tablet 28e17dcb5efe45b8b6a8c42da03b4819 (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260809 15:45:47.106969 23260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28e17dcb5efe45b8b6a8c42da03b4819. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:47.100636 23266 tablet_service.cc:1467] Processing CreateTablet for tablet 4b30f28311964a0b9149a50495e83d8f (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260809 15:45:47.108400 23594 tablet_bootstrap.cc:654] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 67b3aeadfa7a432abe8af4d292e04ab9: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:47.112946 23591 tablet_bootstrap.cc:492] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d: No bootstrap required, opened a new log
I20260809 15:45:47.113369 23591 ts_tablet_manager.cc:1397] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d: Time spent bootstrapping tablet: real 0.372s user 0.025s sys 0.019s
I20260809 15:45:47.118819 23269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3924b648461e463b9d7925e5671d68ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:47.115216 23380 tablet_service.cc:1467] Processing CreateTablet for tablet 4b30f28311964a0b9149a50495e83d8f (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260809 15:45:47.123001 23593 tablet_bootstrap.cc:654] T 2db057fd10e149a9895dff6ef8c9f038 P 65c36d164b0540f099d592f38307ee6c: Neither blocks nor log segments found. Creating new log.
W20260809 15:45:47.126127 23472 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 149206us
I20260809 15:45:47.127668 23529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad8a3b25d7b49e5a2142d5709edd0c7" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:45:47.128224 23380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b30f28311964a0b9149a50495e83d8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:47.128758 23528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2db057fd10e149a9895dff6ef8c9f038" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
W20260809 15:45:47.143100 23228 leader_election.cc:341] T 2db057fd10e149a9895dff6ef8c9f038 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:47.130244 23387 tablet_service.cc:1467] Processing CreateTablet for tablet 28e17dcb5efe45b8b6a8c42da03b4819 (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260809 15:45:47.143544 23228 leader_election.cc:302] T 2db057fd10e149a9895dff6ef8c9f038 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:45:47.132722 23383 tablet_service.cc:1467] Processing CreateTablet for tablet cee02fff2c9c48a19f43eb358a89e7b0 (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260809 15:45:47.144519 23604 raft_consensus.cc:2726] T 2db057fd10e149a9895dff6ef8c9f038 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260809 15:45:47.145617 23228 leader_election.cc:341] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:47.145998 23228 leader_election.cc:302] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:45:47.136221 23285 raft_consensus.cc:2454] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 1.
I20260809 15:45:47.139425 23266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b30f28311964a0b9149a50495e83d8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:47.140177 23591 raft_consensus.cc:348] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:47.149502 23591 raft_consensus.cc:374] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:47.149794 23591 raft_consensus.cc:729] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Initialized, Role: FOLLOWER
I20260809 15:45:47.150521 23591 consensus_queue.cc:260] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:47.143220 23378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3924b648461e463b9d7925e5671d68ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:47.150888 23256 tablet_service.cc:1467] Processing CreateTablet for tablet d3be1e356b254d22b995be32f1c12f6f (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260809 15:45:47.151718 23387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28e17dcb5efe45b8b6a8c42da03b4819. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:47.152024 23256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3be1e356b254d22b995be32f1c12f6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:47.159708 23383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cee02fff2c9c48a19f43eb358a89e7b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:47.164269 23591 ts_tablet_manager.cc:1428] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d: Time spent starting tablet: real 0.051s user 0.008s sys 0.000s
I20260809 15:45:47.165254 23591 tablet_bootstrap.cc:492] T 953eea268e17491ca76f3a555860a512 P c0f38c1908414566aa9115bb2cbdd85d: Bootstrap starting.
I20260809 15:45:47.130112 23389 tablet_service.cc:1467] Processing CreateTablet for tablet d3be1e356b254d22b995be32f1c12f6f (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260809 15:45:47.170338 23389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3be1e356b254d22b995be32f1c12f6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:47.171633 23265 tablet_service.cc:1467] Processing CreateTablet for tablet cee02fff2c9c48a19f43eb358a89e7b0 (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260809 15:45:47.172614 23604 raft_consensus.cc:2726] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 15:45:47.174458 23265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cee02fff2c9c48a19f43eb358a89e7b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:47.182052 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7715d71dbac447b9351fc5eb36c5e80" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:45:47.182626 23402 raft_consensus.cc:3037] T f7715d71dbac447b9351fc5eb36c5e80 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:47.184888 23591 tablet_bootstrap.cc:654] T 953eea268e17491ca76f3a555860a512 P c0f38c1908414566aa9115bb2cbdd85d: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:47.185806 23594 tablet_bootstrap.cc:492] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 67b3aeadfa7a432abe8af4d292e04ab9: No bootstrap required, opened a new log
I20260809 15:45:47.186241 23594 ts_tablet_manager.cc:1397] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent bootstrapping tablet: real 0.178s user 0.015s sys 0.005s
I20260809 15:45:47.189082 23594 raft_consensus.cc:348] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:47.189531 23402 raft_consensus.cc:2454] T f7715d71dbac447b9351fc5eb36c5e80 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 1.
I20260809 15:45:47.189816 23594 raft_consensus.cc:374] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:47.190158 23594 raft_consensus.cc:729] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Initialized, Role: FOLLOWER
I20260809 15:45:47.190811 23260 tablet_service.cc:1467] Processing CreateTablet for tablet 907bacfa79d04a55a68c8dbe60056462 (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260809 15:45:47.191994 23260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 907bacfa79d04a55a68c8dbe60056462. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:47.194857 23351 leader_election.cc:302] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c
I20260809 15:45:47.195950 23602 raft_consensus.cc:2781] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:45:47.200560 23593 tablet_bootstrap.cc:492] T 2db057fd10e149a9895dff6ef8c9f038 P 65c36d164b0540f099d592f38307ee6c: No bootstrap required, opened a new log
I20260809 15:45:47.203794 23593 ts_tablet_manager.cc:1397] T 2db057fd10e149a9895dff6ef8c9f038 P 65c36d164b0540f099d592f38307ee6c: Time spent bootstrapping tablet: real 0.228s user 0.026s sys 0.001s
I20260809 15:45:47.206241 23593 raft_consensus.cc:348] T 2db057fd10e149a9895dff6ef8c9f038 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:47.206934 23593 raft_consensus.cc:374] T 2db057fd10e149a9895dff6ef8c9f038 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:47.206815 23472 leader_election.cc:302] T f7715d71dbac447b9351fc5eb36c5e80 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters: 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:45:47.207223 23593 raft_consensus.cc:729] T 2db057fd10e149a9895dff6ef8c9f038 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Initialized, Role: FOLLOWER
I20260809 15:45:47.207720 23643 raft_consensus.cc:2781] T f7715d71dbac447b9351fc5eb36c5e80 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:45:47.208199 23643 raft_consensus.cc:686] T f7715d71dbac447b9351fc5eb36c5e80 P 65c36d164b0540f099d592f38307ee6c [term 1 LEADER]: Becoming Leader. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:45:47.207971 23593 consensus_queue.cc:260] T 2db057fd10e149a9895dff6ef8c9f038 P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:47.200104 23381 tablet_service.cc:1467] Processing CreateTablet for tablet 907bacfa79d04a55a68c8dbe60056462 (DEFAULT_TABLE table=test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260809 15:45:47.210978 23643 consensus_queue.cc:237] T f7715d71dbac447b9351fc5eb36c5e80 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:47.190852 23594 consensus_queue.cc:260] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 67b3aeadfa7a432abe8af4d292e04ab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:47.224099 23594 ts_tablet_manager.cc:1428] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent starting tablet: real 0.038s user 0.007s sys 0.000s
I20260809 15:45:47.231865 23381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 907bacfa79d04a55a68c8dbe60056462. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:47.238807 23604 raft_consensus.cc:482] T 16ca412a0dae4feebe85dd01bf1a9173 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:45:47.239284 23604 raft_consensus.cc:504] T 16ca412a0dae4feebe85dd01bf1a9173 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:47.241192 23604 leader_election.cc:288] T 16ca412a0dae4feebe85dd01bf1a9173 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:45:47.242930 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16ca412a0dae4feebe85dd01bf1a9173" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:45:47.244562 23529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16ca412a0dae4feebe85dd01bf1a9173" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:45:47.245179 23529 raft_consensus.cc:2454] T 16ca412a0dae4feebe85dd01bf1a9173 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 0.
W20260809 15:45:47.246330 23228 leader_election.cc:341] T 16ca412a0dae4feebe85dd01bf1a9173 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:47.248358 23228 leader_election.cc:302] T 16ca412a0dae4feebe85dd01bf1a9173 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters: c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:45:47.249399 23591 tablet_bootstrap.cc:492] T 953eea268e17491ca76f3a555860a512 P c0f38c1908414566aa9115bb2cbdd85d: No bootstrap required, opened a new log
I20260809 15:45:47.249871 23591 ts_tablet_manager.cc:1397] T 953eea268e17491ca76f3a555860a512 P c0f38c1908414566aa9115bb2cbdd85d: Time spent bootstrapping tablet: real 0.085s user 0.012s sys 0.017s
I20260809 15:45:47.250384 23604 raft_consensus.cc:2781] T 16ca412a0dae4feebe85dd01bf1a9173 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 15:45:47.250770 23604 raft_consensus.cc:482] T 16ca412a0dae4feebe85dd01bf1a9173 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:45:47.251075 23604 raft_consensus.cc:3037] T 16ca412a0dae4feebe85dd01bf1a9173 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:47.251457 23594 tablet_bootstrap.cc:492] T c0741ab44d99439abc3aa3bdd27ee160 P 67b3aeadfa7a432abe8af4d292e04ab9: Bootstrap starting.
I20260809 15:45:47.260998 23604 raft_consensus.cc:504] T 16ca412a0dae4feebe85dd01bf1a9173 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:47.220554 23602 raft_consensus.cc:686] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P c0f38c1908414566aa9115bb2cbdd85d [term 1 LEADER]: Becoming Leader. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:45:47.263618 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16ca412a0dae4feebe85dd01bf1a9173" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:45:47.264143 23602 consensus_queue.cc:237] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:45:47.264895 23228 leader_election.cc:341] T 16ca412a0dae4feebe85dd01bf1a9173 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:47.266932 23529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16ca412a0dae4feebe85dd01bf1a9173" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:45:47.267781 23529 raft_consensus.cc:3037] T 16ca412a0dae4feebe85dd01bf1a9173 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:47.274629 23604 leader_election.cc:288] T 16ca412a0dae4feebe85dd01bf1a9173 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:45:47.282634 23529 raft_consensus.cc:2454] T 16ca412a0dae4feebe85dd01bf1a9173 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 1.
I20260809 15:45:47.283771 23228 leader_election.cc:302] T 16ca412a0dae4feebe85dd01bf1a9173 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters: c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:45:47.284230 23118 catalog_manager.cc:5495] T f7715d71dbac447b9351fc5eb36c5e80 P 65c36d164b0540f099d592f38307ee6c reported cstate change: term changed from 0 to 1, leader changed from <none> to 65c36d164b0540f099d592f38307ee6c (127.20.202.67). New cstate: current_term: 1 leader_uuid: "65c36d164b0540f099d592f38307ee6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } health_report { overall_health: HEALTHY } } }
I20260809 15:45:47.286585 23594 tablet_bootstrap.cc:654] T c0741ab44d99439abc3aa3bdd27ee160 P 67b3aeadfa7a432abe8af4d292e04ab9: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:47.287773 23593 ts_tablet_manager.cc:1428] T 2db057fd10e149a9895dff6ef8c9f038 P 65c36d164b0540f099d592f38307ee6c: Time spent starting tablet: real 0.084s user 0.022s sys 0.030s
I20260809 15:45:47.288792 23593 tablet_bootstrap.cc:492] T c0741ab44d99439abc3aa3bdd27ee160 P 65c36d164b0540f099d592f38307ee6c: Bootstrap starting.
I20260809 15:45:47.289079 23591 raft_consensus.cc:348] T 953eea268e17491ca76f3a555860a512 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:47.290076 23591 raft_consensus.cc:374] T 953eea268e17491ca76f3a555860a512 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:47.290354 23591 raft_consensus.cc:729] T 953eea268e17491ca76f3a555860a512 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Initialized, Role: FOLLOWER
I20260809 15:45:47.286074 23604 raft_consensus.cc:2781] T 16ca412a0dae4feebe85dd01bf1a9173 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:45:47.295475 23591 consensus_queue.cc:260] T 953eea268e17491ca76f3a555860a512 P c0f38c1908414566aa9115bb2cbdd85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:47.296342 23593 tablet_bootstrap.cc:654] T c0741ab44d99439abc3aa3bdd27ee160 P 65c36d164b0540f099d592f38307ee6c: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:47.298799 23648 raft_consensus.cc:482] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:45:47.299324 23648 raft_consensus.cc:504] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:45:47.301602 23193 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260809 15:45:47.303092 23285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a823d35f4f348d1b52ba9e6d80b00d2" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:45:47.303778 23285 raft_consensus.cc:2454] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 0.
I20260809 15:45:47.304922 23351 leader_election.cc:302] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:45:47.305927 23602 raft_consensus.cc:2781] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 15:45:47.306255 23602 raft_consensus.cc:482] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:45:47.306581 23602 raft_consensus.cc:3037] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:47.308307 23648 leader_election.cc:288] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:45:47.308136 23116 catalog_manager.cc:5495] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P c0f38c1908414566aa9115bb2cbdd85d reported cstate change: term changed from 0 to 1, leader changed from <none> to c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66). New cstate: current_term: 1 leader_uuid: "c0f38c1908414566aa9115bb2cbdd85d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } health_report { overall_health: UNKNOWN } } }
I20260809 15:45:47.308590 23529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a823d35f4f348d1b52ba9e6d80b00d2" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:45:47.311033 23604 raft_consensus.cc:686] T 16ca412a0dae4feebe85dd01bf1a9173 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 LEADER]: Becoming Leader. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Running, Role: LEADER
W20260809 15:45:47.311931 23353 leader_election.cc:341] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:47.312187 23604 consensus_queue.cc:237] T 16ca412a0dae4feebe85dd01bf1a9173 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:47.314889 23594 tablet_bootstrap.cc:492] T c0741ab44d99439abc3aa3bdd27ee160 P 67b3aeadfa7a432abe8af4d292e04ab9: No bootstrap required, opened a new log
I20260809 15:45:47.316154 23594 ts_tablet_manager.cc:1397] T c0741ab44d99439abc3aa3bdd27ee160 P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent bootstrapping tablet: real 0.091s user 0.009s sys 0.008s
I20260809 15:45:47.320067 23602 raft_consensus.cc:504] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:47.323302 23602 leader_election.cc:288] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:45:47.323459 23285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a823d35f4f348d1b52ba9e6d80b00d2" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:45:47.325789 23594 raft_consensus.cc:348] T c0741ab44d99439abc3aa3bdd27ee160 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:47.326205 23285 raft_consensus.cc:3037] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:47.326122 23529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a823d35f4f348d1b52ba9e6d80b00d2" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
W20260809 15:45:47.327911 23353 leader_election.cc:341] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:47.329470 23591 ts_tablet_manager.cc:1428] T 953eea268e17491ca76f3a555860a512 P c0f38c1908414566aa9115bb2cbdd85d: Time spent starting tablet: real 0.079s user 0.009s sys 0.005s
I20260809 15:45:47.330535 23591 tablet_bootstrap.cc:492] T a3622f6e3e3641cea1960e3a31498bb7 P c0f38c1908414566aa9115bb2cbdd85d: Bootstrap starting.
I20260809 15:45:47.331725 23285 raft_consensus.cc:2454] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 1.
I20260809 15:45:47.327023 23594 raft_consensus.cc:374] T c0741ab44d99439abc3aa3bdd27ee160 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:47.333362 23655 raft_consensus.cc:482] T 2db057fd10e149a9895dff6ef8c9f038 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:45:47.333480 23351 leader_election.cc:302] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c
I20260809 15:45:47.333830 23655 raft_consensus.cc:504] T 2db057fd10e149a9895dff6ef8c9f038 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:47.335952 23655 leader_election.cc:288] T 2db057fd10e149a9895dff6ef8c9f038 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:45:47.336184 23593 tablet_bootstrap.cc:492] T c0741ab44d99439abc3aa3bdd27ee160 P 65c36d164b0540f099d592f38307ee6c: No bootstrap required, opened a new log
I20260809 15:45:47.336670 23593 ts_tablet_manager.cc:1397] T c0741ab44d99439abc3aa3bdd27ee160 P 65c36d164b0540f099d592f38307ee6c: Time spent bootstrapping tablet: real 0.048s user 0.015s sys 0.005s
I20260809 15:45:47.338277 23602 raft_consensus.cc:2781] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:45:47.338402 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2db057fd10e149a9895dff6ef8c9f038" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:45:47.338748 23602 raft_consensus.cc:686] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d [term 1 LEADER]: Becoming Leader. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:45:47.339202 23593 raft_consensus.cc:348] T c0741ab44d99439abc3aa3bdd27ee160 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:45:47.339931 23472 leader_election.cc:341] T 2db057fd10e149a9895dff6ef8c9f038 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:47.339566 23602 consensus_queue.cc:237] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:47.339916 23593 raft_consensus.cc:374] T c0741ab44d99439abc3aa3bdd27ee160 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:47.340791 23593 raft_consensus.cc:729] T c0741ab44d99439abc3aa3bdd27ee160 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Initialized, Role: FOLLOWER
I20260809 15:45:47.339900 23117 catalog_manager.cc:5495] T 16ca412a0dae4feebe85dd01bf1a9173 P 67b3aeadfa7a432abe8af4d292e04ab9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65). New cstate: current_term: 1 leader_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } health_report { overall_health: UNKNOWN } } }
I20260809 15:45:47.341578 23593 consensus_queue.cc:260] T c0741ab44d99439abc3aa3bdd27ee160 P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:47.342664 23285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2db057fd10e149a9895dff6ef8c9f038" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:45:47.343312 23285 raft_consensus.cc:2454] T 2db057fd10e149a9895dff6ef8c9f038 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 0.
I20260809 15:45:47.344059 23593 ts_tablet_manager.cc:1428] T c0741ab44d99439abc3aa3bdd27ee160 P 65c36d164b0540f099d592f38307ee6c: Time spent starting tablet: real 0.007s user 0.006s sys 0.001s
I20260809 15:45:47.344127 23591 tablet_bootstrap.cc:654] T a3622f6e3e3641cea1960e3a31498bb7 P c0f38c1908414566aa9115bb2cbdd85d: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:47.344255 23470 leader_election.cc:302] T 2db057fd10e149a9895dff6ef8c9f038 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters: c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:45:47.345258 23593 tablet_bootstrap.cc:492] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 65c36d164b0540f099d592f38307ee6c: Bootstrap starting.
I20260809 15:45:47.345571 23655 raft_consensus.cc:2781] T 2db057fd10e149a9895dff6ef8c9f038 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 15:45:47.345918 23655 raft_consensus.cc:482] T 2db057fd10e149a9895dff6ef8c9f038 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:45:47.346239 23655 raft_consensus.cc:3037] T 2db057fd10e149a9895dff6ef8c9f038 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:47.351480 23593 tablet_bootstrap.cc:654] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 65c36d164b0540f099d592f38307ee6c: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:47.353355 23655 raft_consensus.cc:504] T 2db057fd10e149a9895dff6ef8c9f038 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:47.333168 23594 raft_consensus.cc:729] T c0741ab44d99439abc3aa3bdd27ee160 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Initialized, Role: FOLLOWER
I20260809 15:45:47.355881 23591 tablet_bootstrap.cc:492] T a3622f6e3e3641cea1960e3a31498bb7 P c0f38c1908414566aa9115bb2cbdd85d: No bootstrap required, opened a new log
I20260809 15:45:47.356369 23591 ts_tablet_manager.cc:1397] T a3622f6e3e3641cea1960e3a31498bb7 P c0f38c1908414566aa9115bb2cbdd85d: Time spent bootstrapping tablet: real 0.026s user 0.011s sys 0.006s
I20260809 15:45:47.357302 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2db057fd10e149a9895dff6ef8c9f038" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
W20260809 15:45:47.358503 23472 leader_election.cc:341] T 2db057fd10e149a9895dff6ef8c9f038 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:47.359001 23655 leader_election.cc:288] T 2db057fd10e149a9895dff6ef8c9f038 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 1 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:45:47.359094 23591 raft_consensus.cc:348] T a3622f6e3e3641cea1960e3a31498bb7 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:47.359895 23591 raft_consensus.cc:374] T a3622f6e3e3641cea1960e3a31498bb7 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:47.360176 23591 raft_consensus.cc:729] T a3622f6e3e3641cea1960e3a31498bb7 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Initialized, Role: FOLLOWER
I20260809 15:45:47.360831 23591 consensus_queue.cc:260] T a3622f6e3e3641cea1960e3a31498bb7 P c0f38c1908414566aa9115bb2cbdd85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:47.355888 23285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2db057fd10e149a9895dff6ef8c9f038" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:45:47.355888 23594 consensus_queue.cc:260] T c0741ab44d99439abc3aa3bdd27ee160 P 67b3aeadfa7a432abe8af4d292e04ab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:47.363946 23285 raft_consensus.cc:3037] T 2db057fd10e149a9895dff6ef8c9f038 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:47.363109 23591 ts_tablet_manager.cc:1428] T a3622f6e3e3641cea1960e3a31498bb7 P c0f38c1908414566aa9115bb2cbdd85d: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20260809 15:45:47.365108 23591 tablet_bootstrap.cc:492] T 2db057fd10e149a9895dff6ef8c9f038 P c0f38c1908414566aa9115bb2cbdd85d: Bootstrap starting.
I20260809 15:45:47.370653 23285 raft_consensus.cc:2454] T 2db057fd10e149a9895dff6ef8c9f038 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 1.
I20260809 15:45:47.371737 23591 tablet_bootstrap.cc:654] T 2db057fd10e149a9895dff6ef8c9f038 P c0f38c1908414566aa9115bb2cbdd85d: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:47.372187 23470 leader_election.cc:302] T 2db057fd10e149a9895dff6ef8c9f038 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters: c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:45:47.373185 23655 raft_consensus.cc:2781] T 2db057fd10e149a9895dff6ef8c9f038 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:45:47.373307 23594 ts_tablet_manager.cc:1428] T c0741ab44d99439abc3aa3bdd27ee160 P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent starting tablet: real 0.057s user 0.005s sys 0.002s
I20260809 15:45:47.373652 23655 raft_consensus.cc:686] T 2db057fd10e149a9895dff6ef8c9f038 P 65c36d164b0540f099d592f38307ee6c [term 1 LEADER]: Becoming Leader. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:45:47.374398 23594 tablet_bootstrap.cc:492] T 953eea268e17491ca76f3a555860a512 P 67b3aeadfa7a432abe8af4d292e04ab9: Bootstrap starting.
I20260809 15:45:47.374426 23655 consensus_queue.cc:237] T 2db057fd10e149a9895dff6ef8c9f038 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:47.376573 23593 tablet_bootstrap.cc:492] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 65c36d164b0540f099d592f38307ee6c: No bootstrap required, opened a new log
I20260809 15:45:47.377041 23593 ts_tablet_manager.cc:1397] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 65c36d164b0540f099d592f38307ee6c: Time spent bootstrapping tablet: real 0.032s user 0.010s sys 0.015s
I20260809 15:45:47.379686 23593 raft_consensus.cc:348] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:47.380359 23593 raft_consensus.cc:374] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:47.380587 23594 tablet_bootstrap.cc:654] T 953eea268e17491ca76f3a555860a512 P 67b3aeadfa7a432abe8af4d292e04ab9: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:47.380645 23593 raft_consensus.cc:729] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Initialized, Role: FOLLOWER
I20260809 15:45:47.381587 23593 consensus_queue.cc:260] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:47.382622 23591 tablet_bootstrap.cc:492] T 2db057fd10e149a9895dff6ef8c9f038 P c0f38c1908414566aa9115bb2cbdd85d: No bootstrap required, opened a new log
I20260809 15:45:47.383035 23591 ts_tablet_manager.cc:1397] T 2db057fd10e149a9895dff6ef8c9f038 P c0f38c1908414566aa9115bb2cbdd85d: Time spent bootstrapping tablet: real 0.018s user 0.012s sys 0.003s
I20260809 15:45:47.383755 23643 raft_consensus.cc:482] T c0741ab44d99439abc3aa3bdd27ee160 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:45:47.384191 23643 raft_consensus.cc:504] T c0741ab44d99439abc3aa3bdd27ee160 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:47.384438 23116 catalog_manager.cc:5495] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d reported cstate change: term changed from 0 to 1, leader changed from <none> to c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66). New cstate: current_term: 1 leader_uuid: "c0f38c1908414566aa9115bb2cbdd85d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } health_report { overall_health: UNKNOWN } } }
I20260809 15:45:47.385628 23591 raft_consensus.cc:348] T 2db057fd10e149a9895dff6ef8c9f038 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:47.386080 23643 leader_election.cc:288] T c0741ab44d99439abc3aa3bdd27ee160 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:45:47.386588 23591 raft_consensus.cc:374] T 2db057fd10e149a9895dff6ef8c9f038 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:47.386927 23591 raft_consensus.cc:729] T 2db057fd10e149a9895dff6ef8c9f038 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Initialized, Role: FOLLOWER
I20260809 15:45:47.387436 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0741ab44d99439abc3aa3bdd27ee160" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:45:47.387681 23591 consensus_queue.cc:260] T 2db057fd10e149a9895dff6ef8c9f038 P c0f38c1908414566aa9115bb2cbdd85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:45:47.388760 23472 leader_election.cc:341] T c0741ab44d99439abc3aa3bdd27ee160 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:47.389978 23591 ts_tablet_manager.cc:1428] T 2db057fd10e149a9895dff6ef8c9f038 P c0f38c1908414566aa9115bb2cbdd85d: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260809 15:45:47.390174 23285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0741ab44d99439abc3aa3bdd27ee160" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:45:47.390728 23285 raft_consensus.cc:2454] T c0741ab44d99439abc3aa3bdd27ee160 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 0.
I20260809 15:45:47.390908 23591 tablet_bootstrap.cc:492] T cef3e7277d7d4760824a3436ed4a847d P c0f38c1908414566aa9115bb2cbdd85d: Bootstrap starting.
I20260809 15:45:47.391827 23470 leader_election.cc:302] T c0741ab44d99439abc3aa3bdd27ee160 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters: c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:45:47.392830 23643 raft_consensus.cc:2781] T c0741ab44d99439abc3aa3bdd27ee160 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 15:45:47.393170 23643 raft_consensus.cc:482] T c0741ab44d99439abc3aa3bdd27ee160 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:45:47.393489 23643 raft_consensus.cc:3037] T c0741ab44d99439abc3aa3bdd27ee160 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:47.397637 23591 tablet_bootstrap.cc:654] T cef3e7277d7d4760824a3436ed4a847d P c0f38c1908414566aa9115bb2cbdd85d: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:47.401553 23643 raft_consensus.cc:504] T c0741ab44d99439abc3aa3bdd27ee160 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:47.404151 23285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0741ab44d99439abc3aa3bdd27ee160" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:45:47.404703 23285 raft_consensus.cc:3037] T c0741ab44d99439abc3aa3bdd27ee160 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:47.404809 23594 tablet_bootstrap.cc:492] T 953eea268e17491ca76f3a555860a512 P 67b3aeadfa7a432abe8af4d292e04ab9: No bootstrap required, opened a new log
I20260809 15:45:47.409070 23594 ts_tablet_manager.cc:1397] T 953eea268e17491ca76f3a555860a512 P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent bootstrapping tablet: real 0.035s user 0.011s sys 0.012s
I20260809 15:45:47.411080 23285 raft_consensus.cc:2454] T c0741ab44d99439abc3aa3bdd27ee160 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 1.
I20260809 15:45:47.411798 23594 raft_consensus.cc:348] T 953eea268e17491ca76f3a555860a512 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:47.412173 23470 leader_election.cc:302] T c0741ab44d99439abc3aa3bdd27ee160 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:45:47.415088 23594 raft_consensus.cc:374] T 953eea268e17491ca76f3a555860a512 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:47.415232 23593 ts_tablet_manager.cc:1428] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 65c36d164b0540f099d592f38307ee6c: Time spent starting tablet: real 0.038s user 0.007s sys 0.000s
I20260809 15:45:47.415491 23594 raft_consensus.cc:729] T 953eea268e17491ca76f3a555860a512 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Initialized, Role: FOLLOWER
I20260809 15:45:47.415836 23601 raft_consensus.cc:2781] T c0741ab44d99439abc3aa3bdd27ee160 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:45:47.416312 23593 tablet_bootstrap.cc:492] T 953eea268e17491ca76f3a555860a512 P 65c36d164b0540f099d592f38307ee6c: Bootstrap starting.
I20260809 15:45:47.416409 23594 consensus_queue.cc:260] T 953eea268e17491ca76f3a555860a512 P 67b3aeadfa7a432abe8af4d292e04ab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:47.419809 23117 catalog_manager.cc:5495] T 2db057fd10e149a9895dff6ef8c9f038 P 65c36d164b0540f099d592f38307ee6c reported cstate change: term changed from 0 to 1, leader changed from <none> to 65c36d164b0540f099d592f38307ee6c (127.20.202.67). New cstate: current_term: 1 leader_uuid: "65c36d164b0540f099d592f38307ee6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } health_report { overall_health: HEALTHY } } }
I20260809 15:45:47.422336 23593 tablet_bootstrap.cc:654] T 953eea268e17491ca76f3a555860a512 P 65c36d164b0540f099d592f38307ee6c: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:47.424721 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0741ab44d99439abc3aa3bdd27ee160" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:45:47.425441 23594 ts_tablet_manager.cc:1428] T 953eea268e17491ca76f3a555860a512 P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent starting tablet: real 0.016s user 0.003s sys 0.003s
W20260809 15:45:47.426038 23472 leader_election.cc:341] T c0741ab44d99439abc3aa3bdd27ee160 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:47.426698 23643 leader_election.cc:288] T c0741ab44d99439abc3aa3bdd27ee160 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 1 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:45:47.430188 23594 tablet_bootstrap.cc:492] T a3622f6e3e3641cea1960e3a31498bb7 P 67b3aeadfa7a432abe8af4d292e04ab9: Bootstrap starting.
I20260809 15:45:47.432648 23591 tablet_bootstrap.cc:492] T cef3e7277d7d4760824a3436ed4a847d P c0f38c1908414566aa9115bb2cbdd85d: No bootstrap required, opened a new log
I20260809 15:45:47.433069 23591 ts_tablet_manager.cc:1397] T cef3e7277d7d4760824a3436ed4a847d P c0f38c1908414566aa9115bb2cbdd85d: Time spent bootstrapping tablet: real 0.042s user 0.019s sys 0.019s
I20260809 15:45:47.434533 23601 raft_consensus.cc:686] T c0741ab44d99439abc3aa3bdd27ee160 P 65c36d164b0540f099d592f38307ee6c [term 1 LEADER]: Becoming Leader. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:45:47.435009 23591 raft_consensus.cc:348] T cef3e7277d7d4760824a3436ed4a847d P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:47.435562 23591 raft_consensus.cc:374] T cef3e7277d7d4760824a3436ed4a847d P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:47.435786 23591 raft_consensus.cc:729] T cef3e7277d7d4760824a3436ed4a847d P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Initialized, Role: FOLLOWER
I20260809 15:45:47.435390 23601 consensus_queue.cc:237] T c0741ab44d99439abc3aa3bdd27ee160 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:47.436275 23591 consensus_queue.cc:260] T cef3e7277d7d4760824a3436ed4a847d P c0f38c1908414566aa9115bb2cbdd85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:47.437638 23594 tablet_bootstrap.cc:654] T a3622f6e3e3641cea1960e3a31498bb7 P 67b3aeadfa7a432abe8af4d292e04ab9: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:47.438410 23591 ts_tablet_manager.cc:1428] T cef3e7277d7d4760824a3436ed4a847d P c0f38c1908414566aa9115bb2cbdd85d: Time spent starting tablet: real 0.005s user 0.001s sys 0.005s
I20260809 15:45:47.438966 23691 raft_consensus.cc:482] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:45:47.441097 23593 tablet_bootstrap.cc:492] T 953eea268e17491ca76f3a555860a512 P 65c36d164b0540f099d592f38307ee6c: No bootstrap required, opened a new log
I20260809 15:45:47.441558 23593 ts_tablet_manager.cc:1397] T 953eea268e17491ca76f3a555860a512 P 65c36d164b0540f099d592f38307ee6c: Time spent bootstrapping tablet: real 0.025s user 0.006s sys 0.005s
I20260809 15:45:47.443084 23594 tablet_bootstrap.cc:492] T a3622f6e3e3641cea1960e3a31498bb7 P 67b3aeadfa7a432abe8af4d292e04ab9: No bootstrap required, opened a new log
I20260809 15:45:47.439443 23691 raft_consensus.cc:504] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:47.443625 23594 ts_tablet_manager.cc:1397] T a3622f6e3e3641cea1960e3a31498bb7 P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.006s
I20260809 15:45:47.443745 23593 raft_consensus.cc:348] T 953eea268e17491ca76f3a555860a512 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:47.444448 23593 raft_consensus.cc:374] T 953eea268e17491ca76f3a555860a512 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:47.444717 23593 raft_consensus.cc:729] T 953eea268e17491ca76f3a555860a512 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Initialized, Role: FOLLOWER
I20260809 15:45:47.445411 23285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad8a3b25d7b49e5a2142d5709edd0c7" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:45:47.445348 23593 consensus_queue.cc:260] T 953eea268e17491ca76f3a555860a512 P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:47.445709 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad8a3b25d7b49e5a2142d5709edd0c7" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:45:47.446208 23285 raft_consensus.cc:2379] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c in current term 1: Already voted for candidate c0f38c1908414566aa9115bb2cbdd85d in this term.
I20260809 15:45:47.439358 23591 tablet_bootstrap.cc:492] T 16ca412a0dae4feebe85dd01bf1a9173 P c0f38c1908414566aa9115bb2cbdd85d: Bootstrap starting.
I20260809 15:45:47.447357 23470 leader_election.cc:302] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:45:47.448457 23655 raft_consensus.cc:3037] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:47.448843 23691 leader_election.cc:288] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:45:47.447201 23593 ts_tablet_manager.cc:1428] T 953eea268e17491ca76f3a555860a512 P 65c36d164b0540f099d592f38307ee6c: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260809 15:45:47.452124 23593 tablet_bootstrap.cc:492] T cef3e7277d7d4760824a3436ed4a847d P 65c36d164b0540f099d592f38307ee6c: Bootstrap starting.
I20260809 15:45:47.453025 23591 tablet_bootstrap.cc:654] T 16ca412a0dae4feebe85dd01bf1a9173 P c0f38c1908414566aa9115bb2cbdd85d: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:47.455713 23655 raft_consensus.cc:2726] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 15:45:47.458101 23593 tablet_bootstrap.cc:654] T cef3e7277d7d4760824a3436ed4a847d P 65c36d164b0540f099d592f38307ee6c: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:47.458369 23594 raft_consensus.cc:348] T a3622f6e3e3641cea1960e3a31498bb7 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:47.459136 23594 raft_consensus.cc:374] T a3622f6e3e3641cea1960e3a31498bb7 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:47.459513 23594 raft_consensus.cc:729] T a3622f6e3e3641cea1960e3a31498bb7 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Initialized, Role: FOLLOWER
I20260809 15:45:47.460193 23594 consensus_queue.cc:260] T a3622f6e3e3641cea1960e3a31498bb7 P 67b3aeadfa7a432abe8af4d292e04ab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:47.464823 23116 catalog_manager.cc:5495] T c0741ab44d99439abc3aa3bdd27ee160 P 65c36d164b0540f099d592f38307ee6c reported cstate change: term changed from 0 to 1, leader changed from <none> to 65c36d164b0540f099d592f38307ee6c (127.20.202.67). New cstate: current_term: 1 leader_uuid: "65c36d164b0540f099d592f38307ee6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } health_report { overall_health: HEALTHY } } }
I20260809 15:45:47.469746 23594 ts_tablet_manager.cc:1428] T a3622f6e3e3641cea1960e3a31498bb7 P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent starting tablet: real 0.026s user 0.002s sys 0.004s
I20260809 15:45:47.474521 23594 tablet_bootstrap.cc:492] T f7715d71dbac447b9351fc5eb36c5e80 P 67b3aeadfa7a432abe8af4d292e04ab9: Bootstrap starting.
I20260809 15:45:47.480508 23594 tablet_bootstrap.cc:654] T f7715d71dbac447b9351fc5eb36c5e80 P 67b3aeadfa7a432abe8af4d292e04ab9: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:47.489639 23591 tablet_bootstrap.cc:492] T 16ca412a0dae4feebe85dd01bf1a9173 P c0f38c1908414566aa9115bb2cbdd85d: No bootstrap required, opened a new log
I20260809 15:45:47.489718 23593 tablet_bootstrap.cc:492] T cef3e7277d7d4760824a3436ed4a847d P 65c36d164b0540f099d592f38307ee6c: No bootstrap required, opened a new log
I20260809 15:45:47.490146 23591 ts_tablet_manager.cc:1397] T 16ca412a0dae4feebe85dd01bf1a9173 P c0f38c1908414566aa9115bb2cbdd85d: Time spent bootstrapping tablet: real 0.051s user 0.025s sys 0.013s
I20260809 15:45:47.490146 23593 ts_tablet_manager.cc:1397] T cef3e7277d7d4760824a3436ed4a847d P 65c36d164b0540f099d592f38307ee6c: Time spent bootstrapping tablet: real 0.038s user 0.009s sys 0.028s
I20260809 15:45:47.489610 23594 tablet_bootstrap.cc:492] T f7715d71dbac447b9351fc5eb36c5e80 P 67b3aeadfa7a432abe8af4d292e04ab9: No bootstrap required, opened a new log
I20260809 15:45:47.492308 23594 ts_tablet_manager.cc:1397] T f7715d71dbac447b9351fc5eb36c5e80 P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent bootstrapping tablet: real 0.018s user 0.011s sys 0.003s
I20260809 15:45:47.492542 23593 raft_consensus.cc:348] T cef3e7277d7d4760824a3436ed4a847d P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:47.493170 23593 raft_consensus.cc:374] T cef3e7277d7d4760824a3436ed4a847d P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:47.493433 23593 raft_consensus.cc:729] T cef3e7277d7d4760824a3436ed4a847d P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Initialized, Role: FOLLOWER
I20260809 15:45:47.493319 23591 raft_consensus.cc:348] T 16ca412a0dae4feebe85dd01bf1a9173 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:47.494189 23593 consensus_queue.cc:260] T cef3e7277d7d4760824a3436ed4a847d P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:47.494764 23591 raft_consensus.cc:374] T 16ca412a0dae4feebe85dd01bf1a9173 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:47.494693 23594 raft_consensus.cc:348] T f7715d71dbac447b9351fc5eb36c5e80 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:47.495194 23591 raft_consensus.cc:729] T 16ca412a0dae4feebe85dd01bf1a9173 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Initialized, Role: FOLLOWER
I20260809 15:45:47.495361 23594 raft_consensus.cc:374] T f7715d71dbac447b9351fc5eb36c5e80 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:47.495673 23594 raft_consensus.cc:729] T f7715d71dbac447b9351fc5eb36c5e80 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Initialized, Role: FOLLOWER
I20260809 15:45:47.496376 23591 consensus_queue.cc:260] T 16ca412a0dae4feebe85dd01bf1a9173 P c0f38c1908414566aa9115bb2cbdd85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:47.496239 23594 consensus_queue.cc:260] T f7715d71dbac447b9351fc5eb36c5e80 P 67b3aeadfa7a432abe8af4d292e04ab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:47.498172 23594 ts_tablet_manager.cc:1428] T f7715d71dbac447b9351fc5eb36c5e80 P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260809 15:45:47.498986 23594 tablet_bootstrap.cc:492] T cef3e7277d7d4760824a3436ed4a847d P 67b3aeadfa7a432abe8af4d292e04ab9: Bootstrap starting.
I20260809 15:45:47.502368 23593 ts_tablet_manager.cc:1428] T cef3e7277d7d4760824a3436ed4a847d P 65c36d164b0540f099d592f38307ee6c: Time spent starting tablet: real 0.012s user 0.005s sys 0.000s
I20260809 15:45:47.503775 23594 tablet_bootstrap.cc:654] T cef3e7277d7d4760824a3436ed4a847d P 67b3aeadfa7a432abe8af4d292e04ab9: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:47.504148 23591 ts_tablet_manager.cc:1428] T 16ca412a0dae4feebe85dd01bf1a9173 P c0f38c1908414566aa9115bb2cbdd85d: Time spent starting tablet: real 0.014s user 0.006s sys 0.000s
I20260809 15:45:47.504967 23591 tablet_bootstrap.cc:492] T 0c48a3049897452383257aa757834fb5 P c0f38c1908414566aa9115bb2cbdd85d: Bootstrap starting.
I20260809 15:45:47.506474 23593 tablet_bootstrap.cc:492] T a3622f6e3e3641cea1960e3a31498bb7 P 65c36d164b0540f099d592f38307ee6c: Bootstrap starting.
I20260809 15:45:47.510135 23591 tablet_bootstrap.cc:654] T 0c48a3049897452383257aa757834fb5 P c0f38c1908414566aa9115bb2cbdd85d: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:47.511682 23593 tablet_bootstrap.cc:654] T a3622f6e3e3641cea1960e3a31498bb7 P 65c36d164b0540f099d592f38307ee6c: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:47.516539 23594 tablet_bootstrap.cc:492] T cef3e7277d7d4760824a3436ed4a847d P 67b3aeadfa7a432abe8af4d292e04ab9: No bootstrap required, opened a new log
I20260809 15:45:47.516939 23594 ts_tablet_manager.cc:1397] T cef3e7277d7d4760824a3436ed4a847d P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent bootstrapping tablet: real 0.018s user 0.012s sys 0.004s
I20260809 15:45:47.519047 23594 raft_consensus.cc:348] T cef3e7277d7d4760824a3436ed4a847d P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:47.519685 23594 raft_consensus.cc:374] T cef3e7277d7d4760824a3436ed4a847d P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:47.519938 23594 raft_consensus.cc:729] T cef3e7277d7d4760824a3436ed4a847d P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Initialized, Role: FOLLOWER
I20260809 15:45:47.520735 23593 tablet_bootstrap.cc:492] T a3622f6e3e3641cea1960e3a31498bb7 P 65c36d164b0540f099d592f38307ee6c: No bootstrap required, opened a new log
I20260809 15:45:47.520495 23594 consensus_queue.cc:260] T cef3e7277d7d4760824a3436ed4a847d P 67b3aeadfa7a432abe8af4d292e04ab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:47.521054 23591 tablet_bootstrap.cc:492] T 0c48a3049897452383257aa757834fb5 P c0f38c1908414566aa9115bb2cbdd85d: No bootstrap required, opened a new log
I20260809 15:45:47.521107 23593 ts_tablet_manager.cc:1397] T a3622f6e3e3641cea1960e3a31498bb7 P 65c36d164b0540f099d592f38307ee6c: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.003s
I20260809 15:45:47.521549 23591 ts_tablet_manager.cc:1397] T 0c48a3049897452383257aa757834fb5 P c0f38c1908414566aa9115bb2cbdd85d: Time spent bootstrapping tablet: real 0.017s user 0.014s sys 0.002s
I20260809 15:45:47.522382 23594 ts_tablet_manager.cc:1428] T cef3e7277d7d4760824a3436ed4a847d P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260809 15:45:47.523242 23594 tablet_bootstrap.cc:492] T 0c48a3049897452383257aa757834fb5 P 67b3aeadfa7a432abe8af4d292e04ab9: Bootstrap starting.
I20260809 15:45:47.523474 23593 raft_consensus.cc:348] T a3622f6e3e3641cea1960e3a31498bb7 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:47.524080 23593 raft_consensus.cc:374] T a3622f6e3e3641cea1960e3a31498bb7 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:47.523794 23591 raft_consensus.cc:348] T 0c48a3049897452383257aa757834fb5 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:47.524320 23593 raft_consensus.cc:729] T a3622f6e3e3641cea1960e3a31498bb7 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Initialized, Role: FOLLOWER
I20260809 15:45:47.524451 23591 raft_consensus.cc:374] T 0c48a3049897452383257aa757834fb5 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:47.524744 23591 raft_consensus.cc:729] T 0c48a3049897452383257aa757834fb5 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Initialized, Role: FOLLOWER
I20260809 15:45:47.525036 23593 consensus_queue.cc:260] T a3622f6e3e3641cea1960e3a31498bb7 P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:47.525394 23591 consensus_queue.cc:260] T 0c48a3049897452383257aa757834fb5 P c0f38c1908414566aa9115bb2cbdd85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:47.528826 23594 tablet_bootstrap.cc:654] T 0c48a3049897452383257aa757834fb5 P 67b3aeadfa7a432abe8af4d292e04ab9: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:47.530877 23591 ts_tablet_manager.cc:1428] T 0c48a3049897452383257aa757834fb5 P c0f38c1908414566aa9115bb2cbdd85d: Time spent starting tablet: real 0.009s user 0.004s sys 0.000s
I20260809 15:45:47.531735 23591 tablet_bootstrap.cc:492] T c0741ab44d99439abc3aa3bdd27ee160 P c0f38c1908414566aa9115bb2cbdd85d: Bootstrap starting.
I20260809 15:45:47.532572 23593 ts_tablet_manager.cc:1428] T a3622f6e3e3641cea1960e3a31498bb7 P 65c36d164b0540f099d592f38307ee6c: Time spent starting tablet: real 0.011s user 0.004s sys 0.000s
I20260809 15:45:47.533484 23593 tablet_bootstrap.cc:492] T ba56725e42f74a1baa4c04d9660727f9 P 65c36d164b0540f099d592f38307ee6c: Bootstrap starting.
I20260809 15:45:47.536365 23591 tablet_bootstrap.cc:654] T c0741ab44d99439abc3aa3bdd27ee160 P c0f38c1908414566aa9115bb2cbdd85d: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:47.538621 23593 tablet_bootstrap.cc:654] T ba56725e42f74a1baa4c04d9660727f9 P 65c36d164b0540f099d592f38307ee6c: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:47.543061 23594 tablet_bootstrap.cc:492] T 0c48a3049897452383257aa757834fb5 P 67b3aeadfa7a432abe8af4d292e04ab9: No bootstrap required, opened a new log
I20260809 15:45:47.543491 23594 ts_tablet_manager.cc:1397] T 0c48a3049897452383257aa757834fb5 P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent bootstrapping tablet: real 0.020s user 0.001s sys 0.015s
I20260809 15:45:47.546154 23594 raft_consensus.cc:348] T 0c48a3049897452383257aa757834fb5 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:47.546783 23594 raft_consensus.cc:374] T 0c48a3049897452383257aa757834fb5 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:47.547011 23594 raft_consensus.cc:729] T 0c48a3049897452383257aa757834fb5 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Initialized, Role: FOLLOWER
I20260809 15:45:47.547780 23594 consensus_queue.cc:260] T 0c48a3049897452383257aa757834fb5 P 67b3aeadfa7a432abe8af4d292e04ab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:47.551023 23591 tablet_bootstrap.cc:492] T c0741ab44d99439abc3aa3bdd27ee160 P c0f38c1908414566aa9115bb2cbdd85d: No bootstrap required, opened a new log
I20260809 15:45:47.551412 23591 ts_tablet_manager.cc:1397] T c0741ab44d99439abc3aa3bdd27ee160 P c0f38c1908414566aa9115bb2cbdd85d: Time spent bootstrapping tablet: real 0.020s user 0.007s sys 0.008s
I20260809 15:45:47.551879 23594 ts_tablet_manager.cc:1428] T 0c48a3049897452383257aa757834fb5 P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20260809 15:45:47.552767 23594 tablet_bootstrap.cc:492] T e14d118b2eaa42aa8a7cd4025aba7128 P 67b3aeadfa7a432abe8af4d292e04ab9: Bootstrap starting.
I20260809 15:45:47.553072 23591 raft_consensus.cc:348] T c0741ab44d99439abc3aa3bdd27ee160 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:47.553537 23591 raft_consensus.cc:374] T c0741ab44d99439abc3aa3bdd27ee160 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:47.553721 23591 raft_consensus.cc:729] T c0741ab44d99439abc3aa3bdd27ee160 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Initialized, Role: FOLLOWER
I20260809 15:45:47.554214 23591 consensus_queue.cc:260] T c0741ab44d99439abc3aa3bdd27ee160 P c0f38c1908414566aa9115bb2cbdd85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:47.555836 23591 ts_tablet_manager.cc:1428] T c0741ab44d99439abc3aa3bdd27ee160 P c0f38c1908414566aa9115bb2cbdd85d: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260809 15:45:47.556741 23591 tablet_bootstrap.cc:492] T 472816994d814cb198b69e5248697440 P c0f38c1908414566aa9115bb2cbdd85d: Bootstrap starting.
I20260809 15:45:47.560848 23594 tablet_bootstrap.cc:654] T e14d118b2eaa42aa8a7cd4025aba7128 P 67b3aeadfa7a432abe8af4d292e04ab9: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:47.562683 23591 tablet_bootstrap.cc:654] T 472816994d814cb198b69e5248697440 P c0f38c1908414566aa9115bb2cbdd85d: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:47.567972 23593 tablet_bootstrap.cc:492] T ba56725e42f74a1baa4c04d9660727f9 P 65c36d164b0540f099d592f38307ee6c: No bootstrap required, opened a new log
I20260809 15:45:47.568413 23593 ts_tablet_manager.cc:1397] T ba56725e42f74a1baa4c04d9660727f9 P 65c36d164b0540f099d592f38307ee6c: Time spent bootstrapping tablet: real 0.035s user 0.015s sys 0.008s
I20260809 15:45:47.570890 23593 raft_consensus.cc:348] T ba56725e42f74a1baa4c04d9660727f9 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:47.571576 23593 raft_consensus.cc:374] T ba56725e42f74a1baa4c04d9660727f9 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:47.571820 23593 raft_consensus.cc:729] T ba56725e42f74a1baa4c04d9660727f9 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Initialized, Role: FOLLOWER
I20260809 15:45:47.572490 23593 consensus_queue.cc:260] T ba56725e42f74a1baa4c04d9660727f9 P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:47.577080 23593 ts_tablet_manager.cc:1428] T ba56725e42f74a1baa4c04d9660727f9 P 65c36d164b0540f099d592f38307ee6c: Time spent starting tablet: real 0.008s user 0.003s sys 0.003s
I20260809 15:45:47.577894 23593 tablet_bootstrap.cc:492] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 65c36d164b0540f099d592f38307ee6c: Bootstrap starting.
I20260809 15:45:47.578814 23591 tablet_bootstrap.cc:492] T 472816994d814cb198b69e5248697440 P c0f38c1908414566aa9115bb2cbdd85d: No bootstrap required, opened a new log
I20260809 15:45:47.578835 23594 tablet_bootstrap.cc:492] T e14d118b2eaa42aa8a7cd4025aba7128 P 67b3aeadfa7a432abe8af4d292e04ab9: No bootstrap required, opened a new log
I20260809 15:45:47.579476 23591 ts_tablet_manager.cc:1397] T 472816994d814cb198b69e5248697440 P c0f38c1908414566aa9115bb2cbdd85d: Time spent bootstrapping tablet: real 0.023s user 0.010s sys 0.012s
I20260809 15:45:47.579507 23594 ts_tablet_manager.cc:1397] T e14d118b2eaa42aa8a7cd4025aba7128 P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent bootstrapping tablet: real 0.027s user 0.013s sys 0.010s
I20260809 15:45:47.582037 23591 raft_consensus.cc:348] T 472816994d814cb198b69e5248697440 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:47.582547 23591 raft_consensus.cc:374] T 472816994d814cb198b69e5248697440 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:47.582830 23591 raft_consensus.cc:729] T 472816994d814cb198b69e5248697440 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Initialized, Role: FOLLOWER
I20260809 15:45:47.582651 23594 raft_consensus.cc:348] T e14d118b2eaa42aa8a7cd4025aba7128 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:47.583361 23594 raft_consensus.cc:374] T e14d118b2eaa42aa8a7cd4025aba7128 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:47.583338 23604 raft_consensus.cc:482] T 0c48a3049897452383257aa757834fb5 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:45:47.583534 23591 consensus_queue.cc:260] T 472816994d814cb198b69e5248697440 P c0f38c1908414566aa9115bb2cbdd85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:47.583964 23604 raft_consensus.cc:504] T 0c48a3049897452383257aa757834fb5 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:47.585915 23604 leader_election.cc:288] T 0c48a3049897452383257aa757834fb5 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:45:47.586414 23593 tablet_bootstrap.cc:654] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 65c36d164b0540f099d592f38307ee6c: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:47.588177 23591 ts_tablet_manager.cc:1428] T 472816994d814cb198b69e5248697440 P c0f38c1908414566aa9115bb2cbdd85d: Time spent starting tablet: real 0.008s user 0.004s sys 0.002s
I20260809 15:45:47.588299 23529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c48a3049897452383257aa757834fb5" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:45:47.588761 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c48a3049897452383257aa757834fb5" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:45:47.589118 23591 tablet_bootstrap.cc:492] T ba56725e42f74a1baa4c04d9660727f9 P c0f38c1908414566aa9115bb2cbdd85d: Bootstrap starting.
I20260809 15:45:47.589278 23529 raft_consensus.cc:2454] T 0c48a3049897452383257aa757834fb5 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 0.
I20260809 15:45:47.589396 23402 raft_consensus.cc:2454] T 0c48a3049897452383257aa757834fb5 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 0.
I20260809 15:45:47.590556 23228 leader_election.cc:302] T 0c48a3049897452383257aa757834fb5 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:45:47.592058 23604 raft_consensus.cc:2781] T 0c48a3049897452383257aa757834fb5 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 15:45:47.592531 23604 raft_consensus.cc:482] T 0c48a3049897452383257aa757834fb5 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:45:47.593012 23604 raft_consensus.cc:3037] T 0c48a3049897452383257aa757834fb5 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:47.595104 23594 raft_consensus.cc:729] T e14d118b2eaa42aa8a7cd4025aba7128 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Initialized, Role: FOLLOWER
I20260809 15:45:47.595765 23594 consensus_queue.cc:260] T e14d118b2eaa42aa8a7cd4025aba7128 P 67b3aeadfa7a432abe8af4d292e04ab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:47.597651 23594 ts_tablet_manager.cc:1428] T e14d118b2eaa42aa8a7cd4025aba7128 P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent starting tablet: real 0.017s user 0.004s sys 0.000s
I20260809 15:45:47.598212 23593 tablet_bootstrap.cc:492] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 65c36d164b0540f099d592f38307ee6c: No bootstrap required, opened a new log
I20260809 15:45:47.598429 23594 tablet_bootstrap.cc:492] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 67b3aeadfa7a432abe8af4d292e04ab9: Bootstrap starting.
I20260809 15:45:47.598582 23593 ts_tablet_manager.cc:1397] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 65c36d164b0540f099d592f38307ee6c: Time spent bootstrapping tablet: real 0.021s user 0.008s sys 0.006s
I20260809 15:45:47.594936 23591 tablet_bootstrap.cc:654] T ba56725e42f74a1baa4c04d9660727f9 P c0f38c1908414566aa9115bb2cbdd85d: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:47.601946 23593 raft_consensus.cc:348] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:47.602564 23593 raft_consensus.cc:374] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:47.602788 23593 raft_consensus.cc:729] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Initialized, Role: FOLLOWER
I20260809 15:45:47.603297 23594 tablet_bootstrap.cc:654] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 67b3aeadfa7a432abe8af4d292e04ab9: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:47.603526 23593 consensus_queue.cc:260] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:47.608763 23593 ts_tablet_manager.cc:1428] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 65c36d164b0540f099d592f38307ee6c: Time spent starting tablet: real 0.010s user 0.005s sys 0.001s
I20260809 15:45:47.609573 23593 tablet_bootstrap.cc:492] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 65c36d164b0540f099d592f38307ee6c: Bootstrap starting.
I20260809 15:45:47.611553 23285 raft_consensus.cc:3037] T f7715d71dbac447b9351fc5eb36c5e80 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:47.615103 23604 raft_consensus.cc:504] T 0c48a3049897452383257aa757834fb5 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:47.617601 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c48a3049897452383257aa757834fb5" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:45:47.618114 23402 raft_consensus.cc:3037] T 0c48a3049897452383257aa757834fb5 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:47.619216 23601 consensus_queue.cc:1035] T f7715d71dbac447b9351fc5eb36c5e80 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:45:47.622167 23593 tablet_bootstrap.cc:654] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 65c36d164b0540f099d592f38307ee6c: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:47.624372 23402 raft_consensus.cc:2454] T 0c48a3049897452383257aa757834fb5 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 1.
I20260809 15:45:47.625514 23228 leader_election.cc:302] T 0c48a3049897452383257aa757834fb5 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:45:47.626551 23661 raft_consensus.cc:2781] T 0c48a3049897452383257aa757834fb5 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:45:47.627491 23594 tablet_bootstrap.cc:492] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 67b3aeadfa7a432abe8af4d292e04ab9: No bootstrap required, opened a new log
I20260809 15:45:47.627921 23594 ts_tablet_manager.cc:1397] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent bootstrapping tablet: real 0.030s user 0.018s sys 0.004s
I20260809 15:45:47.630271 23594 raft_consensus.cc:348] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:47.630923 23594 raft_consensus.cc:374] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:47.631173 23594 raft_consensus.cc:729] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Initialized, Role: FOLLOWER
I20260809 15:45:47.631835 23594 consensus_queue.cc:260] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 67b3aeadfa7a432abe8af4d292e04ab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:47.634142 23529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c48a3049897452383257aa757834fb5" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:45:47.634706 23529 raft_consensus.cc:3037] T 0c48a3049897452383257aa757834fb5 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:47.641101 23529 raft_consensus.cc:2454] T 0c48a3049897452383257aa757834fb5 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 1.
I20260809 15:45:47.652592 23604 leader_election.cc:288] T 0c48a3049897452383257aa757834fb5 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:45:47.662297 23601 consensus_queue.cc:1035] T f7715d71dbac447b9351fc5eb36c5e80 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:45:47.664752 23722 raft_consensus.cc:482] T a3622f6e3e3641cea1960e3a31498bb7 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:45:47.669791 23601 raft_consensus.cc:482] T cef3e7277d7d4760824a3436ed4a847d P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:45:47.670723 23601 raft_consensus.cc:504] T cef3e7277d7d4760824a3436ed4a847d P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:47.672580 23601 leader_election.cc:288] T cef3e7277d7d4760824a3436ed4a847d P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:45:47.666983 23722 raft_consensus.cc:504] T a3622f6e3e3641cea1960e3a31498bb7 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:47.674728 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cef3e7277d7d4760824a3436ed4a847d" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:45:47.675321 23402 raft_consensus.cc:2454] T cef3e7277d7d4760824a3436ed4a847d P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 0.
I20260809 15:45:47.675748 23285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cef3e7277d7d4760824a3436ed4a847d" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:45:47.676498 23285 raft_consensus.cc:2454] T cef3e7277d7d4760824a3436ed4a847d P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 0.
I20260809 15:45:47.676928 23400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3622f6e3e3641cea1960e3a31498bb7" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:45:47.677654 23722 leader_election.cc:288] T a3622f6e3e3641cea1960e3a31498bb7 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:45:47.665494 23661 raft_consensus.cc:686] T 0c48a3049897452383257aa757834fb5 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 LEADER]: Becoming Leader. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Running, Role: LEADER
I20260809 15:45:47.678580 23529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3622f6e3e3641cea1960e3a31498bb7" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:45:47.682601 23529 raft_consensus.cc:2454] T a3622f6e3e3641cea1960e3a31498bb7 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 0.
I20260809 15:45:47.679119 23472 leader_election.cc:302] T cef3e7277d7d4760824a3436ed4a847d P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:45:47.682837 23661 consensus_queue.cc:237] T 0c48a3049897452383257aa757834fb5 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:47.678134 23400 raft_consensus.cc:2454] T a3622f6e3e3641cea1960e3a31498bb7 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 0.
I20260809 15:45:47.683745 23593 tablet_bootstrap.cc:492] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 65c36d164b0540f099d592f38307ee6c: No bootstrap required, opened a new log
I20260809 15:45:47.685298 23593 ts_tablet_manager.cc:1397] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 65c36d164b0540f099d592f38307ee6c: Time spent bootstrapping tablet: real 0.076s user 0.035s sys 0.009s
I20260809 15:45:47.680073 23601 raft_consensus.cc:482] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:45:47.687197 23722 raft_consensus.cc:482] T e14d118b2eaa42aa8a7cd4025aba7128 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:45:47.683818 23228 leader_election.cc:302] T a3622f6e3e3641cea1960e3a31498bb7 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:45:47.686496 23602 raft_consensus.cc:482] T 2db057fd10e149a9895dff6ef8c9f038 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:45:47.686980 23601 raft_consensus.cc:504] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:47.684355 23691 raft_consensus.cc:2781] T cef3e7277d7d4760824a3436ed4a847d P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 15:45:47.680776 23591 tablet_bootstrap.cc:492] T ba56725e42f74a1baa4c04d9660727f9 P c0f38c1908414566aa9115bb2cbdd85d: No bootstrap required, opened a new log
I20260809 15:45:47.689296 23594 ts_tablet_manager.cc:1428] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent starting tablet: real 0.061s user 0.018s sys 0.036s
I20260809 15:45:47.690001 23591 ts_tablet_manager.cc:1397] T ba56725e42f74a1baa4c04d9660727f9 P c0f38c1908414566aa9115bb2cbdd85d: Time spent bootstrapping tablet: real 0.101s user 0.018s sys 0.012s
I20260809 15:45:47.688640 23722 raft_consensus.cc:504] T e14d118b2eaa42aa8a7cd4025aba7128 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:47.689611 23691 raft_consensus.cc:482] T cef3e7277d7d4760824a3436ed4a847d P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:45:47.689584 23602 raft_consensus.cc:504] T 2db057fd10e149a9895dff6ef8c9f038 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:47.691570 23691 raft_consensus.cc:3037] T cef3e7277d7d4760824a3436ed4a847d P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:47.692961 23602 leader_election.cc:288] T 2db057fd10e149a9895dff6ef8c9f038 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:45:47.693629 23400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e14d118b2eaa42aa8a7cd4025aba7128" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:45:47.695633 23594 tablet_bootstrap.cc:492] T 05ad7d8fdca64e3da527280ca5be9280 P 67b3aeadfa7a432abe8af4d292e04ab9: Bootstrap starting.
I20260809 15:45:47.694105 23604 raft_consensus.cc:2781] T a3622f6e3e3641cea1960e3a31498bb7 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 15:45:47.696643 23604 raft_consensus.cc:482] T a3622f6e3e3641cea1960e3a31498bb7 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:45:47.697289 23529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2db057fd10e149a9895dff6ef8c9f038" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
W20260809 15:45:47.697723 23228 leader_election.cc:341] T e14d118b2eaa42aa8a7cd4025aba7128 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:47.697422 23722 leader_election.cc:288] T e14d118b2eaa42aa8a7cd4025aba7128 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:45:47.696964 23604 raft_consensus.cc:3037] T a3622f6e3e3641cea1960e3a31498bb7 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:47.701320 23286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a823d35f4f348d1b52ba9e6d80b00d2" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:45:47.701699 23528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e14d118b2eaa42aa8a7cd4025aba7128" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:45:47.702069 23286 raft_consensus.cc:2379] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c in current term 1: Already voted for candidate c0f38c1908414566aa9115bb2cbdd85d in this term.
W20260809 15:45:47.702850 23228 leader_election.cc:341] T e14d118b2eaa42aa8a7cd4025aba7128 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:47.697858 23400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a823d35f4f348d1b52ba9e6d80b00d2" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:45:47.695891 23601 leader_election.cc:288] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:45:47.704638 23602 consensus_queue.cc:1035] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:45:47.705909 23117 catalog_manager.cc:5495] T 0c48a3049897452383257aa757834fb5 P 67b3aeadfa7a432abe8af4d292e04ab9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65). New cstate: current_term: 1 leader_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } health_report { overall_health: UNKNOWN } } }
I20260809 15:45:47.707803 23285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2db057fd10e149a9895dff6ef8c9f038" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:45:47.708508 23285 raft_consensus.cc:2379] T 2db057fd10e149a9895dff6ef8c9f038 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d in current term 1: Already voted for candidate 65c36d164b0540f099d592f38307ee6c in this term.
I20260809 15:45:47.706864 23691 raft_consensus.cc:504] T cef3e7277d7d4760824a3436ed4a847d P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:47.709120 23472 leader_election.cc:302] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:45:47.703280 23228 leader_election.cc:302] T e14d118b2eaa42aa8a7cd4025aba7128 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:45:47.711084 23601 raft_consensus.cc:3037] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:47.711045 23593 raft_consensus.cc:348] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:47.711805 23593 raft_consensus.cc:374] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:47.712054 23593 raft_consensus.cc:729] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Initialized, Role: FOLLOWER
I20260809 15:45:47.712622 23593 consensus_queue.cc:260] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:47.715548 23722 raft_consensus.cc:2726] T e14d118b2eaa42aa8a7cd4025aba7128 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 15:45:47.716868 23655 raft_consensus.cc:482] T 953eea268e17491ca76f3a555860a512 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:45:47.717279 23655 raft_consensus.cc:504] T 953eea268e17491ca76f3a555860a512 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:47.718585 23601 raft_consensus.cc:2726] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 15:45:47.718910 23591 raft_consensus.cc:348] T ba56725e42f74a1baa4c04d9660727f9 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:47.719580 23591 raft_consensus.cc:374] T ba56725e42f74a1baa4c04d9660727f9 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:47.719816 23591 raft_consensus.cc:729] T ba56725e42f74a1baa4c04d9660727f9 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Initialized, Role: FOLLOWER
I20260809 15:45:47.719167 23655 leader_election.cc:288] T 953eea268e17491ca76f3a555860a512 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:45:47.721050 23351 leader_election.cc:302] T 2db057fd10e149a9895dff6ef8c9f038 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:45:47.721428 23691 leader_election.cc:288] T cef3e7277d7d4760824a3436ed4a847d P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 1 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:45:47.721776 23602 raft_consensus.cc:3037] T 2db057fd10e149a9895dff6ef8c9f038 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:47.715013 23604 raft_consensus.cc:504] T a3622f6e3e3641cea1960e3a31498bb7 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:47.721357 23593 ts_tablet_manager.cc:1428] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 65c36d164b0540f099d592f38307ee6c: Time spent starting tablet: real 0.035s user 0.005s sys 0.000s
I20260809 15:45:47.720254 23591 consensus_queue.cc:260] T ba56725e42f74a1baa4c04d9660727f9 P c0f38c1908414566aa9115bb2cbdd85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:47.725075 23593 tablet_bootstrap.cc:492] T 472816994d814cb198b69e5248697440 P 65c36d164b0540f099d592f38307ee6c: Bootstrap starting.
I20260809 15:45:47.725159 23604 leader_election.cc:288] T a3622f6e3e3641cea1960e3a31498bb7 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:45:47.724196 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cef3e7277d7d4760824a3436ed4a847d" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:45:47.725749 23402 raft_consensus.cc:3037] T cef3e7277d7d4760824a3436ed4a847d P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:47.725958 23399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3622f6e3e3641cea1960e3a31498bb7" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:45:47.726534 23399 raft_consensus.cc:3037] T a3622f6e3e3641cea1960e3a31498bb7 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:47.726440 23400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "953eea268e17491ca76f3a555860a512" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:45:47.728868 23285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "953eea268e17491ca76f3a555860a512" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:45:47.730014 23285 raft_consensus.cc:2454] T 953eea268e17491ca76f3a555860a512 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 0.
I20260809 15:45:47.730561 23528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3622f6e3e3641cea1960e3a31498bb7" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:45:47.730810 23286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cef3e7277d7d4760824a3436ed4a847d" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:45:47.731104 23528 raft_consensus.cc:3037] T a3622f6e3e3641cea1960e3a31498bb7 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:47.731348 23286 raft_consensus.cc:3037] T cef3e7277d7d4760824a3436ed4a847d P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:47.732800 23602 raft_consensus.cc:2726] T 2db057fd10e149a9895dff6ef8c9f038 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 15:45:47.734130 23591 ts_tablet_manager.cc:1428] T ba56725e42f74a1baa4c04d9660727f9 P c0f38c1908414566aa9115bb2cbdd85d: Time spent starting tablet: real 0.044s user 0.005s sys 0.000s
I20260809 15:45:47.735046 23591 tablet_bootstrap.cc:492] T 2424fd9bd7f14e508fd106cbfda786a4 P c0f38c1908414566aa9115bb2cbdd85d: Bootstrap starting.
I20260809 15:45:47.738551 23470 leader_election.cc:302] T 953eea268e17491ca76f3a555860a512 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:45:47.739820 23691 raft_consensus.cc:2781] T 953eea268e17491ca76f3a555860a512 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 15:45:47.740159 23691 raft_consensus.cc:482] T 953eea268e17491ca76f3a555860a512 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:45:47.740505 23691 raft_consensus.cc:3037] T 953eea268e17491ca76f3a555860a512 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:47.727025 23400 raft_consensus.cc:2454] T 953eea268e17491ca76f3a555860a512 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 0.
I20260809 15:45:47.746299 23691 raft_consensus.cc:504] T 953eea268e17491ca76f3a555860a512 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:47.751104 23402 raft_consensus.cc:2454] T cef3e7277d7d4760824a3436ed4a847d P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 1.
I20260809 15:45:47.753245 23399 raft_consensus.cc:2454] T a3622f6e3e3641cea1960e3a31498bb7 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 1.
I20260809 15:45:47.755887 23285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "953eea268e17491ca76f3a555860a512" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:45:47.758219 23285 raft_consensus.cc:3037] T 953eea268e17491ca76f3a555860a512 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:47.757185 23528 raft_consensus.cc:2454] T a3622f6e3e3641cea1960e3a31498bb7 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 1.
I20260809 15:45:47.760751 23228 leader_election.cc:302] T a3622f6e3e3641cea1960e3a31498bb7 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:45:47.757423 23691 leader_election.cc:288] T 953eea268e17491ca76f3a555860a512 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 1 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:45:47.761863 23604 raft_consensus.cc:2781] T a3622f6e3e3641cea1960e3a31498bb7 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:45:47.762308 23604 raft_consensus.cc:686] T a3622f6e3e3641cea1960e3a31498bb7 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 LEADER]: Becoming Leader. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Running, Role: LEADER
I20260809 15:45:47.762988 23604 consensus_queue.cc:237] T a3622f6e3e3641cea1960e3a31498bb7 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:47.768906 23285 raft_consensus.cc:2454] T 953eea268e17491ca76f3a555860a512 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 1.
I20260809 15:45:47.770812 23472 leader_election.cc:302] T cef3e7277d7d4760824a3436ed4a847d P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:45:47.772092 23691 raft_consensus.cc:2781] T cef3e7277d7d4760824a3436ed4a847d P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:45:47.775575 23399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "953eea268e17491ca76f3a555860a512" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:45:47.770257 23286 raft_consensus.cc:2454] T cef3e7277d7d4760824a3436ed4a847d P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 1.
I20260809 15:45:47.776530 23399 raft_consensus.cc:3037] T 953eea268e17491ca76f3a555860a512 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:47.773382 23691 raft_consensus.cc:686] T cef3e7277d7d4760824a3436ed4a847d P 65c36d164b0540f099d592f38307ee6c [term 1 LEADER]: Becoming Leader. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:45:47.782589 23594 tablet_bootstrap.cc:654] T 05ad7d8fdca64e3da527280ca5be9280 P 67b3aeadfa7a432abe8af4d292e04ab9: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:47.784626 23470 leader_election.cc:302] T 953eea268e17491ca76f3a555860a512 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:45:47.785504 23655 raft_consensus.cc:2781] T 953eea268e17491ca76f3a555860a512 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:45:47.785948 23655 raft_consensus.cc:686] T 953eea268e17491ca76f3a555860a512 P 65c36d164b0540f099d592f38307ee6c [term 1 LEADER]: Becoming Leader. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:45:47.787436 23601 consensus_queue.cc:1035] T 2db057fd10e149a9895dff6ef8c9f038 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:45:47.787985 23655 consensus_queue.cc:237] T 953eea268e17491ca76f3a555860a512 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:47.783107 23664 consensus_queue.cc:1035] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.009s
I20260809 15:45:47.798368 23591 tablet_bootstrap.cc:654] T 2424fd9bd7f14e508fd106cbfda786a4 P c0f38c1908414566aa9115bb2cbdd85d: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:47.799394 23593 tablet_bootstrap.cc:654] T 472816994d814cb198b69e5248697440 P 65c36d164b0540f099d592f38307ee6c: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:47.784916 23691 consensus_queue.cc:237] T cef3e7277d7d4760824a3436ed4a847d P 65c36d164b0540f099d592f38307ee6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:47.808471 23664 raft_consensus.cc:482] T 472816994d814cb198b69e5248697440 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:45:47.808912 23664 raft_consensus.cc:504] T 472816994d814cb198b69e5248697440 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:47.809687 23399 raft_consensus.cc:2454] T 953eea268e17491ca76f3a555860a512 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 1.
I20260809 15:45:47.803539 23648 consensus_queue.cc:1035] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
I20260809 15:45:47.823659 23664 leader_election.cc:288] T 472816994d814cb198b69e5248697440 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:45:47.826941 23285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472816994d814cb198b69e5248697440" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
W20260809 15:45:47.833834 23351 leader_election.cc:341] T 472816994d814cb198b69e5248697440 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:47.831404 23664 raft_consensus.cc:482] T c0741ab44d99439abc3aa3bdd27ee160 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:45:47.845052 23528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472816994d814cb198b69e5248697440" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
W20260809 15:45:47.846426 23353 leader_election.cc:341] T 472816994d814cb198b69e5248697440 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:47.846866 23353 leader_election.cc:302] T 472816994d814cb198b69e5248697440 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:45:47.846972 23116 catalog_manager.cc:5495] T a3622f6e3e3641cea1960e3a31498bb7 P 67b3aeadfa7a432abe8af4d292e04ab9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65). New cstate: current_term: 1 leader_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } health_report { overall_health: UNKNOWN } } }
I20260809 15:45:47.848038 23648 raft_consensus.cc:2726] T 472816994d814cb198b69e5248697440 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 15:45:47.834764 23664 raft_consensus.cc:504] T c0741ab44d99439abc3aa3bdd27ee160 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:47.851293 23664 leader_election.cc:288] T c0741ab44d99439abc3aa3bdd27ee160 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:45:47.854913 23722 consensus_queue.cc:1035] T 16ca412a0dae4feebe85dd01bf1a9173 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:45:47.856503 23285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0741ab44d99439abc3aa3bdd27ee160" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:45:47.857223 23285 raft_consensus.cc:2379] T c0741ab44d99439abc3aa3bdd27ee160 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d in current term 1: Already voted for candidate 65c36d164b0540f099d592f38307ee6c in this term.
I20260809 15:45:47.864030 23528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0741ab44d99439abc3aa3bdd27ee160" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:45:47.865448 23353 leader_election.cc:302] T c0741ab44d99439abc3aa3bdd27ee160 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:45:47.866799 23664 raft_consensus.cc:3037] T c0741ab44d99439abc3aa3bdd27ee160 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:47.872133 23594 tablet_bootstrap.cc:492] T 05ad7d8fdca64e3da527280ca5be9280 P 67b3aeadfa7a432abe8af4d292e04ab9: No bootstrap required, opened a new log
I20260809 15:45:47.872480 23594 ts_tablet_manager.cc:1397] T 05ad7d8fdca64e3da527280ca5be9280 P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent bootstrapping tablet: real 0.177s user 0.008s sys 0.008s
I20260809 15:45:47.870245 23722 raft_consensus.cc:482] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:45:47.873279 23722 raft_consensus.cc:504] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:47.875156 23722 leader_election.cc:288] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:45:47.878454 23594 raft_consensus.cc:348] T 05ad7d8fdca64e3da527280ca5be9280 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:47.879047 23594 raft_consensus.cc:374] T 05ad7d8fdca64e3da527280ca5be9280 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:47.879318 23594 raft_consensus.cc:729] T 05ad7d8fdca64e3da527280ca5be9280 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Initialized, Role: FOLLOWER
I20260809 15:45:47.879885 23594 consensus_queue.cc:260] T 05ad7d8fdca64e3da527280ca5be9280 P 67b3aeadfa7a432abe8af4d292e04ab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:47.880142 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ced718c1a5c4da79cb8c1ebd1435b87" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:45:47.881352 23528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ced718c1a5c4da79cb8c1ebd1435b87" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:45:47.880792 23117 catalog_manager.cc:5495] T cef3e7277d7d4760824a3436ed4a847d P 65c36d164b0540f099d592f38307ee6c reported cstate change: term changed from 0 to 1, leader changed from <none> to 65c36d164b0540f099d592f38307ee6c (127.20.202.67). New cstate: current_term: 1 leader_uuid: "65c36d164b0540f099d592f38307ee6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } health_report { overall_health: HEALTHY } } }
I20260809 15:45:47.882136 23594 ts_tablet_manager.cc:1428] T 05ad7d8fdca64e3da527280ca5be9280 P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent starting tablet: real 0.009s user 0.005s sys 0.001s
W20260809 15:45:47.882509 23228 leader_election.cc:341] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:47.882958 23594 tablet_bootstrap.cc:492] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 67b3aeadfa7a432abe8af4d292e04ab9: Bootstrap starting.
W20260809 15:45:47.883168 23228 leader_election.cc:341] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:47.883603 23228 leader_election.cc:302] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:45:47.886123 23604 raft_consensus.cc:2726] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 15:45:47.889880 23593 tablet_bootstrap.cc:492] T 472816994d814cb198b69e5248697440 P 65c36d164b0540f099d592f38307ee6c: No bootstrap required, opened a new log
I20260809 15:45:47.890291 23593 ts_tablet_manager.cc:1397] T 472816994d814cb198b69e5248697440 P 65c36d164b0540f099d592f38307ee6c: Time spent bootstrapping tablet: real 0.166s user 0.013s sys 0.025s
I20260809 15:45:47.891197 23402 raft_consensus.cc:3037] T 16ca412a0dae4feebe85dd01bf1a9173 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:47.892903 23664 raft_consensus.cc:2726] T c0741ab44d99439abc3aa3bdd27ee160 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 15:45:47.892293 23117 catalog_manager.cc:5495] T 953eea268e17491ca76f3a555860a512 P 65c36d164b0540f099d592f38307ee6c reported cstate change: term changed from 0 to 1, leader changed from <none> to 65c36d164b0540f099d592f38307ee6c (127.20.202.67). New cstate: current_term: 1 leader_uuid: "65c36d164b0540f099d592f38307ee6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } health_report { overall_health: HEALTHY } } }
I20260809 15:45:47.894912 23593 raft_consensus.cc:348] T 472816994d814cb198b69e5248697440 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:47.895606 23593 raft_consensus.cc:374] T 472816994d814cb198b69e5248697440 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:47.895892 23593 raft_consensus.cc:729] T 472816994d814cb198b69e5248697440 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Initialized, Role: FOLLOWER
I20260809 15:45:47.905283 23602 consensus_queue.cc:1035] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260809 15:45:47.906615 23593 consensus_queue.cc:260] T 472816994d814cb198b69e5248697440 P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:47.908634 23591 tablet_bootstrap.cc:492] T 2424fd9bd7f14e508fd106cbfda786a4 P c0f38c1908414566aa9115bb2cbdd85d: No bootstrap required, opened a new log
I20260809 15:45:47.912147 23594 tablet_bootstrap.cc:654] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 67b3aeadfa7a432abe8af4d292e04ab9: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:47.909329 23591 ts_tablet_manager.cc:1397] T 2424fd9bd7f14e508fd106cbfda786a4 P c0f38c1908414566aa9115bb2cbdd85d: Time spent bootstrapping tablet: real 0.174s user 0.010s sys 0.008s
I20260809 15:45:47.916684 23591 raft_consensus.cc:348] T 2424fd9bd7f14e508fd106cbfda786a4 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:47.917290 23591 raft_consensus.cc:374] T 2424fd9bd7f14e508fd106cbfda786a4 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:47.917536 23591 raft_consensus.cc:729] T 2424fd9bd7f14e508fd106cbfda786a4 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Initialized, Role: FOLLOWER
I20260809 15:45:47.920835 23593 ts_tablet_manager.cc:1428] T 472816994d814cb198b69e5248697440 P 65c36d164b0540f099d592f38307ee6c: Time spent starting tablet: real 0.030s user 0.008s sys 0.007s
I20260809 15:45:47.922049 23648 raft_consensus.cc:482] T 16ca412a0dae4feebe85dd01bf1a9173 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:45:47.933362 23648 raft_consensus.cc:504] T 16ca412a0dae4feebe85dd01bf1a9173 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:47.934383 23655 consensus_queue.cc:1035] T c0741ab44d99439abc3aa3bdd27ee160 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:45:47.929049 23591 consensus_queue.cc:260] T 2424fd9bd7f14e508fd106cbfda786a4 P c0f38c1908414566aa9115bb2cbdd85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:47.922830 23723 consensus_queue.cc:1035] T 16ca412a0dae4feebe85dd01bf1a9173 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.013s
I20260809 15:45:47.936718 23591 ts_tablet_manager.cc:1428] T 2424fd9bd7f14e508fd106cbfda786a4 P c0f38c1908414566aa9115bb2cbdd85d: Time spent starting tablet: real 0.023s user 0.005s sys 0.002s
I20260809 15:45:47.941061 23591 tablet_bootstrap.cc:492] T 998c6cee72cf40228c6dba064c0db2da P c0f38c1908414566aa9115bb2cbdd85d: Bootstrap starting.
I20260809 15:45:47.949793 23591 tablet_bootstrap.cc:654] T 998c6cee72cf40228c6dba064c0db2da P c0f38c1908414566aa9115bb2cbdd85d: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:47.952461 23593 tablet_bootstrap.cc:492] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 65c36d164b0540f099d592f38307ee6c: Bootstrap starting.
I20260809 15:45:47.955494 23655 consensus_queue.cc:1035] T 2db057fd10e149a9895dff6ef8c9f038 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260809 15:45:47.959384 23655 raft_consensus.cc:482] T 472816994d814cb198b69e5248697440 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:45:47.964560 23655 raft_consensus.cc:504] T 472816994d814cb198b69e5248697440 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:47.969566 23283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472816994d814cb198b69e5248697440" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:45:47.971587 23655 leader_election.cc:288] T 472816994d814cb198b69e5248697440 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
W20260809 15:45:47.972641 23470 leader_election.cc:341] T 472816994d814cb198b69e5248697440 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:47.973282 23593 tablet_bootstrap.cc:654] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 65c36d164b0540f099d592f38307ee6c: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:47.989264 23643 raft_consensus.cc:482] T ba56725e42f74a1baa4c04d9660727f9 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:45:47.989959 23643 raft_consensus.cc:504] T ba56725e42f74a1baa4c04d9660727f9 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:47.996626 23691 consensus_queue.cc:1035] T c0741ab44d99439abc3aa3bdd27ee160 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260809 15:45:47.997375 23643 leader_election.cc:288] T ba56725e42f74a1baa4c04d9660727f9 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:45:47.998799 23400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba56725e42f74a1baa4c04d9660727f9" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:45:47.999430 23400 raft_consensus.cc:2454] T ba56725e42f74a1baa4c04d9660727f9 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 0.
I20260809 15:45:47.999730 23283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba56725e42f74a1baa4c04d9660727f9" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:45:47.996440 23399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472816994d814cb198b69e5248697440" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:45:48.000506 23399 raft_consensus.cc:2454] T 472816994d814cb198b69e5248697440 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 0.
I20260809 15:45:48.001914 23472 leader_election.cc:302] T ba56725e42f74a1baa4c04d9660727f9 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:45:48.002763 23648 leader_election.cc:288] T 16ca412a0dae4feebe85dd01bf1a9173 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:45:48.002871 23472 leader_election.cc:302] T 472816994d814cb198b69e5248697440 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters: 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:45:48.004091 23529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16ca412a0dae4feebe85dd01bf1a9173" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:45:48.004904 23655 raft_consensus.cc:2781] T ba56725e42f74a1baa4c04d9660727f9 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 15:45:48.005401 23655 raft_consensus.cc:482] T ba56725e42f74a1baa4c04d9660727f9 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:45:48.005925 23655 raft_consensus.cc:3037] T ba56725e42f74a1baa4c04d9660727f9 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Advancing to term 1
W20260809 15:45:48.006961 23470 leader_election.cc:341] T ba56725e42f74a1baa4c04d9660727f9 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:48.010079 23283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16ca412a0dae4feebe85dd01bf1a9173" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:45:48.011435 23691 raft_consensus.cc:2781] T 472816994d814cb198b69e5248697440 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 15:45:48.011801 23691 raft_consensus.cc:482] T 472816994d814cb198b69e5248697440 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:45:48.012125 23691 raft_consensus.cc:3037] T 472816994d814cb198b69e5248697440 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:48.012084 23351 leader_election.cc:302] T 16ca412a0dae4feebe85dd01bf1a9173 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:45:48.012602 23594 tablet_bootstrap.cc:492] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 67b3aeadfa7a432abe8af4d292e04ab9: No bootstrap required, opened a new log
I20260809 15:45:48.013036 23594 ts_tablet_manager.cc:1397] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent bootstrapping tablet: real 0.130s user 0.038s sys 0.037s
I20260809 15:45:48.015220 23594 raft_consensus.cc:348] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:48.021965 23655 raft_consensus.cc:504] T ba56725e42f74a1baa4c04d9660727f9 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:48.022405 23594 raft_consensus.cc:374] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:48.023451 23594 raft_consensus.cc:729] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Initialized, Role: FOLLOWER
I20260809 15:45:48.023691 23664 raft_consensus.cc:2726] T 16ca412a0dae4feebe85dd01bf1a9173 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 15:45:48.019707 23691 raft_consensus.cc:504] T 472816994d814cb198b69e5248697440 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:48.024631 23594 consensus_queue.cc:260] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 67b3aeadfa7a432abe8af4d292e04ab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:48.026517 23594 ts_tablet_manager.cc:1428] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent starting tablet: real 0.013s user 0.005s sys 0.001s
I20260809 15:45:48.026690 23283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472816994d814cb198b69e5248697440" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:45:48.027212 23691 leader_election.cc:288] T 472816994d814cb198b69e5248697440 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 1 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
W20260809 15:45:48.027948 23470 leader_election.cc:341] T 472816994d814cb198b69e5248697440 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:48.028156 23399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472816994d814cb198b69e5248697440" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:45:48.028781 23399 raft_consensus.cc:3037] T 472816994d814cb198b69e5248697440 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:48.029006 23400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba56725e42f74a1baa4c04d9660727f9" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:45:48.029573 23400 raft_consensus.cc:3037] T ba56725e42f74a1baa4c04d9660727f9 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:48.027362 23594 tablet_bootstrap.cc:492] T 2424fd9bd7f14e508fd106cbfda786a4 P 67b3aeadfa7a432abe8af4d292e04ab9: Bootstrap starting.
I20260809 15:45:48.033257 23591 tablet_bootstrap.cc:492] T 998c6cee72cf40228c6dba064c0db2da P c0f38c1908414566aa9115bb2cbdd85d: No bootstrap required, opened a new log
I20260809 15:45:48.033700 23591 ts_tablet_manager.cc:1397] T 998c6cee72cf40228c6dba064c0db2da P c0f38c1908414566aa9115bb2cbdd85d: Time spent bootstrapping tablet: real 0.093s user 0.020s sys 0.020s
I20260809 15:45:48.042991 23399 raft_consensus.cc:2454] T 472816994d814cb198b69e5248697440 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 1.
I20260809 15:45:48.047195 23594 tablet_bootstrap.cc:654] T 2424fd9bd7f14e508fd106cbfda786a4 P 67b3aeadfa7a432abe8af4d292e04ab9: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:48.035463 23655 leader_election.cc:288] T ba56725e42f74a1baa4c04d9660727f9 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 1 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:45:48.051159 23472 leader_election.cc:302] T 472816994d814cb198b69e5248697440 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters: 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:45:48.051971 23655 raft_consensus.cc:2781] T 472816994d814cb198b69e5248697440 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:45:48.053198 23655 raft_consensus.cc:686] T 472816994d814cb198b69e5248697440 P 65c36d164b0540f099d592f38307ee6c [term 1 LEADER]: Becoming Leader. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:45:48.054237 23593 tablet_bootstrap.cc:492] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 65c36d164b0540f099d592f38307ee6c: No bootstrap required, opened a new log
I20260809 15:45:48.054636 23593 ts_tablet_manager.cc:1397] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 65c36d164b0540f099d592f38307ee6c: Time spent bootstrapping tablet: real 0.102s user 0.009s sys 0.027s
I20260809 15:45:48.054454 23655 consensus_queue.cc:237] T 472816994d814cb198b69e5248697440 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:48.055487 23400 raft_consensus.cc:2454] T ba56725e42f74a1baa4c04d9660727f9 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 1.
I20260809 15:45:48.055996 23286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba56725e42f74a1baa4c04d9660727f9" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:45:48.057801 23472 leader_election.cc:302] T ba56725e42f74a1baa4c04d9660727f9 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
W20260809 15:45:48.059309 23470 leader_election.cc:341] T ba56725e42f74a1baa4c04d9660727f9 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:48.060760 23643 raft_consensus.cc:2781] T ba56725e42f74a1baa4c04d9660727f9 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:45:48.061161 23643 raft_consensus.cc:686] T ba56725e42f74a1baa4c04d9660727f9 P 65c36d164b0540f099d592f38307ee6c [term 1 LEADER]: Becoming Leader. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:45:48.060978 23593 raft_consensus.cc:348] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:48.061678 23593 raft_consensus.cc:374] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:48.061966 23593 raft_consensus.cc:729] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Initialized, Role: FOLLOWER
I20260809 15:45:48.061928 23643 consensus_queue.cc:237] T ba56725e42f74a1baa4c04d9660727f9 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:48.062708 23593 consensus_queue.cc:260] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:48.064841 23593 ts_tablet_manager.cc:1428] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 65c36d164b0540f099d592f38307ee6c: Time spent starting tablet: real 0.010s user 0.007s sys 0.000s
I20260809 15:45:48.065721 23593 tablet_bootstrap.cc:492] T 998c6cee72cf40228c6dba064c0db2da P 65c36d164b0540f099d592f38307ee6c: Bootstrap starting.
I20260809 15:45:48.067814 23723 raft_consensus.cc:482] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:45:48.068490 23723 raft_consensus.cc:504] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:48.070863 23723 leader_election.cc:288] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:45:48.072147 23591 raft_consensus.cc:348] T 998c6cee72cf40228c6dba064c0db2da P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:48.073061 23529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5712f6b6fffc4cc5a8ddfbe63e4097df" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:45:48.074115 23591 raft_consensus.cc:374] T 998c6cee72cf40228c6dba064c0db2da P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:48.074430 23529 raft_consensus.cc:2454] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 0.
I20260809 15:45:48.075350 23400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5712f6b6fffc4cc5a8ddfbe63e4097df" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:45:48.076404 23591 raft_consensus.cc:729] T 998c6cee72cf40228c6dba064c0db2da P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Initialized, Role: FOLLOWER
I20260809 15:45:48.077229 23228 leader_election.cc:302] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
W20260809 15:45:48.079610 23228 leader_election.cc:341] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:48.078084 23591 consensus_queue.cc:260] T 998c6cee72cf40228c6dba064c0db2da P c0f38c1908414566aa9115bb2cbdd85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:48.078881 23723 raft_consensus.cc:2781] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 15:45:48.084918 23723 raft_consensus.cc:482] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:45:48.085431 23723 raft_consensus.cc:3037] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:48.086133 23591 ts_tablet_manager.cc:1428] T 998c6cee72cf40228c6dba064c0db2da P c0f38c1908414566aa9115bb2cbdd85d: Time spent starting tablet: real 0.052s user 0.005s sys 0.003s
I20260809 15:45:48.091132 23591 tablet_bootstrap.cc:492] T 7baa2492df0340c4a5ac241959ce1ac3 P c0f38c1908414566aa9115bb2cbdd85d: Bootstrap starting.
I20260809 15:45:48.096333 23594 tablet_bootstrap.cc:492] T 2424fd9bd7f14e508fd106cbfda786a4 P 67b3aeadfa7a432abe8af4d292e04ab9: No bootstrap required, opened a new log
I20260809 15:45:48.096740 23594 ts_tablet_manager.cc:1397] T 2424fd9bd7f14e508fd106cbfda786a4 P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent bootstrapping tablet: real 0.070s user 0.009s sys 0.009s
I20260809 15:45:48.096905 23723 raft_consensus.cc:504] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:48.097893 23591 tablet_bootstrap.cc:654] T 7baa2492df0340c4a5ac241959ce1ac3 P c0f38c1908414566aa9115bb2cbdd85d: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:48.099139 23594 raft_consensus.cc:348] T 2424fd9bd7f14e508fd106cbfda786a4 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:48.099666 23400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5712f6b6fffc4cc5a8ddfbe63e4097df" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:45:48.101148 23594 raft_consensus.cc:374] T 2424fd9bd7f14e508fd106cbfda786a4 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:48.101425 23594 raft_consensus.cc:729] T 2424fd9bd7f14e508fd106cbfda786a4 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Initialized, Role: FOLLOWER
I20260809 15:45:48.101619 23593 tablet_bootstrap.cc:654] T 998c6cee72cf40228c6dba064c0db2da P 65c36d164b0540f099d592f38307ee6c: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:48.102105 23594 consensus_queue.cc:260] T 2424fd9bd7f14e508fd106cbfda786a4 P 67b3aeadfa7a432abe8af4d292e04ab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:45:48.102417 23228 leader_election.cc:341] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:48.104552 23594 ts_tablet_manager.cc:1428] T 2424fd9bd7f14e508fd106cbfda786a4 P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent starting tablet: real 0.008s user 0.000s sys 0.006s
I20260809 15:45:48.105365 23594 tablet_bootstrap.cc:492] T 3d880de657e544468c82157769622a26 P 67b3aeadfa7a432abe8af4d292e04ab9: Bootstrap starting.
I20260809 15:45:48.105933 23529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5712f6b6fffc4cc5a8ddfbe63e4097df" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:45:48.106630 23529 raft_consensus.cc:3037] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:48.112681 23594 tablet_bootstrap.cc:654] T 3d880de657e544468c82157769622a26 P 67b3aeadfa7a432abe8af4d292e04ab9: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:48.113180 23117 catalog_manager.cc:5495] T 472816994d814cb198b69e5248697440 P 65c36d164b0540f099d592f38307ee6c reported cstate change: term changed from 0 to 1, leader changed from <none> to 65c36d164b0540f099d592f38307ee6c (127.20.202.67). New cstate: current_term: 1 leader_uuid: "65c36d164b0540f099d592f38307ee6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } health_report { overall_health: HEALTHY } } }
I20260809 15:45:48.126310 23594 tablet_bootstrap.cc:492] T 3d880de657e544468c82157769622a26 P 67b3aeadfa7a432abe8af4d292e04ab9: No bootstrap required, opened a new log
I20260809 15:45:48.126642 23594 ts_tablet_manager.cc:1397] T 3d880de657e544468c82157769622a26 P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent bootstrapping tablet: real 0.021s user 0.013s sys 0.005s
I20260809 15:45:48.125383 23648 raft_consensus.cc:482] T 998c6cee72cf40228c6dba064c0db2da P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:45:48.127060 23723 leader_election.cc:288] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:45:48.128916 23648 raft_consensus.cc:504] T 998c6cee72cf40228c6dba064c0db2da P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:48.131175 23648 leader_election.cc:288] T 998c6cee72cf40228c6dba064c0db2da P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:45:48.132524 23593 tablet_bootstrap.cc:492] T 998c6cee72cf40228c6dba064c0db2da P 65c36d164b0540f099d592f38307ee6c: No bootstrap required, opened a new log
I20260809 15:45:48.133064 23593 ts_tablet_manager.cc:1397] T 998c6cee72cf40228c6dba064c0db2da P 65c36d164b0540f099d592f38307ee6c: Time spent bootstrapping tablet: real 0.068s user 0.004s sys 0.011s
I20260809 15:45:48.132542 23594 raft_consensus.cc:348] T 3d880de657e544468c82157769622a26 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:48.133524 23594 raft_consensus.cc:374] T 3d880de657e544468c82157769622a26 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:48.133834 23594 raft_consensus.cc:729] T 3d880de657e544468c82157769622a26 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Initialized, Role: FOLLOWER
I20260809 15:45:48.134554 23286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998c6cee72cf40228c6dba064c0db2da" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:45:48.134928 23594 consensus_queue.cc:260] T 3d880de657e544468c82157769622a26 P 67b3aeadfa7a432abe8af4d292e04ab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:48.135818 23591 tablet_bootstrap.cc:492] T 7baa2492df0340c4a5ac241959ce1ac3 P c0f38c1908414566aa9115bb2cbdd85d: No bootstrap required, opened a new log
I20260809 15:45:48.136195 23593 raft_consensus.cc:348] T 998c6cee72cf40228c6dba064c0db2da P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:48.136449 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998c6cee72cf40228c6dba064c0db2da" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:45:48.137044 23593 raft_consensus.cc:374] T 998c6cee72cf40228c6dba064c0db2da P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:48.137328 23593 raft_consensus.cc:729] T 998c6cee72cf40228c6dba064c0db2da P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Initialized, Role: FOLLOWER
I20260809 15:45:48.137281 23529 raft_consensus.cc:2454] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 1.
I20260809 15:45:48.138124 23593 consensus_queue.cc:260] T 998c6cee72cf40228c6dba064c0db2da P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:45:48.138628 23351 leader_election.cc:341] T 998c6cee72cf40228c6dba064c0db2da P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:48.138958 23527 raft_consensus.cc:2454] T 998c6cee72cf40228c6dba064c0db2da P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 0.
I20260809 15:45:48.140560 23594 ts_tablet_manager.cc:1428] T 3d880de657e544468c82157769622a26 P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent starting tablet: real 0.014s user 0.007s sys 0.000s
I20260809 15:45:48.141448 23594 tablet_bootstrap.cc:492] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 67b3aeadfa7a432abe8af4d292e04ab9: Bootstrap starting.
I20260809 15:45:48.136235 23591 ts_tablet_manager.cc:1397] T 7baa2492df0340c4a5ac241959ce1ac3 P c0f38c1908414566aa9115bb2cbdd85d: Time spent bootstrapping tablet: real 0.045s user 0.023s sys 0.014s
I20260809 15:45:48.142683 23353 leader_election.cc:302] T 998c6cee72cf40228c6dba064c0db2da P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters: 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:45:48.142935 23593 ts_tablet_manager.cc:1428] T 998c6cee72cf40228c6dba064c0db2da P 65c36d164b0540f099d592f38307ee6c: Time spent starting tablet: real 0.010s user 0.007s sys 0.000s
I20260809 15:45:48.143815 23593 tablet_bootstrap.cc:492] T 05ad7d8fdca64e3da527280ca5be9280 P 65c36d164b0540f099d592f38307ee6c: Bootstrap starting.
I20260809 15:45:48.143972 23591 raft_consensus.cc:348] T 7baa2492df0340c4a5ac241959ce1ac3 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:48.144598 23591 raft_consensus.cc:374] T 7baa2492df0340c4a5ac241959ce1ac3 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:48.144838 23591 raft_consensus.cc:729] T 7baa2492df0340c4a5ac241959ce1ac3 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Initialized, Role: FOLLOWER
I20260809 15:45:48.145483 23591 consensus_queue.cc:260] T 7baa2492df0340c4a5ac241959ce1ac3 P c0f38c1908414566aa9115bb2cbdd85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:48.146510 23664 raft_consensus.cc:2781] T 998c6cee72cf40228c6dba064c0db2da P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 15:45:48.146855 23664 raft_consensus.cc:482] T 998c6cee72cf40228c6dba064c0db2da P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:45:48.147306 23594 tablet_bootstrap.cc:654] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 67b3aeadfa7a432abe8af4d292e04ab9: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:48.147177 23664 raft_consensus.cc:3037] T 998c6cee72cf40228c6dba064c0db2da P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:48.147560 23116 catalog_manager.cc:5495] T ba56725e42f74a1baa4c04d9660727f9 P 65c36d164b0540f099d592f38307ee6c reported cstate change: term changed from 0 to 1, leader changed from <none> to 65c36d164b0540f099d592f38307ee6c (127.20.202.67). New cstate: current_term: 1 leader_uuid: "65c36d164b0540f099d592f38307ee6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } health_report { overall_health: HEALTHY } } }
I20260809 15:45:48.148554 23591 ts_tablet_manager.cc:1428] T 7baa2492df0340c4a5ac241959ce1ac3 P c0f38c1908414566aa9115bb2cbdd85d: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20260809 15:45:48.149539 23593 tablet_bootstrap.cc:654] T 05ad7d8fdca64e3da527280ca5be9280 P 65c36d164b0540f099d592f38307ee6c: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:48.150604 23228 leader_election.cc:302] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters: c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:45:48.151576 23723 raft_consensus.cc:2781] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:45:48.152032 23723 raft_consensus.cc:686] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 LEADER]: Becoming Leader. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Running, Role: LEADER
I20260809 15:45:48.153179 23591 tablet_bootstrap.cc:492] T 05ad7d8fdca64e3da527280ca5be9280 P c0f38c1908414566aa9115bb2cbdd85d: Bootstrap starting.
I20260809 15:45:48.153052 23723 consensus_queue.cc:237] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:48.155675 23664 raft_consensus.cc:504] T 998c6cee72cf40228c6dba064c0db2da P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:48.156668 23661 consensus_queue.cc:1035] T 0c48a3049897452383257aa757834fb5 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:45:48.158897 23591 tablet_bootstrap.cc:654] T 05ad7d8fdca64e3da527280ca5be9280 P c0f38c1908414566aa9115bb2cbdd85d: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:48.160188 23286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998c6cee72cf40228c6dba064c0db2da" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
W20260809 15:45:48.161429 23351 leader_election.cc:341] T 998c6cee72cf40228c6dba064c0db2da P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:48.162999 23664 leader_election.cc:288] T 998c6cee72cf40228c6dba064c0db2da P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:45:48.163235 23529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998c6cee72cf40228c6dba064c0db2da" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:45:48.163875 23529 raft_consensus.cc:3037] T 998c6cee72cf40228c6dba064c0db2da P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:48.168037 23661 consensus_queue.cc:1035] T a3622f6e3e3641cea1960e3a31498bb7 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:45:48.169675 23643 raft_consensus.cc:482] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:45:48.170133 23643 raft_consensus.cc:504] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:48.172607 23643 leader_election.cc:288] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:45:48.174152 23399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e4ffcf83c234ae7910ddef3ec0b4bda" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:45:48.174580 23664 raft_consensus.cc:482] T 2424fd9bd7f14e508fd106cbfda786a4 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 15:45:48.175781 23472 leader_election.cc:341] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:48.171402 23529 raft_consensus.cc:2454] T 998c6cee72cf40228c6dba064c0db2da P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 1.
I20260809 15:45:48.174048 23286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e4ffcf83c234ae7910ddef3ec0b4bda" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:45:48.181767 23353 leader_election.cc:302] T 998c6cee72cf40228c6dba064c0db2da P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters: 67b3aeadfa7a432abe8af4d292e04ab9
W20260809 15:45:48.194808 23470 leader_election.cc:341] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:48.195864 23591 tablet_bootstrap.cc:492] T 05ad7d8fdca64e3da527280ca5be9280 P c0f38c1908414566aa9115bb2cbdd85d: No bootstrap required, opened a new log
I20260809 15:45:48.196308 23591 ts_tablet_manager.cc:1397] T 05ad7d8fdca64e3da527280ca5be9280 P c0f38c1908414566aa9115bb2cbdd85d: Time spent bootstrapping tablet: real 0.043s user 0.014s sys 0.013s
I20260809 15:45:48.195231 23470 leader_election.cc:302] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:45:48.196707 23661 consensus_queue.cc:1035] T 0c48a3049897452383257aa757834fb5 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260809 15:45:48.197767 23643 raft_consensus.cc:2726] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 15:45:48.195992 23648 raft_consensus.cc:2781] T 998c6cee72cf40228c6dba064c0db2da P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:45:48.198587 23648 raft_consensus.cc:686] T 998c6cee72cf40228c6dba064c0db2da P c0f38c1908414566aa9115bb2cbdd85d [term 1 LEADER]: Becoming Leader. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:45:48.199713 23648 consensus_queue.cc:237] T 998c6cee72cf40228c6dba064c0db2da P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:48.204142 23643 consensus_queue.cc:1035] T cef3e7277d7d4760824a3436ed4a847d P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:45:48.207664 23604 raft_consensus.cc:482] T 05ad7d8fdca64e3da527280ca5be9280 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:45:48.206548 23116 catalog_manager.cc:5495] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 67b3aeadfa7a432abe8af4d292e04ab9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65). New cstate: current_term: 1 leader_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } health_report { overall_health: UNKNOWN } } }
I20260809 15:45:48.175419 23664 raft_consensus.cc:504] T 2424fd9bd7f14e508fd106cbfda786a4 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:48.218148 23723 raft_consensus.cc:482] T 2424fd9bd7f14e508fd106cbfda786a4 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:45:48.218608 23723 raft_consensus.cc:504] T 2424fd9bd7f14e508fd106cbfda786a4 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:48.219439 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2424fd9bd7f14e508fd106cbfda786a4" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:45:48.210585 23604 raft_consensus.cc:504] T 05ad7d8fdca64e3da527280ca5be9280 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:48.220877 23723 leader_election.cc:288] T 2424fd9bd7f14e508fd106cbfda786a4 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
W20260809 15:45:48.221122 23353 leader_election.cc:341] T 2424fd9bd7f14e508fd106cbfda786a4 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:48.221864 23664 leader_election.cc:288] T 2424fd9bd7f14e508fd106cbfda786a4 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:45:48.222661 23604 leader_election.cc:288] T 05ad7d8fdca64e3da527280ca5be9280 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:45:48.224198 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2424fd9bd7f14e508fd106cbfda786a4" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:45:48.224690 23282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2424fd9bd7f14e508fd106cbfda786a4" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:45:48.225306 23282 raft_consensus.cc:2454] T 2424fd9bd7f14e508fd106cbfda786a4 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 0.
I20260809 15:45:48.224784 23402 raft_consensus.cc:2454] T 2424fd9bd7f14e508fd106cbfda786a4 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 0.
I20260809 15:45:48.228189 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05ad7d8fdca64e3da527280ca5be9280" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:45:48.228952 23351 leader_election.cc:302] T 2424fd9bd7f14e508fd106cbfda786a4 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c
I20260809 15:45:48.230165 23528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05ad7d8fdca64e3da527280ca5be9280" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:45:48.230346 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2424fd9bd7f14e508fd106cbfda786a4" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:45:48.231236 23664 raft_consensus.cc:2781] T 2424fd9bd7f14e508fd106cbfda786a4 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 15:45:48.231622 23664 raft_consensus.cc:482] T 2424fd9bd7f14e508fd106cbfda786a4 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:45:48.231925 23664 raft_consensus.cc:3037] T 2424fd9bd7f14e508fd106cbfda786a4 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Advancing to term 1
W20260809 15:45:48.232285 23228 leader_election.cc:341] T 05ad7d8fdca64e3da527280ca5be9280 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553): Illegal state: must be running to vote when last-logged opid is not known
W20260809 15:45:48.232916 23228 leader_election.cc:341] T 2424fd9bd7f14e508fd106cbfda786a4 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:48.233865 23228 leader_election.cc:302] T 2424fd9bd7f14e508fd106cbfda786a4 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c
I20260809 15:45:48.238323 23643 consensus_queue.cc:1035] T 953eea268e17491ca76f3a555860a512 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260809 15:45:48.238404 23723 raft_consensus.cc:2781] T 2424fd9bd7f14e508fd106cbfda786a4 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 15:45:48.241261 23723 raft_consensus.cc:482] T 2424fd9bd7f14e508fd106cbfda786a4 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:45:48.241578 23723 raft_consensus.cc:3037] T 2424fd9bd7f14e508fd106cbfda786a4 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:48.245132 23594 tablet_bootstrap.cc:492] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 67b3aeadfa7a432abe8af4d292e04ab9: No bootstrap required, opened a new log
I20260809 15:45:48.245575 23594 ts_tablet_manager.cc:1397] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent bootstrapping tablet: real 0.104s user 0.021s sys 0.017s
I20260809 15:45:48.247668 23664 raft_consensus.cc:504] T 2424fd9bd7f14e508fd106cbfda786a4 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:48.248386 23594 raft_consensus.cc:348] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:48.249065 23594 raft_consensus.cc:374] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:48.249317 23594 raft_consensus.cc:729] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Initialized, Role: FOLLOWER
I20260809 15:45:48.249923 23594 consensus_queue.cc:260] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 67b3aeadfa7a432abe8af4d292e04ab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:48.251854 23282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2424fd9bd7f14e508fd106cbfda786a4" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:45:48.261608 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2424fd9bd7f14e508fd106cbfda786a4" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:45:48.264123 23593 tablet_bootstrap.cc:492] T 05ad7d8fdca64e3da527280ca5be9280 P 65c36d164b0540f099d592f38307ee6c: No bootstrap required, opened a new log
W20260809 15:45:48.263990 23353 leader_election.cc:341] T 2424fd9bd7f14e508fd106cbfda786a4 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:48.264570 23593 ts_tablet_manager.cc:1397] T 05ad7d8fdca64e3da527280ca5be9280 P 65c36d164b0540f099d592f38307ee6c: Time spent bootstrapping tablet: real 0.121s user 0.016s sys 0.020s
I20260809 15:45:48.263568 23664 leader_election.cc:288] T 2424fd9bd7f14e508fd106cbfda786a4 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:45:48.269100 23591 raft_consensus.cc:348] T 05ad7d8fdca64e3da527280ca5be9280 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:48.283968 23591 raft_consensus.cc:374] T 05ad7d8fdca64e3da527280ca5be9280 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:48.284224 23591 raft_consensus.cc:729] T 05ad7d8fdca64e3da527280ca5be9280 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Initialized, Role: FOLLOWER
I20260809 15:45:48.284880 23591 consensus_queue.cc:260] T 05ad7d8fdca64e3da527280ca5be9280 P c0f38c1908414566aa9115bb2cbdd85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:48.285231 23691 consensus_queue.cc:1035] T 953eea268e17491ca76f3a555860a512 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260809 15:45:48.285776 23402 raft_consensus.cc:2454] T 05ad7d8fdca64e3da527280ca5be9280 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 0.
I20260809 15:45:48.289140 23594 ts_tablet_manager.cc:1428] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent starting tablet: real 0.043s user 0.006s sys 0.000s
I20260809 15:45:48.290062 23594 tablet_bootstrap.cc:492] T 998c6cee72cf40228c6dba064c0db2da P 67b3aeadfa7a432abe8af4d292e04ab9: Bootstrap starting.
I20260809 15:45:48.287101 23591 ts_tablet_manager.cc:1428] T 05ad7d8fdca64e3da527280ca5be9280 P c0f38c1908414566aa9115bb2cbdd85d: Time spent starting tablet: real 0.090s user 0.005s sys 0.001s
I20260809 15:45:48.291921 23593 raft_consensus.cc:348] T 05ad7d8fdca64e3da527280ca5be9280 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:48.292987 23593 raft_consensus.cc:374] T 05ad7d8fdca64e3da527280ca5be9280 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:48.293206 23593 raft_consensus.cc:729] T 05ad7d8fdca64e3da527280ca5be9280 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Initialized, Role: FOLLOWER
I20260809 15:45:48.293684 23593 consensus_queue.cc:260] T 05ad7d8fdca64e3da527280ca5be9280 P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:48.295862 23591 tablet_bootstrap.cc:492] T 3d880de657e544468c82157769622a26 P c0f38c1908414566aa9115bb2cbdd85d: Bootstrap starting.
I20260809 15:45:48.291325 23228 leader_election.cc:302] T 05ad7d8fdca64e3da527280ca5be9280 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c
I20260809 15:45:48.299376 23604 raft_consensus.cc:2781] T 05ad7d8fdca64e3da527280ca5be9280 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 15:45:48.299829 23604 raft_consensus.cc:482] T 05ad7d8fdca64e3da527280ca5be9280 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:45:48.300287 23604 raft_consensus.cc:3037] T 05ad7d8fdca64e3da527280ca5be9280 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:48.302599 23723 raft_consensus.cc:504] T 2424fd9bd7f14e508fd106cbfda786a4 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:48.309725 23282 raft_consensus.cc:2379] T 2424fd9bd7f14e508fd106cbfda786a4 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d in current term 1: Already voted for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in this term.
I20260809 15:45:48.311638 23593 ts_tablet_manager.cc:1428] T 05ad7d8fdca64e3da527280ca5be9280 P 65c36d164b0540f099d592f38307ee6c: Time spent starting tablet: real 0.047s user 0.006s sys 0.001s
I20260809 15:45:48.314021 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2424fd9bd7f14e508fd106cbfda786a4" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:45:48.314756 23402 raft_consensus.cc:2379] T 2424fd9bd7f14e508fd106cbfda786a4 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 in current term 1: Already voted for candidate c0f38c1908414566aa9115bb2cbdd85d in this term.
I20260809 15:45:48.315929 23351 leader_election.cc:302] T 2424fd9bd7f14e508fd106cbfda786a4 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:45:48.320103 23604 raft_consensus.cc:504] T 05ad7d8fdca64e3da527280ca5be9280 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:48.321274 23648 raft_consensus.cc:2726] T 2424fd9bd7f14e508fd106cbfda786a4 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260809 15:45:48.321921 23604 leader_election.cc:288] T 05ad7d8fdca64e3da527280ca5be9280 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:45:48.343658 23723 leader_election.cc:288] T 2424fd9bd7f14e508fd106cbfda786a4 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:45:48.351486 23528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2424fd9bd7f14e508fd106cbfda786a4" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
W20260809 15:45:48.352732 23228 leader_election.cc:341] T 2424fd9bd7f14e508fd106cbfda786a4 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:48.353188 23228 leader_election.cc:302] T 2424fd9bd7f14e508fd106cbfda786a4 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:45:48.353415 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05ad7d8fdca64e3da527280ca5be9280" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:45:48.353979 23527 raft_consensus.cc:3037] T 05ad7d8fdca64e3da527280ca5be9280 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:48.353912 23655 consensus_queue.cc:1035] T cef3e7277d7d4760824a3436ed4a847d P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:45:48.354374 23593 tablet_bootstrap.cc:492] T e14d118b2eaa42aa8a7cd4025aba7128 P 65c36d164b0540f099d592f38307ee6c: Bootstrap starting.
I20260809 15:45:48.363631 23722 consensus_queue.cc:1035] T a3622f6e3e3641cea1960e3a31498bb7 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:45:48.357002 23661 raft_consensus.cc:2726] T 2424fd9bd7f14e508fd106cbfda786a4 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260809 15:45:48.370919 23527 raft_consensus.cc:2454] T 05ad7d8fdca64e3da527280ca5be9280 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 1.
I20260809 15:45:48.374979 23399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05ad7d8fdca64e3da527280ca5be9280" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:45:48.377907 23116 catalog_manager.cc:5495] T 998c6cee72cf40228c6dba064c0db2da P c0f38c1908414566aa9115bb2cbdd85d reported cstate change: term changed from 0 to 1, leader changed from <none> to c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66). New cstate: current_term: 1 leader_uuid: "c0f38c1908414566aa9115bb2cbdd85d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } health_report { overall_health: UNKNOWN } } }
W20260809 15:45:48.358989 23193 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260809 15:45:48.368758 23594 tablet_bootstrap.cc:654] T 998c6cee72cf40228c6dba064c0db2da P 67b3aeadfa7a432abe8af4d292e04ab9: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:48.398537 23648 raft_consensus.cc:482] T 7baa2492df0340c4a5ac241959ce1ac3 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:45:48.403502 23648 raft_consensus.cc:504] T 7baa2492df0340c4a5ac241959ce1ac3 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:48.405491 23648 leader_election.cc:288] T 7baa2492df0340c4a5ac241959ce1ac3 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:45:48.408780 23399 raft_consensus.cc:3037] T 05ad7d8fdca64e3da527280ca5be9280 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:48.413213 23228 leader_election.cc:302] T 05ad7d8fdca64e3da527280ca5be9280 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:45:48.420728 23722 raft_consensus.cc:2781] T 05ad7d8fdca64e3da527280ca5be9280 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:45:48.426347 23722 raft_consensus.cc:686] T 05ad7d8fdca64e3da527280ca5be9280 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 LEADER]: Becoming Leader. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Running, Role: LEADER
I20260809 15:45:48.427177 23722 consensus_queue.cc:237] T 05ad7d8fdca64e3da527280ca5be9280 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:48.437084 23399 raft_consensus.cc:2454] T 05ad7d8fdca64e3da527280ca5be9280 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 1.
I20260809 15:45:48.438030 23282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7baa2492df0340c4a5ac241959ce1ac3" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
W20260809 15:45:48.439399 23351 leader_election.cc:341] T 7baa2492df0340c4a5ac241959ce1ac3 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:48.443665 23593 tablet_bootstrap.cc:654] T e14d118b2eaa42aa8a7cd4025aba7128 P 65c36d164b0540f099d592f38307ee6c: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:48.434749 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7baa2492df0340c4a5ac241959ce1ac3" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
W20260809 15:45:48.445690 23353 leader_election.cc:341] T 7baa2492df0340c4a5ac241959ce1ac3 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:48.446125 23353 leader_election.cc:302] T 7baa2492df0340c4a5ac241959ce1ac3 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:45:48.447906 23648 raft_consensus.cc:2726] T 7baa2492df0340c4a5ac241959ce1ac3 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 15:45:48.449361 23594 tablet_bootstrap.cc:492] T 998c6cee72cf40228c6dba064c0db2da P 67b3aeadfa7a432abe8af4d292e04ab9: No bootstrap required, opened a new log
I20260809 15:45:48.449695 23594 ts_tablet_manager.cc:1397] T 998c6cee72cf40228c6dba064c0db2da P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent bootstrapping tablet: real 0.160s user 0.007s sys 0.011s
I20260809 15:45:48.450687 23591 tablet_bootstrap.cc:654] T 3d880de657e544468c82157769622a26 P c0f38c1908414566aa9115bb2cbdd85d: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:48.453922 23594 raft_consensus.cc:348] T 998c6cee72cf40228c6dba064c0db2da P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:48.454586 23594 raft_consensus.cc:374] T 998c6cee72cf40228c6dba064c0db2da P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:48.454839 23594 raft_consensus.cc:729] T 998c6cee72cf40228c6dba064c0db2da P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Initialized, Role: FOLLOWER
I20260809 15:45:48.455472 23594 consensus_queue.cc:260] T 998c6cee72cf40228c6dba064c0db2da P 67b3aeadfa7a432abe8af4d292e04ab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:48.457300 23594 ts_tablet_manager.cc:1428] T 998c6cee72cf40228c6dba064c0db2da P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent starting tablet: real 0.007s user 0.003s sys 0.002s
I20260809 15:45:48.458062 23594 tablet_bootstrap.cc:492] T 7baa2492df0340c4a5ac241959ce1ac3 P 67b3aeadfa7a432abe8af4d292e04ab9: Bootstrap starting.
I20260809 15:45:48.477942 23594 tablet_bootstrap.cc:654] T 7baa2492df0340c4a5ac241959ce1ac3 P 67b3aeadfa7a432abe8af4d292e04ab9: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:48.487314 23594 tablet_bootstrap.cc:492] T 7baa2492df0340c4a5ac241959ce1ac3 P 67b3aeadfa7a432abe8af4d292e04ab9: No bootstrap required, opened a new log
I20260809 15:45:48.487742 23594 ts_tablet_manager.cc:1397] T 7baa2492df0340c4a5ac241959ce1ac3 P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent bootstrapping tablet: real 0.030s user 0.013s sys 0.000s
I20260809 15:45:48.490109 23594 raft_consensus.cc:348] T 7baa2492df0340c4a5ac241959ce1ac3 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:48.490749 23594 raft_consensus.cc:374] T 7baa2492df0340c4a5ac241959ce1ac3 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:48.491008 23594 raft_consensus.cc:729] T 7baa2492df0340c4a5ac241959ce1ac3 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Initialized, Role: FOLLOWER
I20260809 15:45:48.491683 23594 consensus_queue.cc:260] T 7baa2492df0340c4a5ac241959ce1ac3 P 67b3aeadfa7a432abe8af4d292e04ab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:48.492844 23116 catalog_manager.cc:5495] T 05ad7d8fdca64e3da527280ca5be9280 P 67b3aeadfa7a432abe8af4d292e04ab9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65). New cstate: current_term: 1 leader_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } health_report { overall_health: UNKNOWN } } }
I20260809 15:45:48.493880 23594 ts_tablet_manager.cc:1428] T 7baa2492df0340c4a5ac241959ce1ac3 P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260809 15:45:48.494753 23594 tablet_bootstrap.cc:492] T ba56725e42f74a1baa4c04d9660727f9 P 67b3aeadfa7a432abe8af4d292e04ab9: Bootstrap starting.
I20260809 15:45:48.496073 23591 tablet_bootstrap.cc:492] T 3d880de657e544468c82157769622a26 P c0f38c1908414566aa9115bb2cbdd85d: No bootstrap required, opened a new log
I20260809 15:45:48.496487 23591 ts_tablet_manager.cc:1397] T 3d880de657e544468c82157769622a26 P c0f38c1908414566aa9115bb2cbdd85d: Time spent bootstrapping tablet: real 0.201s user 0.008s sys 0.013s
I20260809 15:45:48.496892 23722 raft_consensus.cc:482] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:45:48.497351 23722 raft_consensus.cc:504] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:48.498842 23591 raft_consensus.cc:348] T 3d880de657e544468c82157769622a26 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:48.499512 23591 raft_consensus.cc:374] T 3d880de657e544468c82157769622a26 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:48.499703 23722 leader_election.cc:288] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:45:48.499752 23591 raft_consensus.cc:729] T 3d880de657e544468c82157769622a26 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Initialized, Role: FOLLOWER
I20260809 15:45:48.500849 23591 consensus_queue.cc:260] T 3d880de657e544468c82157769622a26 P c0f38c1908414566aa9115bb2cbdd85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:48.501991 23400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e4ffcf83c234ae7910ddef3ec0b4bda" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:45:48.502822 23591 ts_tablet_manager.cc:1428] T 3d880de657e544468c82157769622a26 P c0f38c1908414566aa9115bb2cbdd85d: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260809 15:45:48.503620 23529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e4ffcf83c234ae7910ddef3ec0b4bda" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:45:48.504227 23529 raft_consensus.cc:2454] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 0.
I20260809 15:45:48.505331 23228 leader_election.cc:302] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:45:48.506157 23591 tablet_bootstrap.cc:492] T 6ced718c1a5c4da79cb8c1ebd1435b87 P c0f38c1908414566aa9115bb2cbdd85d: Bootstrap starting.
W20260809 15:45:48.506762 23228 leader_election.cc:341] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:48.507635 23722 raft_consensus.cc:2781] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 15:45:48.508035 23722 raft_consensus.cc:482] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:45:48.509344 23722 raft_consensus.cc:3037] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:48.517678 23593 tablet_bootstrap.cc:492] T e14d118b2eaa42aa8a7cd4025aba7128 P 65c36d164b0540f099d592f38307ee6c: No bootstrap required, opened a new log
I20260809 15:45:48.518069 23593 ts_tablet_manager.cc:1397] T e14d118b2eaa42aa8a7cd4025aba7128 P 65c36d164b0540f099d592f38307ee6c: Time spent bootstrapping tablet: real 0.164s user 0.018s sys 0.020s
I20260809 15:45:48.518944 23808 raft_consensus.cc:482] T 3d880de657e544468c82157769622a26 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:45:48.519322 23808 raft_consensus.cc:504] T 3d880de657e544468c82157769622a26 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:48.521055 23722 raft_consensus.cc:504] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:48.521723 23808 leader_election.cc:288] T 3d880de657e544468c82157769622a26 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:45:48.522122 23594 tablet_bootstrap.cc:654] T ba56725e42f74a1baa4c04d9660727f9 P 67b3aeadfa7a432abe8af4d292e04ab9: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:48.522733 23529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d880de657e544468c82157769622a26" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:45:48.524024 23591 tablet_bootstrap.cc:654] T 6ced718c1a5c4da79cb8c1ebd1435b87 P c0f38c1908414566aa9115bb2cbdd85d: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:48.524539 23593 raft_consensus.cc:348] T e14d118b2eaa42aa8a7cd4025aba7128 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:48.524729 23400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d880de657e544468c82157769622a26" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:45:48.525244 23593 raft_consensus.cc:374] T e14d118b2eaa42aa8a7cd4025aba7128 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:48.525545 23400 raft_consensus.cc:2454] T 3d880de657e544468c82157769622a26 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 0.
I20260809 15:45:48.526069 23722 leader_election.cc:288] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:45:48.525570 23593 raft_consensus.cc:729] T e14d118b2eaa42aa8a7cd4025aba7128 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Initialized, Role: FOLLOWER
I20260809 15:45:48.527704 23593 consensus_queue.cc:260] T e14d118b2eaa42aa8a7cd4025aba7128 P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:48.528095 23400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e4ffcf83c234ae7910ddef3ec0b4bda" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:45:48.528090 23228 leader_election.cc:302] T 3d880de657e544468c82157769622a26 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:45:48.528690 23529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e4ffcf83c234ae7910ddef3ec0b4bda" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:45:48.529408 23529 raft_consensus.cc:3037] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:48.530149 23722 raft_consensus.cc:2781] T 3d880de657e544468c82157769622a26 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 15:45:48.530786 23722 raft_consensus.cc:482] T 3d880de657e544468c82157769622a26 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:45:48.531176 23722 raft_consensus.cc:3037] T 3d880de657e544468c82157769622a26 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Advancing to term 1
W20260809 15:45:48.529635 23228 leader_election.cc:341] T 3d880de657e544468c82157769622a26 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553): Illegal state: must be running to vote when last-logged opid is not known
W20260809 15:45:48.532466 23228 leader_election.cc:341] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:48.536618 23594 tablet_bootstrap.cc:492] T ba56725e42f74a1baa4c04d9660727f9 P 67b3aeadfa7a432abe8af4d292e04ab9: No bootstrap required, opened a new log
I20260809 15:45:48.537070 23594 ts_tablet_manager.cc:1397] T ba56725e42f74a1baa4c04d9660727f9 P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent bootstrapping tablet: real 0.042s user 0.011s sys 0.001s
I20260809 15:45:48.537986 23593 ts_tablet_manager.cc:1428] T e14d118b2eaa42aa8a7cd4025aba7128 P 65c36d164b0540f099d592f38307ee6c: Time spent starting tablet: real 0.020s user 0.007s sys 0.000s
I20260809 15:45:48.539213 23529 raft_consensus.cc:2454] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 1.
I20260809 15:45:48.540205 23228 leader_election.cc:302] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters: c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:45:48.540913 23808 raft_consensus.cc:2781] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:45:48.541378 23808 raft_consensus.cc:686] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 LEADER]: Becoming Leader. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Running, Role: LEADER
I20260809 15:45:48.541216 23593 tablet_bootstrap.cc:492] T 2424fd9bd7f14e508fd106cbfda786a4 P 65c36d164b0540f099d592f38307ee6c: Bootstrap starting.
I20260809 15:45:48.540441 23594 raft_consensus.cc:348] T ba56725e42f74a1baa4c04d9660727f9 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:48.542164 23808 consensus_queue.cc:237] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:48.539917 23722 raft_consensus.cc:504] T 3d880de657e544468c82157769622a26 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:48.548961 23400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d880de657e544468c82157769622a26" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:45:48.549507 23400 raft_consensus.cc:3037] T 3d880de657e544468c82157769622a26 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:48.549659 23722 leader_election.cc:288] T 3d880de657e544468c82157769622a26 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:45:48.542620 23594 raft_consensus.cc:374] T ba56725e42f74a1baa4c04d9660727f9 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:48.550187 23594 raft_consensus.cc:729] T ba56725e42f74a1baa4c04d9660727f9 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Initialized, Role: FOLLOWER
I20260809 15:45:48.549479 23529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d880de657e544468c82157769622a26" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
W20260809 15:45:48.551648 23228 leader_election.cc:341] T 3d880de657e544468c82157769622a26 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:48.552034 23117 catalog_manager.cc:5495] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 67b3aeadfa7a432abe8af4d292e04ab9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65). New cstate: current_term: 1 leader_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } health_report { overall_health: UNKNOWN } } }
I20260809 15:45:48.552415 23594 consensus_queue.cc:260] T ba56725e42f74a1baa4c04d9660727f9 P 67b3aeadfa7a432abe8af4d292e04ab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:48.554265 23594 ts_tablet_manager.cc:1428] T ba56725e42f74a1baa4c04d9660727f9 P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent starting tablet: real 0.017s user 0.005s sys 0.000s
I20260809 15:45:48.555008 23594 tablet_bootstrap.cc:492] T ae6d2c08a7a94f58ac8dd698e3fd456c P 67b3aeadfa7a432abe8af4d292e04ab9: Bootstrap starting.
I20260809 15:45:48.556528 23400 raft_consensus.cc:2454] T 3d880de657e544468c82157769622a26 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 1.
I20260809 15:45:48.557605 23228 leader_election.cc:302] T 3d880de657e544468c82157769622a26 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c
I20260809 15:45:48.558923 23808 raft_consensus.cc:2781] T 3d880de657e544468c82157769622a26 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:45:48.559499 23808 raft_consensus.cc:686] T 3d880de657e544468c82157769622a26 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 LEADER]: Becoming Leader. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Running, Role: LEADER
I20260809 15:45:48.560616 23808 consensus_queue.cc:237] T 3d880de657e544468c82157769622a26 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:48.562443 23591 tablet_bootstrap.cc:492] T 6ced718c1a5c4da79cb8c1ebd1435b87 P c0f38c1908414566aa9115bb2cbdd85d: No bootstrap required, opened a new log
I20260809 15:45:48.563086 23591 ts_tablet_manager.cc:1397] T 6ced718c1a5c4da79cb8c1ebd1435b87 P c0f38c1908414566aa9115bb2cbdd85d: Time spent bootstrapping tablet: real 0.057s user 0.011s sys 0.019s
I20260809 15:45:48.564375 23594 tablet_bootstrap.cc:654] T ae6d2c08a7a94f58ac8dd698e3fd456c P 67b3aeadfa7a432abe8af4d292e04ab9: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:48.565480 23802 raft_consensus.cc:482] T ba56725e42f74a1baa4c04d9660727f9 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:45:48.566075 23591 raft_consensus.cc:348] T 6ced718c1a5c4da79cb8c1ebd1435b87 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:48.566879 23591 raft_consensus.cc:374] T 6ced718c1a5c4da79cb8c1ebd1435b87 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:48.566116 23802 raft_consensus.cc:504] T ba56725e42f74a1baa4c04d9660727f9 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:48.567145 23591 raft_consensus.cc:729] T 6ced718c1a5c4da79cb8c1ebd1435b87 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Initialized, Role: FOLLOWER
I20260809 15:45:48.568277 23591 consensus_queue.cc:260] T 6ced718c1a5c4da79cb8c1ebd1435b87 P c0f38c1908414566aa9115bb2cbdd85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:48.569113 23648 raft_consensus.cc:482] T 6ced718c1a5c4da79cb8c1ebd1435b87 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:45:48.570032 23400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba56725e42f74a1baa4c04d9660727f9" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:45:48.569764 23648 raft_consensus.cc:504] T 6ced718c1a5c4da79cb8c1ebd1435b87 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:48.570142 23802 leader_election.cc:288] T ba56725e42f74a1baa4c04d9660727f9 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:45:48.571169 23400 raft_consensus.cc:2379] T ba56725e42f74a1baa4c04d9660727f9 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 in current term 1: Already voted for candidate 65c36d164b0540f099d592f38307ee6c in this term.
I20260809 15:45:48.574867 23529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba56725e42f74a1baa4c04d9660727f9" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:45:48.575644 23593 tablet_bootstrap.cc:654] T 2424fd9bd7f14e508fd106cbfda786a4 P 65c36d164b0540f099d592f38307ee6c: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:48.576488 23282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ced718c1a5c4da79cb8c1ebd1435b87" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:45:48.577461 23282 raft_consensus.cc:2454] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 0.
I20260809 15:45:48.578583 23228 leader_election.cc:302] T ba56725e42f74a1baa4c04d9660727f9 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:45:48.578682 23648 leader_election.cc:288] T 6ced718c1a5c4da79cb8c1ebd1435b87 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:45:48.579591 23529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ced718c1a5c4da79cb8c1ebd1435b87" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:45:48.579910 23802 raft_consensus.cc:3037] T ba56725e42f74a1baa4c04d9660727f9 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:48.579763 23351 leader_election.cc:302] T 6ced718c1a5c4da79cb8c1ebd1435b87 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters:
W20260809 15:45:48.583340 23353 leader_election.cc:341] T 6ced718c1a5c4da79cb8c1ebd1435b87 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:48.584254 23648 raft_consensus.cc:2781] T 6ced718c1a5c4da79cb8c1ebd1435b87 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 15:45:48.584610 23648 raft_consensus.cc:482] T 6ced718c1a5c4da79cb8c1ebd1435b87 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:45:48.584935 23648 raft_consensus.cc:3037] T 6ced718c1a5c4da79cb8c1ebd1435b87 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:48.586452 23802 raft_consensus.cc:2726] T ba56725e42f74a1baa4c04d9660727f9 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 15:45:48.593344 23648 raft_consensus.cc:504] T 6ced718c1a5c4da79cb8c1ebd1435b87 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:48.593195 23116 catalog_manager.cc:5495] T 3d880de657e544468c82157769622a26 P 67b3aeadfa7a432abe8af4d292e04ab9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65). New cstate: current_term: 1 leader_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } health_report { overall_health: UNKNOWN } } }
I20260809 15:45:48.597371 23591 ts_tablet_manager.cc:1428] T 6ced718c1a5c4da79cb8c1ebd1435b87 P c0f38c1908414566aa9115bb2cbdd85d: Time spent starting tablet: real 0.034s user 0.002s sys 0.006s
I20260809 15:45:48.598217 23591 tablet_bootstrap.cc:492] T e14d118b2eaa42aa8a7cd4025aba7128 P c0f38c1908414566aa9115bb2cbdd85d: Bootstrap starting.
I20260809 15:45:48.603520 23591 tablet_bootstrap.cc:654] T e14d118b2eaa42aa8a7cd4025aba7128 P c0f38c1908414566aa9115bb2cbdd85d: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:48.608492 23282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ced718c1a5c4da79cb8c1ebd1435b87" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:45:48.609025 23282 raft_consensus.cc:3037] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:48.609337 23593 tablet_bootstrap.cc:492] T 2424fd9bd7f14e508fd106cbfda786a4 P 65c36d164b0540f099d592f38307ee6c: No bootstrap required, opened a new log
I20260809 15:45:48.609824 23593 ts_tablet_manager.cc:1397] T 2424fd9bd7f14e508fd106cbfda786a4 P 65c36d164b0540f099d592f38307ee6c: Time spent bootstrapping tablet: real 0.069s user 0.011s sys 0.002s
I20260809 15:45:48.608351 23648 leader_election.cc:288] T 6ced718c1a5c4da79cb8c1ebd1435b87 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:45:48.615972 23529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ced718c1a5c4da79cb8c1ebd1435b87" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
W20260809 15:45:48.617488 23353 leader_election.cc:341] T 6ced718c1a5c4da79cb8c1ebd1435b87 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:48.619117 23593 raft_consensus.cc:348] T 2424fd9bd7f14e508fd106cbfda786a4 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:48.619819 23593 raft_consensus.cc:374] T 2424fd9bd7f14e508fd106cbfda786a4 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:48.620095 23593 raft_consensus.cc:729] T 2424fd9bd7f14e508fd106cbfda786a4 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Initialized, Role: FOLLOWER
I20260809 15:45:48.620695 23593 consensus_queue.cc:260] T 2424fd9bd7f14e508fd106cbfda786a4 P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:45:48.621650 23470 consensus_peers.cc:487] T 472816994d814cb198b69e5248697440 P 65c36d164b0540f099d592f38307ee6c -> Peer 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459): Couldn't send request to peer 67b3aeadfa7a432abe8af4d292e04ab9. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260809 15:45:48.622671 23593 ts_tablet_manager.cc:1428] T 2424fd9bd7f14e508fd106cbfda786a4 P 65c36d164b0540f099d592f38307ee6c: Time spent starting tablet: real 0.013s user 0.003s sys 0.000s
I20260809 15:45:48.623497 23655 consensus_queue.cc:1035] T ba56725e42f74a1baa4c04d9660727f9 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:45:48.624056 23593 tablet_bootstrap.cc:492] T 0c29937a80724ddb9d07a27534a146f3 P 65c36d164b0540f099d592f38307ee6c: Bootstrap starting.
I20260809 15:45:48.631203 23593 tablet_bootstrap.cc:654] T 0c29937a80724ddb9d07a27534a146f3 P 65c36d164b0540f099d592f38307ee6c: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:48.627915 23655 raft_consensus.cc:482] T e14d118b2eaa42aa8a7cd4025aba7128 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:45:48.637910 23655 raft_consensus.cc:504] T e14d118b2eaa42aa8a7cd4025aba7128 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:48.638898 23643 consensus_queue.cc:1035] T 472816994d814cb198b69e5248697440 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:45:48.641417 23283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e14d118b2eaa42aa8a7cd4025aba7128" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:45:48.642021 23283 raft_consensus.cc:2454] T e14d118b2eaa42aa8a7cd4025aba7128 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 0.
I20260809 15:45:48.642077 23282 raft_consensus.cc:2454] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 1.
I20260809 15:45:48.643049 23591 tablet_bootstrap.cc:492] T e14d118b2eaa42aa8a7cd4025aba7128 P c0f38c1908414566aa9115bb2cbdd85d: No bootstrap required, opened a new log
I20260809 15:45:48.642920 23399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e14d118b2eaa42aa8a7cd4025aba7128" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:45:48.643484 23591 ts_tablet_manager.cc:1397] T e14d118b2eaa42aa8a7cd4025aba7128 P c0f38c1908414566aa9115bb2cbdd85d: Time spent bootstrapping tablet: real 0.045s user 0.008s sys 0.006s
I20260809 15:45:48.643405 23351 leader_election.cc:302] T 6ced718c1a5c4da79cb8c1ebd1435b87 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c
W20260809 15:45:48.644155 23472 leader_election.cc:341] T e14d118b2eaa42aa8a7cd4025aba7128 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:48.644860 23790 raft_consensus.cc:2781] T 6ced718c1a5c4da79cb8c1ebd1435b87 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:45:48.645157 23655 leader_election.cc:288] T e14d118b2eaa42aa8a7cd4025aba7128 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:45:48.645332 23790 raft_consensus.cc:686] T 6ced718c1a5c4da79cb8c1ebd1435b87 P c0f38c1908414566aa9115bb2cbdd85d [term 1 LEADER]: Becoming Leader. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:45:48.645646 23655 consensus_queue.cc:1035] T ba56725e42f74a1baa4c04d9660727f9 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:45:48.648152 23470 leader_election.cc:302] T e14d118b2eaa42aa8a7cd4025aba7128 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters: c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:45:48.649837 23643 raft_consensus.cc:2781] T e14d118b2eaa42aa8a7cd4025aba7128 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 15:45:48.650378 23643 raft_consensus.cc:482] T e14d118b2eaa42aa8a7cd4025aba7128 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:45:48.650837 23643 raft_consensus.cc:3037] T e14d118b2eaa42aa8a7cd4025aba7128 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:48.646351 23790 consensus_queue.cc:237] T 6ced718c1a5c4da79cb8c1ebd1435b87 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:48.654870 23594 tablet_bootstrap.cc:492] T ae6d2c08a7a94f58ac8dd698e3fd456c P 67b3aeadfa7a432abe8af4d292e04ab9: No bootstrap required, opened a new log
I20260809 15:45:48.655313 23594 ts_tablet_manager.cc:1397] T ae6d2c08a7a94f58ac8dd698e3fd456c P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent bootstrapping tablet: real 0.100s user 0.012s sys 0.033s
I20260809 15:45:48.658562 23594 raft_consensus.cc:348] T ae6d2c08a7a94f58ac8dd698e3fd456c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:48.659044 23594 raft_consensus.cc:374] T ae6d2c08a7a94f58ac8dd698e3fd456c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:48.659245 23594 raft_consensus.cc:729] T ae6d2c08a7a94f58ac8dd698e3fd456c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Initialized, Role: FOLLOWER
I20260809 15:45:48.659415 23591 raft_consensus.cc:348] T e14d118b2eaa42aa8a7cd4025aba7128 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:48.660123 23591 raft_consensus.cc:374] T e14d118b2eaa42aa8a7cd4025aba7128 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:48.660496 23591 raft_consensus.cc:729] T e14d118b2eaa42aa8a7cd4025aba7128 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Initialized, Role: FOLLOWER
I20260809 15:45:48.660091 23594 consensus_queue.cc:260] T ae6d2c08a7a94f58ac8dd698e3fd456c P 67b3aeadfa7a432abe8af4d292e04ab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:48.661103 23591 consensus_queue.cc:260] T e14d118b2eaa42aa8a7cd4025aba7128 P c0f38c1908414566aa9115bb2cbdd85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:45:48.662281 23228 consensus_peers.cc:487] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 67b3aeadfa7a432abe8af4d292e04ab9 -> Peer c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263): Couldn't send request to peer c0f38c1908414566aa9115bb2cbdd85d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260809 15:45:48.662777 23594 ts_tablet_manager.cc:1428] T ae6d2c08a7a94f58ac8dd698e3fd456c P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent starting tablet: real 0.007s user 0.002s sys 0.004s
I20260809 15:45:48.663206 23591 ts_tablet_manager.cc:1428] T e14d118b2eaa42aa8a7cd4025aba7128 P c0f38c1908414566aa9115bb2cbdd85d: Time spent starting tablet: real 0.019s user 0.003s sys 0.000s
I20260809 15:45:48.663661 23594 tablet_bootstrap.cc:492] T 472816994d814cb198b69e5248697440 P 67b3aeadfa7a432abe8af4d292e04ab9: Bootstrap starting.
I20260809 15:45:48.664094 23591 tablet_bootstrap.cc:492] T ae6d2c08a7a94f58ac8dd698e3fd456c P c0f38c1908414566aa9115bb2cbdd85d: Bootstrap starting.
I20260809 15:45:48.677800 23808 consensus_queue.cc:1035] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:45:48.683327 23664 raft_consensus.cc:482] T e14d118b2eaa42aa8a7cd4025aba7128 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:45:48.683759 23664 raft_consensus.cc:504] T e14d118b2eaa42aa8a7cd4025aba7128 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:48.685210 23648 consensus_queue.cc:1035] T 998c6cee72cf40228c6dba064c0db2da P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:45:48.685694 23664 leader_election.cc:288] T e14d118b2eaa42aa8a7cd4025aba7128 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:45:48.688391 23283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e14d118b2eaa42aa8a7cd4025aba7128" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:45:48.689327 23283 raft_consensus.cc:2454] T e14d118b2eaa42aa8a7cd4025aba7128 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 0.
I20260809 15:45:48.682952 23643 raft_consensus.cc:504] T e14d118b2eaa42aa8a7cd4025aba7128 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:48.691300 23351 leader_election.cc:302] T e14d118b2eaa42aa8a7cd4025aba7128 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:45:48.692883 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e14d118b2eaa42aa8a7cd4025aba7128" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:45:48.693630 23283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e14d118b2eaa42aa8a7cd4025aba7128" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:45:48.694165 23594 tablet_bootstrap.cc:654] T 472816994d814cb198b69e5248697440 P 67b3aeadfa7a432abe8af4d292e04ab9: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:48.694294 23283 raft_consensus.cc:3037] T e14d118b2eaa42aa8a7cd4025aba7128 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:48.694296 23790 raft_consensus.cc:2781] T e14d118b2eaa42aa8a7cd4025aba7128 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 15:45:48.696940 23116 catalog_manager.cc:5495] T 6ced718c1a5c4da79cb8c1ebd1435b87 P c0f38c1908414566aa9115bb2cbdd85d reported cstate change: term changed from 0 to 1, leader changed from <none> to c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66). New cstate: current_term: 1 leader_uuid: "c0f38c1908414566aa9115bb2cbdd85d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } health_report { overall_health: UNKNOWN } } }
I20260809 15:45:48.693670 23527 raft_consensus.cc:2379] T e14d118b2eaa42aa8a7cd4025aba7128 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d in current term 1: Already voted for candidate 65c36d164b0540f099d592f38307ee6c in this term.
I20260809 15:45:48.701483 23802 raft_consensus.cc:482] T 998c6cee72cf40228c6dba064c0db2da P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:45:48.704653 23802 raft_consensus.cc:504] T 998c6cee72cf40228c6dba064c0db2da P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:48.708316 23643 leader_election.cc:288] T e14d118b2eaa42aa8a7cd4025aba7128 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 1 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:45:48.708863 23283 raft_consensus.cc:2454] T e14d118b2eaa42aa8a7cd4025aba7128 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 1.
I20260809 15:45:48.709659 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998c6cee72cf40228c6dba064c0db2da" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:45:48.710248 23470 leader_election.cc:302] T e14d118b2eaa42aa8a7cd4025aba7128 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:45:48.711443 23643 raft_consensus.cc:2781] T e14d118b2eaa42aa8a7cd4025aba7128 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:45:48.712404 23643 raft_consensus.cc:686] T e14d118b2eaa42aa8a7cd4025aba7128 P 65c36d164b0540f099d592f38307ee6c [term 1 LEADER]: Becoming Leader. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:45:48.708465 23802 leader_election.cc:288] T 998c6cee72cf40228c6dba064c0db2da P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:45:48.712971 23400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998c6cee72cf40228c6dba064c0db2da" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:45:48.703956 23593 tablet_bootstrap.cc:492] T 0c29937a80724ddb9d07a27534a146f3 P 65c36d164b0540f099d592f38307ee6c: No bootstrap required, opened a new log
I20260809 15:45:48.714013 23593 ts_tablet_manager.cc:1397] T 0c29937a80724ddb9d07a27534a146f3 P 65c36d164b0540f099d592f38307ee6c: Time spent bootstrapping tablet: real 0.090s user 0.009s sys 0.006s
I20260809 15:45:48.714059 23643 consensus_queue.cc:237] T e14d118b2eaa42aa8a7cd4025aba7128 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:48.714727 23228 leader_election.cc:302] T 998c6cee72cf40228c6dba064c0db2da P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:45:48.723517 23802 raft_consensus.cc:2726] T 998c6cee72cf40228c6dba064c0db2da P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 15:45:48.700075 23790 raft_consensus.cc:482] T e14d118b2eaa42aa8a7cd4025aba7128 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:45:48.724470 23790 raft_consensus.cc:3037] T e14d118b2eaa42aa8a7cd4025aba7128 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:48.727425 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e14d118b2eaa42aa8a7cd4025aba7128" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:45:48.730345 23593 raft_consensus.cc:348] T 0c29937a80724ddb9d07a27534a146f3 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:48.730996 23593 raft_consensus.cc:374] T 0c29937a80724ddb9d07a27534a146f3 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:48.731287 23593 raft_consensus.cc:729] T 0c29937a80724ddb9d07a27534a146f3 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Initialized, Role: FOLLOWER
I20260809 15:45:48.732069 23591 tablet_bootstrap.cc:654] T ae6d2c08a7a94f58ac8dd698e3fd456c P c0f38c1908414566aa9115bb2cbdd85d: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:48.731890 23593 consensus_queue.cc:260] T 0c29937a80724ddb9d07a27534a146f3 P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:48.735608 23802 raft_consensus.cc:482] T 7baa2492df0340c4a5ac241959ce1ac3 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:45:48.736277 23802 raft_consensus.cc:504] T 7baa2492df0340c4a5ac241959ce1ac3 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:48.741814 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7baa2492df0340c4a5ac241959ce1ac3" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
W20260809 15:45:48.743651 23228 leader_election.cc:341] T 7baa2492df0340c4a5ac241959ce1ac3 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:48.745184 23400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7baa2492df0340c4a5ac241959ce1ac3" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:45:48.745967 23400 raft_consensus.cc:2454] T 7baa2492df0340c4a5ac241959ce1ac3 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 0.
I20260809 15:45:48.747641 23228 leader_election.cc:302] T 7baa2492df0340c4a5ac241959ce1ac3 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c
I20260809 15:45:48.748814 23808 raft_consensus.cc:2781] T 7baa2492df0340c4a5ac241959ce1ac3 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 15:45:48.749315 23808 raft_consensus.cc:482] T 7baa2492df0340c4a5ac241959ce1ac3 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:45:48.749819 23808 raft_consensus.cc:3037] T 7baa2492df0340c4a5ac241959ce1ac3 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:48.751905 23802 leader_election.cc:288] T 7baa2492df0340c4a5ac241959ce1ac3 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:45:48.754968 23601 raft_consensus.cc:482] T 2424fd9bd7f14e508fd106cbfda786a4 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:45:48.755625 23601 raft_consensus.cc:504] T 2424fd9bd7f14e508fd106cbfda786a4 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:48.759292 23283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2424fd9bd7f14e508fd106cbfda786a4" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:45:48.760097 23283 raft_consensus.cc:2379] T 2424fd9bd7f14e508fd106cbfda786a4 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c in current term 1: Already voted for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in this term.
I20260809 15:45:48.760514 23790 raft_consensus.cc:504] T e14d118b2eaa42aa8a7cd4025aba7128 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:48.762409 23601 leader_election.cc:288] T 2424fd9bd7f14e508fd106cbfda786a4 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:45:48.762573 23790 leader_election.cc:288] T e14d118b2eaa42aa8a7cd4025aba7128 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:45:48.767408 23593 ts_tablet_manager.cc:1428] T 0c29937a80724ddb9d07a27534a146f3 P 65c36d164b0540f099d592f38307ee6c: Time spent starting tablet: real 0.053s user 0.007s sys 0.000s
I20260809 15:45:48.767542 23402 raft_consensus.cc:2379] T e14d118b2eaa42aa8a7cd4025aba7128 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c in current term 1: Already voted for candidate c0f38c1908414566aa9115bb2cbdd85d in this term.
I20260809 15:45:48.767782 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e14d118b2eaa42aa8a7cd4025aba7128" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:45:48.777693 23593 tablet_bootstrap.cc:492] T 3d880de657e544468c82157769622a26 P 65c36d164b0540f099d592f38307ee6c: Bootstrap starting.
I20260809 15:45:48.824101 23286 raft_consensus.cc:3037] T 998c6cee72cf40228c6dba064c0db2da P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:48.818336 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2424fd9bd7f14e508fd106cbfda786a4" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:45:48.827131 23283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e14d118b2eaa42aa8a7cd4025aba7128" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:45:48.827852 23283 raft_consensus.cc:2379] T e14d118b2eaa42aa8a7cd4025aba7128 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d in current term 1: Already voted for candidate 65c36d164b0540f099d592f38307ee6c in this term.
I20260809 15:45:48.834542 23402 raft_consensus.cc:2379] T 2424fd9bd7f14e508fd106cbfda786a4 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c in current term 1: Already voted for candidate c0f38c1908414566aa9115bb2cbdd85d in this term.
I20260809 15:45:48.841063 23808 raft_consensus.cc:504] T 7baa2492df0340c4a5ac241959ce1ac3 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:48.842466 23472 leader_election.cc:302] T 2424fd9bd7f14e508fd106cbfda786a4 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:45:48.842900 23808 leader_election.cc:288] T 7baa2492df0340c4a5ac241959ce1ac3 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:45:48.843562 23655 raft_consensus.cc:3037] T 2424fd9bd7f14e508fd106cbfda786a4 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:48.846442 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7baa2492df0340c4a5ac241959ce1ac3" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:45:48.847254 23402 raft_consensus.cc:3037] T 7baa2492df0340c4a5ac241959ce1ac3 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:48.834518 23351 leader_election.cc:302] T e14d118b2eaa42aa8a7cd4025aba7128 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:45:48.855459 23116 catalog_manager.cc:5495] T e14d118b2eaa42aa8a7cd4025aba7128 P 65c36d164b0540f099d592f38307ee6c reported cstate change: term changed from 0 to 1, leader changed from <none> to 65c36d164b0540f099d592f38307ee6c (127.20.202.67). New cstate: current_term: 1 leader_uuid: "65c36d164b0540f099d592f38307ee6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } health_report { overall_health: HEALTHY } } }
I20260809 15:45:48.856238 23528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7baa2492df0340c4a5ac241959ce1ac3" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:45:48.856410 23648 raft_consensus.cc:2726] T e14d118b2eaa42aa8a7cd4025aba7128 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260809 15:45:48.858582 23655 raft_consensus.cc:2726] T 2424fd9bd7f14e508fd106cbfda786a4 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 15:45:48.863121 23593 tablet_bootstrap.cc:654] T 3d880de657e544468c82157769622a26 P 65c36d164b0540f099d592f38307ee6c: Neither blocks nor log segments found. Creating new log.
W20260809 15:45:48.871309 23228 leader_election.cc:341] T 7baa2492df0340c4a5ac241959ce1ac3 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:48.878396 23790 consensus_queue.cc:1035] T 998c6cee72cf40228c6dba064c0db2da P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:45:48.880149 23594 tablet_bootstrap.cc:492] T 472816994d814cb198b69e5248697440 P 67b3aeadfa7a432abe8af4d292e04ab9: No bootstrap required, opened a new log
I20260809 15:45:48.880544 23594 ts_tablet_manager.cc:1397] T 472816994d814cb198b69e5248697440 P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent bootstrapping tablet: real 0.217s user 0.026s sys 0.047s
I20260809 15:45:48.880417 23723 consensus_queue.cc:1035] T 05ad7d8fdca64e3da527280ca5be9280 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:45:48.884308 23594 raft_consensus.cc:348] T 472816994d814cb198b69e5248697440 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:48.884934 23594 raft_consensus.cc:374] T 472816994d814cb198b69e5248697440 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:48.885138 23594 raft_consensus.cc:729] T 472816994d814cb198b69e5248697440 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Initialized, Role: FOLLOWER
I20260809 15:45:48.885988 23594 consensus_queue.cc:260] T 472816994d814cb198b69e5248697440 P 67b3aeadfa7a432abe8af4d292e04ab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:48.888274 23594 ts_tablet_manager.cc:1428] T 472816994d814cb198b69e5248697440 P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent starting tablet: real 0.007s user 0.005s sys 0.001s
I20260809 15:45:48.889127 23594 tablet_bootstrap.cc:492] T 3296ef50be254fc29cfc77d22a8f7a23 P 67b3aeadfa7a432abe8af4d292e04ab9: Bootstrap starting.
I20260809 15:45:48.892920 23723 raft_consensus.cc:482] T 472816994d814cb198b69e5248697440 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:45:48.896991 23723 raft_consensus.cc:504] T 472816994d814cb198b69e5248697440 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:48.899902 23723 leader_election.cc:288] T 472816994d814cb198b69e5248697440 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:45:48.902793 23402 raft_consensus.cc:2454] T 7baa2492df0340c4a5ac241959ce1ac3 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 1.
I20260809 15:45:48.903800 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472816994d814cb198b69e5248697440" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:45:48.904917 23528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472816994d814cb198b69e5248697440" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:45:48.905791 23723 raft_consensus.cc:482] T ae6d2c08a7a94f58ac8dd698e3fd456c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:45:48.906298 23723 raft_consensus.cc:504] T ae6d2c08a7a94f58ac8dd698e3fd456c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:48.907335 23722 consensus_queue.cc:1035] T 05ad7d8fdca64e3da527280ca5be9280 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260809 15:45:48.907554 23228 leader_election.cc:302] T 7baa2492df0340c4a5ac241959ce1ac3 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c
I20260809 15:45:48.908762 23722 raft_consensus.cc:2781] T 7baa2492df0340c4a5ac241959ce1ac3 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:45:48.909205 23723 leader_election.cc:288] T ae6d2c08a7a94f58ac8dd698e3fd456c P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:45:48.909310 23722 raft_consensus.cc:686] T 7baa2492df0340c4a5ac241959ce1ac3 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 LEADER]: Becoming Leader. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Running, Role: LEADER
I20260809 15:45:48.910055 23722 consensus_queue.cc:237] T 7baa2492df0340c4a5ac241959ce1ac3 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:48.910532 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae6d2c08a7a94f58ac8dd698e3fd456c" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:45:48.917049 23591 tablet_bootstrap.cc:492] T ae6d2c08a7a94f58ac8dd698e3fd456c P c0f38c1908414566aa9115bb2cbdd85d: No bootstrap required, opened a new log
I20260809 15:45:48.917464 23591 ts_tablet_manager.cc:1397] T ae6d2c08a7a94f58ac8dd698e3fd456c P c0f38c1908414566aa9115bb2cbdd85d: Time spent bootstrapping tablet: real 0.254s user 0.025s sys 0.011s
I20260809 15:45:48.922940 23528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae6d2c08a7a94f58ac8dd698e3fd456c" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:45:48.924959 23591 raft_consensus.cc:348] T ae6d2c08a7a94f58ac8dd698e3fd456c P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:48.925815 23591 raft_consensus.cc:374] T ae6d2c08a7a94f58ac8dd698e3fd456c P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:48.926074 23591 raft_consensus.cc:729] T ae6d2c08a7a94f58ac8dd698e3fd456c P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Initialized, Role: FOLLOWER
I20260809 15:45:48.926808 23591 consensus_queue.cc:260] T ae6d2c08a7a94f58ac8dd698e3fd456c P c0f38c1908414566aa9115bb2cbdd85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:48.930157 23228 leader_election.cc:302] T 472816994d814cb198b69e5248697440 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
W20260809 15:45:48.935426 23842 log.cc:924] Time spent T ba56725e42f74a1baa4c04d9660727f9 P 67b3aeadfa7a432abe8af4d292e04ab9: Append to log took a long time: real 0.090s user 0.001s sys 0.000s
I20260809 15:45:48.936964 23591 ts_tablet_manager.cc:1428] T ae6d2c08a7a94f58ac8dd698e3fd456c P c0f38c1908414566aa9115bb2cbdd85d: Time spent starting tablet: real 0.019s user 0.006s sys 0.000s
W20260809 15:45:48.937943 23228 leader_election.cc:341] T ae6d2c08a7a94f58ac8dd698e3fd456c P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:48.942270 23591 tablet_bootstrap.cc:492] T 73f6f44c259547f1b05a3a434616fa54 P c0f38c1908414566aa9115bb2cbdd85d: Bootstrap starting.
I20260809 15:45:48.932459 23722 raft_consensus.cc:2726] T 472816994d814cb198b69e5248697440 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260809 15:45:48.945762 23228 leader_election.cc:341] T ae6d2c08a7a94f58ac8dd698e3fd456c P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:48.947008 23228 leader_election.cc:302] T ae6d2c08a7a94f58ac8dd698e3fd456c P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:45:48.944885 23117 catalog_manager.cc:5495] T 7baa2492df0340c4a5ac241959ce1ac3 P 67b3aeadfa7a432abe8af4d292e04ab9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65). New cstate: current_term: 1 leader_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } health_report { overall_health: UNKNOWN } } }
I20260809 15:45:48.951040 23802 raft_consensus.cc:2726] T ae6d2c08a7a94f58ac8dd698e3fd456c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 15:45:48.959796 23593 tablet_bootstrap.cc:492] T 3d880de657e544468c82157769622a26 P 65c36d164b0540f099d592f38307ee6c: No bootstrap required, opened a new log
I20260809 15:45:48.960223 23593 ts_tablet_manager.cc:1397] T 3d880de657e544468c82157769622a26 P 65c36d164b0540f099d592f38307ee6c: Time spent bootstrapping tablet: real 0.183s user 0.012s sys 0.012s
I20260809 15:45:48.967335 23593 raft_consensus.cc:348] T 3d880de657e544468c82157769622a26 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:48.968075 23593 raft_consensus.cc:374] T 3d880de657e544468c82157769622a26 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:48.968307 23593 raft_consensus.cc:729] T 3d880de657e544468c82157769622a26 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Initialized, Role: FOLLOWER
I20260809 15:45:48.968776 23593 consensus_queue.cc:260] T 3d880de657e544468c82157769622a26 P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:48.971858 23593 ts_tablet_manager.cc:1428] T 3d880de657e544468c82157769622a26 P 65c36d164b0540f099d592f38307ee6c: Time spent starting tablet: real 0.011s user 0.006s sys 0.001s
I20260809 15:45:48.973181 23593 tablet_bootstrap.cc:492] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 65c36d164b0540f099d592f38307ee6c: Bootstrap starting.
I20260809 15:45:48.974151 23661 consensus_queue.cc:1035] T 3d880de657e544468c82157769622a26 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:45:48.983790 23594 tablet_bootstrap.cc:654] T 3296ef50be254fc29cfc77d22a8f7a23 P 67b3aeadfa7a432abe8af4d292e04ab9: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:48.989933 23591 tablet_bootstrap.cc:654] T 73f6f44c259547f1b05a3a434616fa54 P c0f38c1908414566aa9115bb2cbdd85d: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:48.999550 23593 tablet_bootstrap.cc:654] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 65c36d164b0540f099d592f38307ee6c: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:49.004475 23594 tablet_bootstrap.cc:492] T 3296ef50be254fc29cfc77d22a8f7a23 P 67b3aeadfa7a432abe8af4d292e04ab9: No bootstrap required, opened a new log
I20260809 15:45:49.004931 23594 ts_tablet_manager.cc:1397] T 3296ef50be254fc29cfc77d22a8f7a23 P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent bootstrapping tablet: real 0.116s user 0.007s sys 0.008s
I20260809 15:45:49.005784 23529 raft_consensus.cc:3037] T 3d880de657e544468c82157769622a26 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:49.008368 23594 raft_consensus.cc:348] T 3296ef50be254fc29cfc77d22a8f7a23 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:49.009075 23594 raft_consensus.cc:374] T 3296ef50be254fc29cfc77d22a8f7a23 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:49.009328 23594 raft_consensus.cc:729] T 3296ef50be254fc29cfc77d22a8f7a23 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Initialized, Role: FOLLOWER
I20260809 15:45:49.009958 23594 consensus_queue.cc:260] T 3296ef50be254fc29cfc77d22a8f7a23 P 67b3aeadfa7a432abe8af4d292e04ab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:49.013139 23591 tablet_bootstrap.cc:492] T 73f6f44c259547f1b05a3a434616fa54 P c0f38c1908414566aa9115bb2cbdd85d: No bootstrap required, opened a new log
I20260809 15:45:49.013554 23591 ts_tablet_manager.cc:1397] T 73f6f44c259547f1b05a3a434616fa54 P c0f38c1908414566aa9115bb2cbdd85d: Time spent bootstrapping tablet: real 0.071s user 0.005s sys 0.011s
I20260809 15:45:49.015161 23591 raft_consensus.cc:348] T 73f6f44c259547f1b05a3a434616fa54 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:49.015647 23591 raft_consensus.cc:374] T 73f6f44c259547f1b05a3a434616fa54 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:49.015837 23591 raft_consensus.cc:729] T 73f6f44c259547f1b05a3a434616fa54 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Initialized, Role: FOLLOWER
I20260809 15:45:49.016325 23591 consensus_queue.cc:260] T 73f6f44c259547f1b05a3a434616fa54 P c0f38c1908414566aa9115bb2cbdd85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:49.017683 23661 consensus_queue.cc:1035] T 3d880de657e544468c82157769622a26 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:45:49.018951 23591 ts_tablet_manager.cc:1428] T 73f6f44c259547f1b05a3a434616fa54 P c0f38c1908414566aa9115bb2cbdd85d: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260809 15:45:49.019784 23591 tablet_bootstrap.cc:492] T e71a05ba1d4d42bd95c3f85f9313399e P c0f38c1908414566aa9115bb2cbdd85d: Bootstrap starting.
I20260809 15:45:49.024314 23591 tablet_bootstrap.cc:654] T e71a05ba1d4d42bd95c3f85f9313399e P c0f38c1908414566aa9115bb2cbdd85d: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:49.030313 23594 ts_tablet_manager.cc:1428] T 3296ef50be254fc29cfc77d22a8f7a23 P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent starting tablet: real 0.025s user 0.001s sys 0.006s
I20260809 15:45:49.031358 23594 tablet_bootstrap.cc:492] T e71a05ba1d4d42bd95c3f85f9313399e P 67b3aeadfa7a432abe8af4d292e04ab9: Bootstrap starting.
I20260809 15:45:49.034857 23591 tablet_bootstrap.cc:492] T e71a05ba1d4d42bd95c3f85f9313399e P c0f38c1908414566aa9115bb2cbdd85d: No bootstrap required, opened a new log
I20260809 15:45:49.035281 23591 ts_tablet_manager.cc:1397] T e71a05ba1d4d42bd95c3f85f9313399e P c0f38c1908414566aa9115bb2cbdd85d: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.002s
I20260809 15:45:49.037587 23591 raft_consensus.cc:348] T e71a05ba1d4d42bd95c3f85f9313399e P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:49.038259 23591 raft_consensus.cc:374] T e71a05ba1d4d42bd95c3f85f9313399e P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:49.038514 23591 raft_consensus.cc:729] T e71a05ba1d4d42bd95c3f85f9313399e P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Initialized, Role: FOLLOWER
I20260809 15:45:49.039291 23591 consensus_queue.cc:260] T e71a05ba1d4d42bd95c3f85f9313399e P c0f38c1908414566aa9115bb2cbdd85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:49.040347 23594 tablet_bootstrap.cc:654] T e71a05ba1d4d42bd95c3f85f9313399e P 67b3aeadfa7a432abe8af4d292e04ab9: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:49.042593 23790 raft_consensus.cc:482] T 73f6f44c259547f1b05a3a434616fa54 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:45:49.043017 23790 raft_consensus.cc:504] T 73f6f44c259547f1b05a3a434616fa54 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:49.044842 23790 leader_election.cc:288] T 73f6f44c259547f1b05a3a434616fa54 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:45:49.045713 23283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73f6f44c259547f1b05a3a434616fa54" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
W20260809 15:45:49.046588 23353 consensus_peers.cc:487] T 6ced718c1a5c4da79cb8c1ebd1435b87 P c0f38c1908414566aa9115bb2cbdd85d -> Peer 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553): Couldn't send request to peer 65c36d164b0540f099d592f38307ee6c. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260809 15:45:49.047986 23528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73f6f44c259547f1b05a3a434616fa54" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
W20260809 15:45:49.049299 23353 leader_election.cc:341] T 73f6f44c259547f1b05a3a434616fa54 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:49.050884 23661 consensus_queue.cc:1035] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:45:49.052881 23591 ts_tablet_manager.cc:1428] T e71a05ba1d4d42bd95c3f85f9313399e P c0f38c1908414566aa9115bb2cbdd85d: Time spent starting tablet: real 0.017s user 0.005s sys 0.000s
W20260809 15:45:49.053094 23351 leader_election.cc:341] T 73f6f44c259547f1b05a3a434616fa54 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:49.053545 23351 leader_election.cc:302] T 73f6f44c259547f1b05a3a434616fa54 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:45:49.054128 23591 tablet_bootstrap.cc:492] T 8e4ffcf83c234ae7910ddef3ec0b4bda P c0f38c1908414566aa9115bb2cbdd85d: Bootstrap starting.
I20260809 15:45:49.054070 23790 raft_consensus.cc:2726] T 73f6f44c259547f1b05a3a434616fa54 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 15:45:49.051209 23593 tablet_bootstrap.cc:492] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 65c36d164b0540f099d592f38307ee6c: No bootstrap required, opened a new log
I20260809 15:45:49.057862 23593 ts_tablet_manager.cc:1397] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 65c36d164b0540f099d592f38307ee6c: Time spent bootstrapping tablet: real 0.085s user 0.012s sys 0.014s
I20260809 15:45:49.060228 23593 raft_consensus.cc:348] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:49.060865 23593 raft_consensus.cc:374] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:49.061472 23593 raft_consensus.cc:729] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Initialized, Role: FOLLOWER
I20260809 15:45:49.062203 23593 consensus_queue.cc:260] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:45:49.063308 23228 consensus_peers.cc:487] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 67b3aeadfa7a432abe8af4d292e04ab9 -> Peer c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263): Couldn't send request to peer c0f38c1908414566aa9115bb2cbdd85d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260809 15:45:49.063881 23591 tablet_bootstrap.cc:654] T 8e4ffcf83c234ae7910ddef3ec0b4bda P c0f38c1908414566aa9115bb2cbdd85d: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:49.064495 23593 ts_tablet_manager.cc:1428] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 65c36d164b0540f099d592f38307ee6c: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260809 15:45:49.064653 23790 raft_consensus.cc:482] T ae6d2c08a7a94f58ac8dd698e3fd456c P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:45:49.065388 23593 tablet_bootstrap.cc:492] T 3296ef50be254fc29cfc77d22a8f7a23 P 65c36d164b0540f099d592f38307ee6c: Bootstrap starting.
I20260809 15:45:49.065110 23790 raft_consensus.cc:504] T ae6d2c08a7a94f58ac8dd698e3fd456c P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:49.067866 23283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae6d2c08a7a94f58ac8dd698e3fd456c" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:45:49.067219 23790 leader_election.cc:288] T ae6d2c08a7a94f58ac8dd698e3fd456c P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:45:49.068447 23283 raft_consensus.cc:2454] T ae6d2c08a7a94f58ac8dd698e3fd456c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 0.
I20260809 15:45:49.068917 23594 tablet_bootstrap.cc:492] T e71a05ba1d4d42bd95c3f85f9313399e P 67b3aeadfa7a432abe8af4d292e04ab9: No bootstrap required, opened a new log
I20260809 15:45:49.069314 23594 ts_tablet_manager.cc:1397] T e71a05ba1d4d42bd95c3f85f9313399e P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent bootstrapping tablet: real 0.038s user 0.012s sys 0.008s
I20260809 15:45:49.071722 23351 leader_election.cc:302] T ae6d2c08a7a94f58ac8dd698e3fd456c P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:45:49.073205 23648 raft_consensus.cc:2781] T ae6d2c08a7a94f58ac8dd698e3fd456c P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 15:45:49.073525 23648 raft_consensus.cc:482] T ae6d2c08a7a94f58ac8dd698e3fd456c P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:45:49.073818 23648 raft_consensus.cc:3037] T ae6d2c08a7a94f58ac8dd698e3fd456c P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:49.075430 23643 raft_consensus.cc:482] T 0c29937a80724ddb9d07a27534a146f3 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:45:49.075851 23643 raft_consensus.cc:504] T 0c29937a80724ddb9d07a27534a146f3 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:49.077602 23643 leader_election.cc:288] T 0c29937a80724ddb9d07a27534a146f3 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:45:49.079552 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c29937a80724ddb9d07a27534a146f3" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:45:49.080310 23283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c29937a80724ddb9d07a27534a146f3" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
W20260809 15:45:49.080807 23472 leader_election.cc:341] T 0c29937a80724ddb9d07a27534a146f3 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263): Illegal state: must be running to vote when last-logged opid is not known
W20260809 15:45:49.081615 23470 leader_election.cc:341] T 0c29937a80724ddb9d07a27534a146f3 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:49.082017 23470 leader_election.cc:302] T 0c29937a80724ddb9d07a27534a146f3 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:45:49.084913 23643 raft_consensus.cc:2726] T 0c29937a80724ddb9d07a27534a146f3 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 15:45:49.087803 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae6d2c08a7a94f58ac8dd698e3fd456c" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:45:49.088960 23594 raft_consensus.cc:348] T e71a05ba1d4d42bd95c3f85f9313399e P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:45:49.089206 23353 leader_election.cc:341] T ae6d2c08a7a94f58ac8dd698e3fd456c P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:49.091722 23594 raft_consensus.cc:374] T e71a05ba1d4d42bd95c3f85f9313399e P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:49.092047 23594 raft_consensus.cc:729] T e71a05ba1d4d42bd95c3f85f9313399e P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Initialized, Role: FOLLOWER
I20260809 15:45:49.092670 23594 consensus_queue.cc:260] T e71a05ba1d4d42bd95c3f85f9313399e P 67b3aeadfa7a432abe8af4d292e04ab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:49.090538 23648 raft_consensus.cc:504] T ae6d2c08a7a94f58ac8dd698e3fd456c P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:49.093848 23593 tablet_bootstrap.cc:654] T 3296ef50be254fc29cfc77d22a8f7a23 P 65c36d164b0540f099d592f38307ee6c: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:49.094956 23594 ts_tablet_manager.cc:1428] T e71a05ba1d4d42bd95c3f85f9313399e P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent starting tablet: real 0.025s user 0.005s sys 0.001s
I20260809 15:45:49.096524 23648 leader_election.cc:288] T ae6d2c08a7a94f58ac8dd698e3fd456c P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:45:49.097007 23283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae6d2c08a7a94f58ac8dd698e3fd456c" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:45:49.097890 23283 raft_consensus.cc:3037] T ae6d2c08a7a94f58ac8dd698e3fd456c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:49.105158 23283 raft_consensus.cc:2454] T ae6d2c08a7a94f58ac8dd698e3fd456c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 1.
I20260809 15:45:49.097213 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae6d2c08a7a94f58ac8dd698e3fd456c" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:45:49.106155 23351 leader_election.cc:302] T ae6d2c08a7a94f58ac8dd698e3fd456c P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters:
W20260809 15:45:49.106664 23353 leader_election.cc:341] T ae6d2c08a7a94f58ac8dd698e3fd456c P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:49.106968 23591 tablet_bootstrap.cc:492] T 8e4ffcf83c234ae7910ddef3ec0b4bda P c0f38c1908414566aa9115bb2cbdd85d: No bootstrap required, opened a new log
I20260809 15:45:49.106914 23648 raft_consensus.cc:2781] T ae6d2c08a7a94f58ac8dd698e3fd456c P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:45:49.107297 23593 tablet_bootstrap.cc:492] T 3296ef50be254fc29cfc77d22a8f7a23 P 65c36d164b0540f099d592f38307ee6c: No bootstrap required, opened a new log
I20260809 15:45:49.107468 23591 ts_tablet_manager.cc:1397] T 8e4ffcf83c234ae7910ddef3ec0b4bda P c0f38c1908414566aa9115bb2cbdd85d: Time spent bootstrapping tablet: real 0.053s user 0.015s sys 0.018s
I20260809 15:45:49.107689 23593 ts_tablet_manager.cc:1397] T 3296ef50be254fc29cfc77d22a8f7a23 P 65c36d164b0540f099d592f38307ee6c: Time spent bootstrapping tablet: real 0.043s user 0.008s sys 0.007s
I20260809 15:45:49.109045 23648 raft_consensus.cc:686] T ae6d2c08a7a94f58ac8dd698e3fd456c P c0f38c1908414566aa9115bb2cbdd85d [term 1 LEADER]: Becoming Leader. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:45:49.109817 23591 raft_consensus.cc:348] T 8e4ffcf83c234ae7910ddef3ec0b4bda P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:49.109843 23648 consensus_queue.cc:237] T ae6d2c08a7a94f58ac8dd698e3fd456c P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:49.097472 23594 tablet_bootstrap.cc:492] T d1a134ba33ea4076bbaac0ae460c17ba P 67b3aeadfa7a432abe8af4d292e04ab9: Bootstrap starting.
I20260809 15:45:49.110462 23593 raft_consensus.cc:348] T 3296ef50be254fc29cfc77d22a8f7a23 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:49.111126 23593 raft_consensus.cc:374] T 3296ef50be254fc29cfc77d22a8f7a23 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:49.110594 23591 raft_consensus.cc:374] T 8e4ffcf83c234ae7910ddef3ec0b4bda P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:49.111428 23593 raft_consensus.cc:729] T 3296ef50be254fc29cfc77d22a8f7a23 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Initialized, Role: FOLLOWER
I20260809 15:45:49.111663 23591 raft_consensus.cc:729] T 8e4ffcf83c234ae7910ddef3ec0b4bda P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Initialized, Role: FOLLOWER
I20260809 15:45:49.112152 23593 consensus_queue.cc:260] T 3296ef50be254fc29cfc77d22a8f7a23 P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:49.112351 23591 consensus_queue.cc:260] T 8e4ffcf83c234ae7910ddef3ec0b4bda P c0f38c1908414566aa9115bb2cbdd85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:49.114984 23591 ts_tablet_manager.cc:1428] T 8e4ffcf83c234ae7910ddef3ec0b4bda P c0f38c1908414566aa9115bb2cbdd85d: Time spent starting tablet: real 0.007s user 0.004s sys 0.001s
I20260809 15:45:49.115758 23594 tablet_bootstrap.cc:654] T d1a134ba33ea4076bbaac0ae460c17ba P 67b3aeadfa7a432abe8af4d292e04ab9: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:49.115846 23591 tablet_bootstrap.cc:492] T 3296ef50be254fc29cfc77d22a8f7a23 P c0f38c1908414566aa9115bb2cbdd85d: Bootstrap starting.
I20260809 15:45:49.116957 23593 ts_tablet_manager.cc:1428] T 3296ef50be254fc29cfc77d22a8f7a23 P 65c36d164b0540f099d592f38307ee6c: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20260809 15:45:49.117805 23593 tablet_bootstrap.cc:492] T e71a05ba1d4d42bd95c3f85f9313399e P 65c36d164b0540f099d592f38307ee6c: Bootstrap starting.
I20260809 15:45:49.121373 23591 tablet_bootstrap.cc:654] T 3296ef50be254fc29cfc77d22a8f7a23 P c0f38c1908414566aa9115bb2cbdd85d: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:49.126582 23593 tablet_bootstrap.cc:654] T e71a05ba1d4d42bd95c3f85f9313399e P 65c36d164b0540f099d592f38307ee6c: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:49.132215 23594 tablet_bootstrap.cc:492] T d1a134ba33ea4076bbaac0ae460c17ba P 67b3aeadfa7a432abe8af4d292e04ab9: No bootstrap required, opened a new log
I20260809 15:45:49.132591 23594 ts_tablet_manager.cc:1397] T d1a134ba33ea4076bbaac0ae460c17ba P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent bootstrapping tablet: real 0.035s user 0.008s sys 0.013s
I20260809 15:45:49.132769 23591 tablet_bootstrap.cc:492] T 3296ef50be254fc29cfc77d22a8f7a23 P c0f38c1908414566aa9115bb2cbdd85d: No bootstrap required, opened a new log
I20260809 15:45:49.133147 23591 ts_tablet_manager.cc:1397] T 3296ef50be254fc29cfc77d22a8f7a23 P c0f38c1908414566aa9115bb2cbdd85d: Time spent bootstrapping tablet: real 0.017s user 0.011s sys 0.005s
I20260809 15:45:49.135119 23594 raft_consensus.cc:348] T d1a134ba33ea4076bbaac0ae460c17ba P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:49.135792 23594 raft_consensus.cc:374] T d1a134ba33ea4076bbaac0ae460c17ba P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:49.135596 23591 raft_consensus.cc:348] T 3296ef50be254fc29cfc77d22a8f7a23 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:49.136047 23594 raft_consensus.cc:729] T d1a134ba33ea4076bbaac0ae460c17ba P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Initialized, Role: FOLLOWER
I20260809 15:45:49.136195 23591 raft_consensus.cc:374] T 3296ef50be254fc29cfc77d22a8f7a23 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:49.136478 23591 raft_consensus.cc:729] T 3296ef50be254fc29cfc77d22a8f7a23 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Initialized, Role: FOLLOWER
I20260809 15:45:49.136606 23594 consensus_queue.cc:260] T d1a134ba33ea4076bbaac0ae460c17ba P 67b3aeadfa7a432abe8af4d292e04ab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:49.136333 23118 catalog_manager.cc:5495] T ae6d2c08a7a94f58ac8dd698e3fd456c P c0f38c1908414566aa9115bb2cbdd85d reported cstate change: term changed from 0 to 1, leader changed from <none> to c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66). New cstate: current_term: 1 leader_uuid: "c0f38c1908414566aa9115bb2cbdd85d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } health_report { overall_health: UNKNOWN } } }
I20260809 15:45:49.137056 23591 consensus_queue.cc:260] T 3296ef50be254fc29cfc77d22a8f7a23 P c0f38c1908414566aa9115bb2cbdd85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:49.138950 23591 ts_tablet_manager.cc:1428] T 3296ef50be254fc29cfc77d22a8f7a23 P c0f38c1908414566aa9115bb2cbdd85d: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20260809 15:45:49.139170 23790 consensus_queue.cc:1035] T 6ced718c1a5c4da79cb8c1ebd1435b87 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:45:49.139740 23591 tablet_bootstrap.cc:492] T 5712f6b6fffc4cc5a8ddfbe63e4097df P c0f38c1908414566aa9115bb2cbdd85d: Bootstrap starting.
I20260809 15:45:49.143782 23594 ts_tablet_manager.cc:1428] T d1a134ba33ea4076bbaac0ae460c17ba P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent starting tablet: real 0.011s user 0.002s sys 0.005s
I20260809 15:45:49.144677 23594 tablet_bootstrap.cc:492] T d2d4f749fcfd4fb3bfb8115af3471598 P 67b3aeadfa7a432abe8af4d292e04ab9: Bootstrap starting.
I20260809 15:45:49.145388 23591 tablet_bootstrap.cc:654] T 5712f6b6fffc4cc5a8ddfbe63e4097df P c0f38c1908414566aa9115bb2cbdd85d: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:49.150831 23594 tablet_bootstrap.cc:654] T d2d4f749fcfd4fb3bfb8115af3471598 P 67b3aeadfa7a432abe8af4d292e04ab9: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:49.151750 23648 raft_consensus.cc:482] T 8e4ffcf83c234ae7910ddef3ec0b4bda P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:45:49.152182 23648 raft_consensus.cc:504] T 8e4ffcf83c234ae7910ddef3ec0b4bda P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:49.153996 23648 leader_election.cc:288] T 8e4ffcf83c234ae7910ddef3ec0b4bda P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:45:49.155365 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e4ffcf83c234ae7910ddef3ec0b4bda" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:45:49.146742 23593 tablet_bootstrap.cc:492] T e71a05ba1d4d42bd95c3f85f9313399e P 65c36d164b0540f099d592f38307ee6c: No bootstrap required, opened a new log
I20260809 15:45:49.156628 23593 ts_tablet_manager.cc:1397] T e71a05ba1d4d42bd95c3f85f9313399e P 65c36d164b0540f099d592f38307ee6c: Time spent bootstrapping tablet: real 0.039s user 0.009s sys 0.001s
I20260809 15:45:49.157442 23282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e4ffcf83c234ae7910ddef3ec0b4bda" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:45:49.158820 23351 leader_election.cc:302] T 8e4ffcf83c234ae7910ddef3ec0b4bda P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:45:49.159062 23593 raft_consensus.cc:348] T e71a05ba1d4d42bd95c3f85f9313399e P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:49.159847 23593 raft_consensus.cc:374] T e71a05ba1d4d42bd95c3f85f9313399e P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:49.159868 23648 raft_consensus.cc:2726] T 8e4ffcf83c234ae7910ddef3ec0b4bda P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 15:45:49.160341 23593 raft_consensus.cc:729] T e71a05ba1d4d42bd95c3f85f9313399e P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Initialized, Role: FOLLOWER
I20260809 15:45:49.161092 23593 consensus_queue.cc:260] T e71a05ba1d4d42bd95c3f85f9313399e P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:49.166249 23593 ts_tablet_manager.cc:1428] T e71a05ba1d4d42bd95c3f85f9313399e P 65c36d164b0540f099d592f38307ee6c: Time spent starting tablet: real 0.009s user 0.005s sys 0.001s
I20260809 15:45:49.167105 23593 tablet_bootstrap.cc:492] T 99bdb93503db4de4a1cfb0161ea0c07c P 65c36d164b0540f099d592f38307ee6c: Bootstrap starting.
I20260809 15:45:49.170002 23591 tablet_bootstrap.cc:492] T 5712f6b6fffc4cc5a8ddfbe63e4097df P c0f38c1908414566aa9115bb2cbdd85d: No bootstrap required, opened a new log
I20260809 15:45:49.170425 23591 ts_tablet_manager.cc:1397] T 5712f6b6fffc4cc5a8ddfbe63e4097df P c0f38c1908414566aa9115bb2cbdd85d: Time spent bootstrapping tablet: real 0.031s user 0.013s sys 0.011s
I20260809 15:45:49.172327 23593 tablet_bootstrap.cc:654] T 99bdb93503db4de4a1cfb0161ea0c07c P 65c36d164b0540f099d592f38307ee6c: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:49.172640 23591 raft_consensus.cc:348] T 5712f6b6fffc4cc5a8ddfbe63e4097df P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:49.173270 23591 raft_consensus.cc:374] T 5712f6b6fffc4cc5a8ddfbe63e4097df P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:49.173447 23594 tablet_bootstrap.cc:492] T d2d4f749fcfd4fb3bfb8115af3471598 P 67b3aeadfa7a432abe8af4d292e04ab9: No bootstrap required, opened a new log
I20260809 15:45:49.173527 23591 raft_consensus.cc:729] T 5712f6b6fffc4cc5a8ddfbe63e4097df P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Initialized, Role: FOLLOWER
I20260809 15:45:49.173866 23594 ts_tablet_manager.cc:1397] T d2d4f749fcfd4fb3bfb8115af3471598 P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent bootstrapping tablet: real 0.029s user 0.009s sys 0.003s
I20260809 15:45:49.174227 23591 consensus_queue.cc:260] T 5712f6b6fffc4cc5a8ddfbe63e4097df P c0f38c1908414566aa9115bb2cbdd85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:49.176473 23594 raft_consensus.cc:348] T d2d4f749fcfd4fb3bfb8115af3471598 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:49.177100 23594 raft_consensus.cc:374] T d2d4f749fcfd4fb3bfb8115af3471598 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:49.177328 23594 raft_consensus.cc:729] T d2d4f749fcfd4fb3bfb8115af3471598 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Initialized, Role: FOLLOWER
I20260809 15:45:49.177858 23594 consensus_queue.cc:260] T d2d4f749fcfd4fb3bfb8115af3471598 P 67b3aeadfa7a432abe8af4d292e04ab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:49.180461 23591 ts_tablet_manager.cc:1428] T 5712f6b6fffc4cc5a8ddfbe63e4097df P c0f38c1908414566aa9115bb2cbdd85d: Time spent starting tablet: real 0.010s user 0.005s sys 0.001s
I20260809 15:45:49.181270 23591 tablet_bootstrap.cc:492] T 6ae14a74a26f434db08c652e097385c8 P c0f38c1908414566aa9115bb2cbdd85d: Bootstrap starting.
I20260809 15:45:49.183840 23594 ts_tablet_manager.cc:1428] T d2d4f749fcfd4fb3bfb8115af3471598 P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent starting tablet: real 0.010s user 0.006s sys 0.001s
I20260809 15:45:49.184731 23594 tablet_bootstrap.cc:492] T 0a51c0baaa6e432997c9d3552af4f254 P 67b3aeadfa7a432abe8af4d292e04ab9: Bootstrap starting.
I20260809 15:45:49.186405 23591 tablet_bootstrap.cc:654] T 6ae14a74a26f434db08c652e097385c8 P c0f38c1908414566aa9115bb2cbdd85d: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:49.189523 23594 tablet_bootstrap.cc:654] T 0a51c0baaa6e432997c9d3552af4f254 P 67b3aeadfa7a432abe8af4d292e04ab9: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:49.196542 23593 tablet_bootstrap.cc:492] T 99bdb93503db4de4a1cfb0161ea0c07c P 65c36d164b0540f099d592f38307ee6c: No bootstrap required, opened a new log
I20260809 15:45:49.196859 23593 ts_tablet_manager.cc:1397] T 99bdb93503db4de4a1cfb0161ea0c07c P 65c36d164b0540f099d592f38307ee6c: Time spent bootstrapping tablet: real 0.030s user 0.006s sys 0.009s
I20260809 15:45:49.200214 23593 raft_consensus.cc:348] T 99bdb93503db4de4a1cfb0161ea0c07c P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:49.200840 23593 raft_consensus.cc:374] T 99bdb93503db4de4a1cfb0161ea0c07c P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:49.201077 23593 raft_consensus.cc:729] T 99bdb93503db4de4a1cfb0161ea0c07c P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Initialized, Role: FOLLOWER
I20260809 15:45:49.201637 23593 consensus_queue.cc:260] T 99bdb93503db4de4a1cfb0161ea0c07c P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:49.203696 23593 ts_tablet_manager.cc:1428] T 99bdb93503db4de4a1cfb0161ea0c07c P 65c36d164b0540f099d592f38307ee6c: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260809 15:45:49.204255 23661 raft_consensus.cc:482] T d2d4f749fcfd4fb3bfb8115af3471598 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:45:49.204608 23594 tablet_bootstrap.cc:492] T 0a51c0baaa6e432997c9d3552af4f254 P 67b3aeadfa7a432abe8af4d292e04ab9: No bootstrap required, opened a new log
I20260809 15:45:49.204730 23593 tablet_bootstrap.cc:492] T ae6d2c08a7a94f58ac8dd698e3fd456c P 65c36d164b0540f099d592f38307ee6c: Bootstrap starting.
I20260809 15:45:49.205107 23594 ts_tablet_manager.cc:1397] T 0a51c0baaa6e432997c9d3552af4f254 P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent bootstrapping tablet: real 0.021s user 0.010s sys 0.007s
I20260809 15:45:49.204792 23661 raft_consensus.cc:504] T d2d4f749fcfd4fb3bfb8115af3471598 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:49.206676 23591 tablet_bootstrap.cc:492] T 6ae14a74a26f434db08c652e097385c8 P c0f38c1908414566aa9115bb2cbdd85d: No bootstrap required, opened a new log
I20260809 15:45:49.207067 23591 ts_tablet_manager.cc:1397] T 6ae14a74a26f434db08c652e097385c8 P c0f38c1908414566aa9115bb2cbdd85d: Time spent bootstrapping tablet: real 0.026s user 0.001s sys 0.013s
I20260809 15:45:49.208794 23594 raft_consensus.cc:348] T 0a51c0baaa6e432997c9d3552af4f254 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:49.209453 23594 raft_consensus.cc:374] T 0a51c0baaa6e432997c9d3552af4f254 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:49.209721 23594 raft_consensus.cc:729] T 0a51c0baaa6e432997c9d3552af4f254 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Initialized, Role: FOLLOWER
I20260809 15:45:49.210422 23594 consensus_queue.cc:260] T 0a51c0baaa6e432997c9d3552af4f254 P 67b3aeadfa7a432abe8af4d292e04ab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:49.210804 23591 raft_consensus.cc:348] T 6ae14a74a26f434db08c652e097385c8 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:49.211614 23591 raft_consensus.cc:374] T 6ae14a74a26f434db08c652e097385c8 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:49.211903 23591 raft_consensus.cc:729] T 6ae14a74a26f434db08c652e097385c8 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Initialized, Role: FOLLOWER
I20260809 15:45:49.213037 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2d4f749fcfd4fb3bfb8115af3471598" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:45:49.212853 23591 consensus_queue.cc:260] T 6ae14a74a26f434db08c652e097385c8 P c0f38c1908414566aa9115bb2cbdd85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:49.213587 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2d4f749fcfd4fb3bfb8115af3471598" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:45:49.215138 23594 ts_tablet_manager.cc:1428] T 0a51c0baaa6e432997c9d3552af4f254 P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent starting tablet: real 0.010s user 0.003s sys 0.003s
I20260809 15:45:49.215335 23591 ts_tablet_manager.cc:1428] T 6ae14a74a26f434db08c652e097385c8 P c0f38c1908414566aa9115bb2cbdd85d: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
W20260809 15:45:49.215673 23228 leader_election.cc:341] T d2d4f749fcfd4fb3bfb8115af3471598 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:49.215971 23594 tablet_bootstrap.cc:492] T 73f6f44c259547f1b05a3a434616fa54 P 67b3aeadfa7a432abe8af4d292e04ab9: Bootstrap starting.
I20260809 15:45:49.216449 23591 tablet_bootstrap.cc:492] T 99bdb93503db4de4a1cfb0161ea0c07c P c0f38c1908414566aa9115bb2cbdd85d: Bootstrap starting.
I20260809 15:45:49.216882 23593 tablet_bootstrap.cc:654] T ae6d2c08a7a94f58ac8dd698e3fd456c P 65c36d164b0540f099d592f38307ee6c: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:49.217655 23722 raft_consensus.cc:482] T d1a134ba33ea4076bbaac0ae460c17ba P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:45:49.217985 23722 raft_consensus.cc:504] T d1a134ba33ea4076bbaac0ae460c17ba P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:49.219383 23722 leader_election.cc:288] T d1a134ba33ea4076bbaac0ae460c17ba P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:45:49.220371 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1a134ba33ea4076bbaac0ae460c17ba" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:45:49.221239 23661 leader_election.cc:288] T d2d4f749fcfd4fb3bfb8115af3471598 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:45:49.221501 23399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1a134ba33ea4076bbaac0ae460c17ba" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
W20260809 15:45:49.222527 23228 leader_election.cc:341] T d1a134ba33ea4076bbaac0ae460c17ba P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553): Illegal state: must be running to vote when last-logged opid is not known
W20260809 15:45:49.223206 23228 leader_election.cc:341] T d2d4f749fcfd4fb3bfb8115af3471598 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:49.223642 23228 leader_election.cc:302] T d2d4f749fcfd4fb3bfb8115af3471598 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:45:49.225457 23661 raft_consensus.cc:2726] T d2d4f749fcfd4fb3bfb8115af3471598 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260809 15:45:49.226223 23228 leader_election.cc:341] T d1a134ba33ea4076bbaac0ae460c17ba P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:49.226643 23228 leader_election.cc:302] T d1a134ba33ea4076bbaac0ae460c17ba P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:45:49.227692 23661 raft_consensus.cc:2726] T d1a134ba33ea4076bbaac0ae460c17ba P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 15:45:49.228515 23594 tablet_bootstrap.cc:654] T 73f6f44c259547f1b05a3a434616fa54 P 67b3aeadfa7a432abe8af4d292e04ab9: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:49.234678 23591 tablet_bootstrap.cc:654] T 99bdb93503db4de4a1cfb0161ea0c07c P c0f38c1908414566aa9115bb2cbdd85d: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:49.236521 23648 raft_consensus.cc:482] T 5712f6b6fffc4cc5a8ddfbe63e4097df P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:45:49.236985 23648 raft_consensus.cc:504] T 5712f6b6fffc4cc5a8ddfbe63e4097df P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:49.238880 23648 leader_election.cc:288] T 5712f6b6fffc4cc5a8ddfbe63e4097df P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:45:49.240368 23283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5712f6b6fffc4cc5a8ddfbe63e4097df" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:45:49.242414 23591 tablet_bootstrap.cc:492] T 99bdb93503db4de4a1cfb0161ea0c07c P c0f38c1908414566aa9115bb2cbdd85d: No bootstrap required, opened a new log
I20260809 15:45:49.242841 23591 ts_tablet_manager.cc:1397] T 99bdb93503db4de4a1cfb0161ea0c07c P c0f38c1908414566aa9115bb2cbdd85d: Time spent bootstrapping tablet: real 0.027s user 0.008s sys 0.004s
I20260809 15:45:49.242709 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5712f6b6fffc4cc5a8ddfbe63e4097df" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:45:49.244180 23353 leader_election.cc:302] T 5712f6b6fffc4cc5a8ddfbe63e4097df P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:45:49.244704 23283 raft_consensus.cc:3037] T 472816994d814cb198b69e5248697440 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:49.244922 23648 raft_consensus.cc:2726] T 5712f6b6fffc4cc5a8ddfbe63e4097df P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 15:45:49.245589 23591 raft_consensus.cc:348] T 99bdb93503db4de4a1cfb0161ea0c07c P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:49.246089 23591 raft_consensus.cc:374] T 99bdb93503db4de4a1cfb0161ea0c07c P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:49.246279 23593 tablet_bootstrap.cc:492] T ae6d2c08a7a94f58ac8dd698e3fd456c P 65c36d164b0540f099d592f38307ee6c: No bootstrap required, opened a new log
I20260809 15:45:49.246280 23591 raft_consensus.cc:729] T 99bdb93503db4de4a1cfb0161ea0c07c P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Initialized, Role: FOLLOWER
I20260809 15:45:49.246827 23593 ts_tablet_manager.cc:1397] T ae6d2c08a7a94f58ac8dd698e3fd456c P 65c36d164b0540f099d592f38307ee6c: Time spent bootstrapping tablet: real 0.042s user 0.002s sys 0.030s
I20260809 15:45:49.246999 23591 consensus_queue.cc:260] T 99bdb93503db4de4a1cfb0161ea0c07c P c0f38c1908414566aa9115bb2cbdd85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:49.249054 23591 ts_tablet_manager.cc:1428] T 99bdb93503db4de4a1cfb0161ea0c07c P c0f38c1908414566aa9115bb2cbdd85d: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260809 15:45:49.249030 23593 raft_consensus.cc:348] T ae6d2c08a7a94f58ac8dd698e3fd456c P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:49.250247 23593 raft_consensus.cc:374] T ae6d2c08a7a94f58ac8dd698e3fd456c P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:49.250447 23591 tablet_bootstrap.cc:492] T 0a51c0baaa6e432997c9d3552af4f254 P c0f38c1908414566aa9115bb2cbdd85d: Bootstrap starting.
I20260809 15:45:49.250519 23593 raft_consensus.cc:729] T ae6d2c08a7a94f58ac8dd698e3fd456c P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Initialized, Role: FOLLOWER
I20260809 15:45:49.251314 23593 consensus_queue.cc:260] T ae6d2c08a7a94f58ac8dd698e3fd456c P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:49.255581 23591 tablet_bootstrap.cc:654] T 0a51c0baaa6e432997c9d3552af4f254 P c0f38c1908414566aa9115bb2cbdd85d: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:49.256327 23593 ts_tablet_manager.cc:1428] T ae6d2c08a7a94f58ac8dd698e3fd456c P 65c36d164b0540f099d592f38307ee6c: Time spent starting tablet: real 0.009s user 0.001s sys 0.006s
I20260809 15:45:49.257182 23593 tablet_bootstrap.cc:492] T 73f6f44c259547f1b05a3a434616fa54 P 65c36d164b0540f099d592f38307ee6c: Bootstrap starting.
I20260809 15:45:49.258975 23594 tablet_bootstrap.cc:492] T 73f6f44c259547f1b05a3a434616fa54 P 67b3aeadfa7a432abe8af4d292e04ab9: No bootstrap required, opened a new log
I20260809 15:45:49.259419 23594 ts_tablet_manager.cc:1397] T 73f6f44c259547f1b05a3a434616fa54 P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent bootstrapping tablet: real 0.044s user 0.009s sys 0.001s
I20260809 15:45:49.261713 23594 raft_consensus.cc:348] T 73f6f44c259547f1b05a3a434616fa54 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:49.262347 23594 raft_consensus.cc:374] T 73f6f44c259547f1b05a3a434616fa54 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:49.262598 23594 raft_consensus.cc:729] T 73f6f44c259547f1b05a3a434616fa54 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Initialized, Role: FOLLOWER
I20260809 15:45:49.263355 23594 consensus_queue.cc:260] T 73f6f44c259547f1b05a3a434616fa54 P 67b3aeadfa7a432abe8af4d292e04ab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:49.265321 23643 consensus_queue.cc:1035] T e14d118b2eaa42aa8a7cd4025aba7128 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:45:49.265655 23594 ts_tablet_manager.cc:1428] T 73f6f44c259547f1b05a3a434616fa54 P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260809 15:45:49.267699 23594 tablet_bootstrap.cc:492] T 0da4d5283c8a46878bd39b85e5c71f66 P 67b3aeadfa7a432abe8af4d292e04ab9: Bootstrap starting.
I20260809 15:45:49.273216 23594 tablet_bootstrap.cc:654] T 0da4d5283c8a46878bd39b85e5c71f66 P 67b3aeadfa7a432abe8af4d292e04ab9: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:49.275476 23593 tablet_bootstrap.cc:654] T 73f6f44c259547f1b05a3a434616fa54 P 65c36d164b0540f099d592f38307ee6c: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:49.276597 23661 raft_consensus.cc:482] T 3296ef50be254fc29cfc77d22a8f7a23 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:45:49.277668 23661 raft_consensus.cc:504] T 3296ef50be254fc29cfc77d22a8f7a23 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:49.280462 23661 leader_election.cc:288] T 3296ef50be254fc29cfc77d22a8f7a23 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:45:49.282112 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3296ef50be254fc29cfc77d22a8f7a23" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:45:49.282694 23527 raft_consensus.cc:2454] T 3296ef50be254fc29cfc77d22a8f7a23 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 0.
I20260809 15:45:49.283815 23399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3296ef50be254fc29cfc77d22a8f7a23" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:45:49.284574 23399 raft_consensus.cc:2454] T 3296ef50be254fc29cfc77d22a8f7a23 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 0.
I20260809 15:45:49.287760 23228 leader_election.cc:302] T 3296ef50be254fc29cfc77d22a8f7a23 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:45:49.289352 23661 raft_consensus.cc:2781] T 3296ef50be254fc29cfc77d22a8f7a23 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 15:45:49.289685 23661 raft_consensus.cc:482] T 3296ef50be254fc29cfc77d22a8f7a23 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:45:49.290066 23661 raft_consensus.cc:3037] T 3296ef50be254fc29cfc77d22a8f7a23 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:49.296422 23661 raft_consensus.cc:504] T 3296ef50be254fc29cfc77d22a8f7a23 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:49.299007 23399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3296ef50be254fc29cfc77d22a8f7a23" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:45:49.299613 23661 leader_election.cc:288] T 3296ef50be254fc29cfc77d22a8f7a23 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:45:49.299762 23399 raft_consensus.cc:3037] T 3296ef50be254fc29cfc77d22a8f7a23 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:49.303489 23591 tablet_bootstrap.cc:492] T 0a51c0baaa6e432997c9d3552af4f254 P c0f38c1908414566aa9115bb2cbdd85d: No bootstrap required, opened a new log
I20260809 15:45:49.303632 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3296ef50be254fc29cfc77d22a8f7a23" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:45:49.303884 23591 ts_tablet_manager.cc:1397] T 0a51c0baaa6e432997c9d3552af4f254 P c0f38c1908414566aa9115bb2cbdd85d: Time spent bootstrapping tablet: real 0.054s user 0.019s sys 0.015s
I20260809 15:45:49.304869 23655 consensus_queue.cc:1035] T e14d118b2eaa42aa8a7cd4025aba7128 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:45:49.305394 23594 tablet_bootstrap.cc:492] T 0da4d5283c8a46878bd39b85e5c71f66 P 67b3aeadfa7a432abe8af4d292e04ab9: No bootstrap required, opened a new log
I20260809 15:45:49.305783 23594 ts_tablet_manager.cc:1397] T 0da4d5283c8a46878bd39b85e5c71f66 P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent bootstrapping tablet: real 0.038s user 0.014s sys 0.002s
I20260809 15:45:49.307751 23591 raft_consensus.cc:348] T 0a51c0baaa6e432997c9d3552af4f254 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:49.308367 23591 raft_consensus.cc:374] T 0a51c0baaa6e432997c9d3552af4f254 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:49.308617 23591 raft_consensus.cc:729] T 0a51c0baaa6e432997c9d3552af4f254 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Initialized, Role: FOLLOWER
I20260809 15:45:49.309172 23591 consensus_queue.cc:260] T 0a51c0baaa6e432997c9d3552af4f254 P c0f38c1908414566aa9115bb2cbdd85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:45:49.310761 23228 consensus_peers.cc:487] T 7baa2492df0340c4a5ac241959ce1ac3 P 67b3aeadfa7a432abe8af4d292e04ab9 -> Peer 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553): Couldn't send request to peer 65c36d164b0540f099d592f38307ee6c. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260809 15:45:49.311043 23591 ts_tablet_manager.cc:1428] T 0a51c0baaa6e432997c9d3552af4f254 P c0f38c1908414566aa9115bb2cbdd85d: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260809 15:45:49.304189 23527 raft_consensus.cc:3037] T 3296ef50be254fc29cfc77d22a8f7a23 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:49.316967 23594 raft_consensus.cc:348] T 0da4d5283c8a46878bd39b85e5c71f66 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:49.321468 23594 raft_consensus.cc:374] T 0da4d5283c8a46878bd39b85e5c71f66 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:49.321718 23594 raft_consensus.cc:729] T 0da4d5283c8a46878bd39b85e5c71f66 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Initialized, Role: FOLLOWER
I20260809 15:45:49.322358 23594 consensus_queue.cc:260] T 0da4d5283c8a46878bd39b85e5c71f66 P 67b3aeadfa7a432abe8af4d292e04ab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:49.324458 23594 ts_tablet_manager.cc:1428] T 0da4d5283c8a46878bd39b85e5c71f66 P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent starting tablet: real 0.018s user 0.004s sys 0.003s
I20260809 15:45:49.318956 23399 raft_consensus.cc:2454] T 3296ef50be254fc29cfc77d22a8f7a23 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 1.
I20260809 15:45:49.325389 23228 leader_election.cc:302] T 3296ef50be254fc29cfc77d22a8f7a23 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:45:49.326136 23661 raft_consensus.cc:2781] T 3296ef50be254fc29cfc77d22a8f7a23 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:45:49.321138 23591 tablet_bootstrap.cc:492] T d2d4f749fcfd4fb3bfb8115af3471598 P c0f38c1908414566aa9115bb2cbdd85d: Bootstrap starting.
I20260809 15:45:49.326638 23664 raft_consensus.cc:482] T 6ae14a74a26f434db08c652e097385c8 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:45:49.326713 23661 raft_consensus.cc:686] T 3296ef50be254fc29cfc77d22a8f7a23 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 LEADER]: Becoming Leader. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Running, Role: LEADER
I20260809 15:45:49.327193 23664 raft_consensus.cc:504] T 6ae14a74a26f434db08c652e097385c8 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:49.327605 23661 consensus_queue.cc:237] T 3296ef50be254fc29cfc77d22a8f7a23 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:49.329161 23664 leader_election.cc:288] T 6ae14a74a26f434db08c652e097385c8 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:45:49.330554 23527 raft_consensus.cc:2454] T 3296ef50be254fc29cfc77d22a8f7a23 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 1.
I20260809 15:45:49.330626 23529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ae14a74a26f434db08c652e097385c8" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
W20260809 15:45:49.335211 23353 leader_election.cc:341] T 6ae14a74a26f434db08c652e097385c8 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:49.332481 23282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ae14a74a26f434db08c652e097385c8" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:45:49.339161 23591 tablet_bootstrap.cc:654] T d2d4f749fcfd4fb3bfb8115af3471598 P c0f38c1908414566aa9115bb2cbdd85d: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:49.351174 23593 tablet_bootstrap.cc:492] T 73f6f44c259547f1b05a3a434616fa54 P 65c36d164b0540f099d592f38307ee6c: No bootstrap required, opened a new log
I20260809 15:45:49.351559 23593 ts_tablet_manager.cc:1397] T 73f6f44c259547f1b05a3a434616fa54 P 65c36d164b0540f099d592f38307ee6c: Time spent bootstrapping tablet: real 0.095s user 0.019s sys 0.029s
I20260809 15:45:49.351959 23722 consensus_queue.cc:1035] T 7baa2492df0340c4a5ac241959ce1ac3 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260809 15:45:49.354861 23593 raft_consensus.cc:348] T 73f6f44c259547f1b05a3a434616fa54 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:49.355517 23593 raft_consensus.cc:374] T 73f6f44c259547f1b05a3a434616fa54 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:49.355762 23593 raft_consensus.cc:729] T 73f6f44c259547f1b05a3a434616fa54 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Initialized, Role: FOLLOWER
I20260809 15:45:49.356420 23593 consensus_queue.cc:260] T 73f6f44c259547f1b05a3a434616fa54 P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:49.358144 23594 tablet_bootstrap.cc:492] T 6ae14a74a26f434db08c652e097385c8 P 67b3aeadfa7a432abe8af4d292e04ab9: Bootstrap starting.
I20260809 15:45:49.358585 23593 ts_tablet_manager.cc:1428] T 73f6f44c259547f1b05a3a434616fa54 P 65c36d164b0540f099d592f38307ee6c: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
W20260809 15:45:49.358580 23351 leader_election.cc:341] T 6ae14a74a26f434db08c652e097385c8 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:49.362267 23351 leader_election.cc:302] T 6ae14a74a26f434db08c652e097385c8 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:45:49.363432 23664 raft_consensus.cc:2726] T 6ae14a74a26f434db08c652e097385c8 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 15:45:49.364265 23691 raft_consensus.cc:482] T 73f6f44c259547f1b05a3a434616fa54 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:45:49.364857 23691 raft_consensus.cc:504] T 73f6f44c259547f1b05a3a434616fa54 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:49.369380 23664 raft_consensus.cc:482] T e71a05ba1d4d42bd95c3f85f9313399e P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:45:49.369586 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73f6f44c259547f1b05a3a434616fa54" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:45:49.369817 23664 raft_consensus.cc:504] T e71a05ba1d4d42bd95c3f85f9313399e P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:49.370193 23402 raft_consensus.cc:2454] T 73f6f44c259547f1b05a3a434616fa54 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 0.
I20260809 15:45:49.371661 23664 leader_election.cc:288] T e71a05ba1d4d42bd95c3f85f9313399e P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:45:49.373181 23529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e71a05ba1d4d42bd95c3f85f9313399e" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:45:49.373670 23643 raft_consensus.cc:482] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:45:49.373811 23529 raft_consensus.cc:2454] T e71a05ba1d4d42bd95c3f85f9313399e P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 0.
I20260809 15:45:49.375483 23282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e71a05ba1d4d42bd95c3f85f9313399e" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:45:49.375584 23353 leader_election.cc:302] T e71a05ba1d4d42bd95c3f85f9313399e P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:45:49.376109 23282 raft_consensus.cc:2454] T e71a05ba1d4d42bd95c3f85f9313399e P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 0.
I20260809 15:45:49.377426 23643 raft_consensus.cc:504] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:49.378820 23664 raft_consensus.cc:2781] T e71a05ba1d4d42bd95c3f85f9313399e P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 15:45:49.379133 23691 leader_election.cc:288] T 73f6f44c259547f1b05a3a434616fa54 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:45:49.379245 23643 leader_election.cc:288] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:45:49.379257 23664 raft_consensus.cc:482] T e71a05ba1d4d42bd95c3f85f9313399e P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:45:49.380360 23593 tablet_bootstrap.cc:492] T 0da4d5283c8a46878bd39b85e5c71f66 P 65c36d164b0540f099d592f38307ee6c: Bootstrap starting.
I20260809 15:45:49.380786 23664 raft_consensus.cc:3037] T e71a05ba1d4d42bd95c3f85f9313399e P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:49.391801 23282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ced718c1a5c4da79cb8c1ebd1435b87" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:45:49.397701 23283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73f6f44c259547f1b05a3a434616fa54" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:45:49.397871 23591 tablet_bootstrap.cc:492] T d2d4f749fcfd4fb3bfb8115af3471598 P c0f38c1908414566aa9115bb2cbdd85d: No bootstrap required, opened a new log
I20260809 15:45:49.453522 23591 ts_tablet_manager.cc:1397] T d2d4f749fcfd4fb3bfb8115af3471598 P c0f38c1908414566aa9115bb2cbdd85d: Time spent bootstrapping tablet: real 0.133s user 0.013s sys 0.014s
I20260809 15:45:49.452219 23594 tablet_bootstrap.cc:654] T 6ae14a74a26f434db08c652e097385c8 P 67b3aeadfa7a432abe8af4d292e04ab9: Neither blocks nor log segments found. Creating new log.
W20260809 15:45:49.452963 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 75859.8us
I20260809 15:45:49.426757 23664 raft_consensus.cc:504] T e71a05ba1d4d42bd95c3f85f9313399e P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:45:49.441299 23193 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260809 15:45:49.465133 23593 tablet_bootstrap.cc:654] T 0da4d5283c8a46878bd39b85e5c71f66 P 65c36d164b0540f099d592f38307ee6c: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:49.456615 23283 raft_consensus.cc:2454] T 73f6f44c259547f1b05a3a434616fa54 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 0.
I20260809 15:45:49.465636 23118 catalog_manager.cc:5495] T 3296ef50be254fc29cfc77d22a8f7a23 P 67b3aeadfa7a432abe8af4d292e04ab9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65). New cstate: current_term: 1 leader_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } health_report { overall_health: UNKNOWN } } }
I20260809 15:45:49.470907 23664 leader_election.cc:288] T e71a05ba1d4d42bd95c3f85f9313399e P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:45:49.453723 23472 leader_election.cc:302] T 73f6f44c259547f1b05a3a434616fa54 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:45:49.488349 23282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e71a05ba1d4d42bd95c3f85f9313399e" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:45:49.488912 23282 raft_consensus.cc:3037] T e71a05ba1d4d42bd95c3f85f9313399e P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:49.491631 23528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e71a05ba1d4d42bd95c3f85f9313399e" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:45:49.492198 23528 raft_consensus.cc:3037] T e71a05ba1d4d42bd95c3f85f9313399e P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:49.501291 23282 raft_consensus.cc:2454] T e71a05ba1d4d42bd95c3f85f9313399e P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 1.
I20260809 15:45:49.503600 23648 raft_consensus.cc:482] T 99bdb93503db4de4a1cfb0161ea0c07c P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:45:49.505618 23691 raft_consensus.cc:2781] T 73f6f44c259547f1b05a3a434616fa54 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 15:45:49.508263 23691 raft_consensus.cc:482] T 73f6f44c259547f1b05a3a434616fa54 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:45:49.508615 23691 raft_consensus.cc:3037] T 73f6f44c259547f1b05a3a434616fa54 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:49.511713 23643 raft_consensus.cc:482] T 99bdb93503db4de4a1cfb0161ea0c07c P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:45:49.508920 23648 raft_consensus.cc:504] T 99bdb93503db4de4a1cfb0161ea0c07c P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:49.512465 23643 raft_consensus.cc:504] T 99bdb93503db4de4a1cfb0161ea0c07c P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:49.514110 23643 leader_election.cc:288] T 99bdb93503db4de4a1cfb0161ea0c07c P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:45:49.514621 23351 leader_election.cc:302] T e71a05ba1d4d42bd95c3f85f9313399e P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:45:49.516233 23790 raft_consensus.cc:2781] T e71a05ba1d4d42bd95c3f85f9313399e P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:45:49.516678 23790 raft_consensus.cc:686] T e71a05ba1d4d42bd95c3f85f9313399e P c0f38c1908414566aa9115bb2cbdd85d [term 1 LEADER]: Becoming Leader. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:45:49.517227 23648 leader_election.cc:288] T 99bdb93503db4de4a1cfb0161ea0c07c P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
W20260809 15:45:49.517629 23472 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 64550.7us
I20260809 15:45:49.517418 23790 consensus_queue.cc:237] T e71a05ba1d4d42bd95c3f85f9313399e P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:49.518728 23402 raft_consensus.cc:3037] T 8e4ffcf83c234ae7910ddef3ec0b4bda P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:49.520358 23691 raft_consensus.cc:504] T 73f6f44c259547f1b05a3a434616fa54 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:49.522522 23282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99bdb93503db4de4a1cfb0161ea0c07c" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:45:49.523994 23283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99bdb93503db4de4a1cfb0161ea0c07c" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:45:49.525318 23528 raft_consensus.cc:2454] T e71a05ba1d4d42bd95c3f85f9313399e P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 1.
I20260809 15:45:49.522298 23691 leader_election.cc:288] T 73f6f44c259547f1b05a3a434616fa54 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 1 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:45:49.528965 23283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73f6f44c259547f1b05a3a434616fa54" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:45:49.529502 23283 raft_consensus.cc:3037] T 73f6f44c259547f1b05a3a434616fa54 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:49.531154 23529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99bdb93503db4de4a1cfb0161ea0c07c" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:45:49.532788 23529 raft_consensus.cc:2454] T 99bdb93503db4de4a1cfb0161ea0c07c P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 0.
I20260809 15:45:49.532706 23591 raft_consensus.cc:348] T d2d4f749fcfd4fb3bfb8115af3471598 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:49.533563 23591 raft_consensus.cc:374] T d2d4f749fcfd4fb3bfb8115af3471598 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:49.533826 23591 raft_consensus.cc:729] T d2d4f749fcfd4fb3bfb8115af3471598 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Initialized, Role: FOLLOWER
I20260809 15:45:49.533972 23353 leader_election.cc:302] T 99bdb93503db4de4a1cfb0161ea0c07c P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:45:49.534395 23591 consensus_queue.cc:260] T d2d4f749fcfd4fb3bfb8115af3471598 P c0f38c1908414566aa9115bb2cbdd85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:49.534986 23404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ced718c1a5c4da79cb8c1ebd1435b87" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:45:49.535331 23648 raft_consensus.cc:2781] T 99bdb93503db4de4a1cfb0161ea0c07c P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 15:45:49.535674 23648 raft_consensus.cc:482] T 99bdb93503db4de4a1cfb0161ea0c07c P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:45:49.535983 23648 raft_consensus.cc:3037] T 99bdb93503db4de4a1cfb0161ea0c07c P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:49.536671 23400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99bdb93503db4de4a1cfb0161ea0c07c" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:45:49.539551 23397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73f6f44c259547f1b05a3a434616fa54" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:45:49.540211 23397 raft_consensus.cc:3037] T 73f6f44c259547f1b05a3a434616fa54 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:49.542053 23283 raft_consensus.cc:2454] T 73f6f44c259547f1b05a3a434616fa54 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 1.
I20260809 15:45:49.544652 23591 ts_tablet_manager.cc:1428] T d2d4f749fcfd4fb3bfb8115af3471598 P c0f38c1908414566aa9115bb2cbdd85d: Time spent starting tablet: real 0.091s user 0.005s sys 0.001s
W20260809 15:45:49.543469 23351 leader_election.cc:341] T 99bdb93503db4de4a1cfb0161ea0c07c P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:49.547770 23591 tablet_bootstrap.cc:492] T 0c29937a80724ddb9d07a27534a146f3 P c0f38c1908414566aa9115bb2cbdd85d: Bootstrap starting.
W20260809 15:45:49.548360 23470 leader_election.cc:341] T 99bdb93503db4de4a1cfb0161ea0c07c P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:49.555164 23397 raft_consensus.cc:2454] T 73f6f44c259547f1b05a3a434616fa54 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 1.
I20260809 15:45:49.549551 23472 leader_election.cc:302] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:45:49.556394 23691 raft_consensus.cc:2726] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 15:45:49.561105 23722 raft_consensus.cc:482] T 0a51c0baaa6e432997c9d3552af4f254 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:45:49.561573 23722 raft_consensus.cc:504] T 0a51c0baaa6e432997c9d3552af4f254 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:49.551648 23648 raft_consensus.cc:504] T 99bdb93503db4de4a1cfb0161ea0c07c P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:49.564414 23400 raft_consensus.cc:2379] T 99bdb93503db4de4a1cfb0161ea0c07c P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c in current term 1: Already voted for candidate c0f38c1908414566aa9115bb2cbdd85d in this term.
I20260809 15:45:49.564802 23529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a51c0baaa6e432997c9d3552af4f254" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
W20260809 15:45:49.566100 23228 leader_election.cc:341] T 0a51c0baaa6e432997c9d3552af4f254 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:49.566627 23722 leader_election.cc:288] T 0a51c0baaa6e432997c9d3552af4f254 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:45:49.578297 23648 leader_election.cc:288] T 99bdb93503db4de4a1cfb0161ea0c07c P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:45:49.579324 23529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99bdb93503db4de4a1cfb0161ea0c07c" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:45:49.581562 23529 raft_consensus.cc:3037] T 99bdb93503db4de4a1cfb0161ea0c07c P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:49.582481 23470 leader_election.cc:302] T 73f6f44c259547f1b05a3a434616fa54 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:45:49.584993 23594 tablet_bootstrap.cc:492] T 6ae14a74a26f434db08c652e097385c8 P 67b3aeadfa7a432abe8af4d292e04ab9: No bootstrap required, opened a new log
I20260809 15:45:49.593729 23691 raft_consensus.cc:2781] T 73f6f44c259547f1b05a3a434616fa54 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:45:49.594198 23691 raft_consensus.cc:686] T 73f6f44c259547f1b05a3a434616fa54 P 65c36d164b0540f099d592f38307ee6c [term 1 LEADER]: Becoming Leader. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:45:49.594965 23691 consensus_queue.cc:237] T 73f6f44c259547f1b05a3a434616fa54 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:49.595835 23591 tablet_bootstrap.cc:654] T 0c29937a80724ddb9d07a27534a146f3 P c0f38c1908414566aa9115bb2cbdd85d: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:49.599191 23594 ts_tablet_manager.cc:1397] T 6ae14a74a26f434db08c652e097385c8 P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent bootstrapping tablet: real 0.274s user 0.016s sys 0.004s
I20260809 15:45:49.601898 23594 raft_consensus.cc:348] T 6ae14a74a26f434db08c652e097385c8 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:49.602522 23594 raft_consensus.cc:374] T 6ae14a74a26f434db08c652e097385c8 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:49.602766 23594 raft_consensus.cc:729] T 6ae14a74a26f434db08c652e097385c8 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Initialized, Role: FOLLOWER
I20260809 15:45:49.588505 23593 tablet_bootstrap.cc:492] T 0da4d5283c8a46878bd39b85e5c71f66 P 65c36d164b0540f099d592f38307ee6c: No bootstrap required, opened a new log
I20260809 15:45:49.605158 23593 ts_tablet_manager.cc:1397] T 0da4d5283c8a46878bd39b85e5c71f66 P 65c36d164b0540f099d592f38307ee6c: Time spent bootstrapping tablet: real 0.225s user 0.040s sys 0.015s
I20260809 15:45:49.607978 23593 raft_consensus.cc:348] T 0da4d5283c8a46878bd39b85e5c71f66 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:49.585156 23118 catalog_manager.cc:5495] T e71a05ba1d4d42bd95c3f85f9313399e P c0f38c1908414566aa9115bb2cbdd85d reported cstate change: term changed from 0 to 1, leader changed from <none> to c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66). New cstate: current_term: 1 leader_uuid: "c0f38c1908414566aa9115bb2cbdd85d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } health_report { overall_health: UNKNOWN } } }
I20260809 15:45:49.588554 23282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99bdb93503db4de4a1cfb0161ea0c07c" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:45:49.588503 23529 raft_consensus.cc:2454] T 99bdb93503db4de4a1cfb0161ea0c07c P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 1.
I20260809 15:45:49.590643 23790 raft_consensus.cc:482] T 0a51c0baaa6e432997c9d3552af4f254 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:45:49.610554 23594 consensus_queue.cc:260] T 6ae14a74a26f434db08c652e097385c8 P 67b3aeadfa7a432abe8af4d292e04ab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:45:49.611061 23351 leader_election.cc:341] T 99bdb93503db4de4a1cfb0161ea0c07c P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:49.610589 23790 raft_consensus.cc:504] T 0a51c0baaa6e432997c9d3552af4f254 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:49.591408 23397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a51c0baaa6e432997c9d3552af4f254" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
W20260809 15:45:49.592162 23472 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 47869.8us
I20260809 15:45:49.608584 23593 raft_consensus.cc:374] T 0da4d5283c8a46878bd39b85e5c71f66 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:49.613947 23594 ts_tablet_manager.cc:1428] T 6ae14a74a26f434db08c652e097385c8 P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent starting tablet: real 0.014s user 0.005s sys 0.000s
I20260809 15:45:49.616688 23594 tablet_bootstrap.cc:492] T 763e58d6ba644eea830a6c1664d39c9b P 67b3aeadfa7a432abe8af4d292e04ab9: Bootstrap starting.
I20260809 15:45:49.619441 23591 tablet_bootstrap.cc:492] T 0c29937a80724ddb9d07a27534a146f3 P c0f38c1908414566aa9115bb2cbdd85d: No bootstrap required, opened a new log
I20260809 15:45:49.620077 23591 ts_tablet_manager.cc:1397] T 0c29937a80724ddb9d07a27534a146f3 P c0f38c1908414566aa9115bb2cbdd85d: Time spent bootstrapping tablet: real 0.072s user 0.017s sys 0.006s
I20260809 15:45:49.623641 23591 raft_consensus.cc:348] T 0c29937a80724ddb9d07a27534a146f3 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:49.624351 23591 raft_consensus.cc:374] T 0c29937a80724ddb9d07a27534a146f3 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:49.624655 23591 raft_consensus.cc:729] T 0c29937a80724ddb9d07a27534a146f3 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Initialized, Role: FOLLOWER
I20260809 15:45:49.616109 23593 raft_consensus.cc:729] T 0da4d5283c8a46878bd39b85e5c71f66 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Initialized, Role: FOLLOWER
I20260809 15:45:49.625722 23593 consensus_queue.cc:260] T 0da4d5283c8a46878bd39b85e5c71f66 P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:49.627390 23591 consensus_queue.cc:260] T 0c29937a80724ddb9d07a27534a146f3 P c0f38c1908414566aa9115bb2cbdd85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:49.649641 23661 raft_consensus.cc:482] T 6ae14a74a26f434db08c652e097385c8 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:45:49.650097 23661 raft_consensus.cc:504] T 6ae14a74a26f434db08c652e097385c8 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:49.650466 23790 leader_election.cc:288] T 0a51c0baaa6e432997c9d3552af4f254 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:45:49.628077 23397 raft_consensus.cc:2454] T 0a51c0baaa6e432997c9d3552af4f254 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 0.
I20260809 15:45:49.651958 23661 leader_election.cc:288] T 6ae14a74a26f434db08c652e097385c8 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:45:49.654767 23591 ts_tablet_manager.cc:1428] T 0c29937a80724ddb9d07a27534a146f3 P c0f38c1908414566aa9115bb2cbdd85d: Time spent starting tablet: real 0.034s user 0.007s sys 0.001s
I20260809 15:45:49.696295 23282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a51c0baaa6e432997c9d3552af4f254" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:45:49.696539 23591 tablet_bootstrap.cc:492] T a88421dba80d4d3388aa5eb62e23853b P c0f38c1908414566aa9115bb2cbdd85d: Bootstrap starting.
I20260809 15:45:49.696904 23282 raft_consensus.cc:2454] T 0a51c0baaa6e432997c9d3552af4f254 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 0.
I20260809 15:45:49.664975 23790 raft_consensus.cc:482] T 0c29937a80724ddb9d07a27534a146f3 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:45:49.697861 23790 raft_consensus.cc:504] T 0c29937a80724ddb9d07a27534a146f3 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:49.698380 23228 leader_election.cc:302] T 0a51c0baaa6e432997c9d3552af4f254 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c
I20260809 15:45:49.699105 23722 raft_consensus.cc:2781] T 0a51c0baaa6e432997c9d3552af4f254 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 15:45:49.699472 23722 raft_consensus.cc:482] T 0a51c0baaa6e432997c9d3552af4f254 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:45:49.699795 23722 raft_consensus.cc:3037] T 0a51c0baaa6e432997c9d3552af4f254 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:49.700769 23351 leader_election.cc:302] T 0a51c0baaa6e432997c9d3552af4f254 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:45:49.701661 23648 raft_consensus.cc:2781] T 0a51c0baaa6e432997c9d3552af4f254 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 15:45:49.702006 23648 raft_consensus.cc:482] T 0a51c0baaa6e432997c9d3552af4f254 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:45:49.702315 23648 raft_consensus.cc:3037] T 0a51c0baaa6e432997c9d3552af4f254 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:49.702397 23591 tablet_bootstrap.cc:654] T a88421dba80d4d3388aa5eb62e23853b P c0f38c1908414566aa9115bb2cbdd85d: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:49.704030 23397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ae14a74a26f434db08c652e097385c8" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:45:49.704612 23397 raft_consensus.cc:2454] T 6ae14a74a26f434db08c652e097385c8 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 0.
I20260809 15:45:49.708511 23648 raft_consensus.cc:504] T 0a51c0baaa6e432997c9d3552af4f254 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:49.710717 23648 leader_election.cc:288] T 0a51c0baaa6e432997c9d3552af4f254 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:45:49.710616 23116 catalog_manager.cc:5495] T 73f6f44c259547f1b05a3a434616fa54 P 65c36d164b0540f099d592f38307ee6c reported cstate change: term changed from 0 to 1, leader changed from <none> to 65c36d164b0540f099d592f38307ee6c (127.20.202.67). New cstate: current_term: 1 leader_uuid: "65c36d164b0540f099d592f38307ee6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } health_report { overall_health: HEALTHY } } }
I20260809 15:45:49.696741 23594 tablet_bootstrap.cc:654] T 763e58d6ba644eea830a6c1664d39c9b P 67b3aeadfa7a432abe8af4d292e04ab9: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:49.714794 23593 ts_tablet_manager.cc:1428] T 0da4d5283c8a46878bd39b85e5c71f66 P 65c36d164b0540f099d592f38307ee6c: Time spent starting tablet: real 0.109s user 0.007s sys 0.032s
I20260809 15:45:49.711596 23790 leader_election.cc:288] T 0c29937a80724ddb9d07a27534a146f3 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
W20260809 15:45:49.717235 23472 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 68943.9us
I20260809 15:45:49.717690 23643 raft_consensus.cc:482] T ae6d2c08a7a94f58ac8dd698e3fd456c P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:45:49.719480 23472 leader_election.cc:302] T 99bdb93503db4de4a1cfb0161ea0c07c P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:45:49.719694 23643 raft_consensus.cc:504] T ae6d2c08a7a94f58ac8dd698e3fd456c P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:49.720219 23691 raft_consensus.cc:2726] T 99bdb93503db4de4a1cfb0161ea0c07c P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 15:45:49.721427 23643 leader_election.cc:288] T ae6d2c08a7a94f58ac8dd698e3fd456c P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:45:49.723101 23353 leader_election.cc:302] T 99bdb93503db4de4a1cfb0161ea0c07c P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters: 67b3aeadfa7a432abe8af4d292e04ab9
W20260809 15:45:49.723964 23353 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 112695us
I20260809 15:45:49.724978 23664 raft_consensus.cc:2781] T 99bdb93503db4de4a1cfb0161ea0c07c P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:45:49.725407 23664 raft_consensus.cc:686] T 99bdb93503db4de4a1cfb0161ea0c07c P c0f38c1908414566aa9115bb2cbdd85d [term 1 LEADER]: Becoming Leader. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:45:49.726730 23529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a51c0baaa6e432997c9d3552af4f254" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:45:49.726445 23664 consensus_queue.cc:237] T 99bdb93503db4de4a1cfb0161ea0c07c P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:49.727246 23722 raft_consensus.cc:504] T 0a51c0baaa6e432997c9d3552af4f254 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:49.728830 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a51c0baaa6e432997c9d3552af4f254" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:45:49.729645 23722 leader_election.cc:288] T 0a51c0baaa6e432997c9d3552af4f254 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:45:49.729887 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c29937a80724ddb9d07a27534a146f3" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:45:49.730423 23400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae6d2c08a7a94f58ac8dd698e3fd456c" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:45:49.731009 23522 raft_consensus.cc:3037] T ae6d2c08a7a94f58ac8dd698e3fd456c P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:49.732574 23282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c29937a80724ddb9d07a27534a146f3" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:45:49.733918 23282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a51c0baaa6e432997c9d3552af4f254" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:45:49.734586 23282 raft_consensus.cc:2379] T 0a51c0baaa6e432997c9d3552af4f254 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d in current term 1: Already voted for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in this term.
I20260809 15:45:49.730458 23526 raft_consensus.cc:2454] T 0c29937a80724ddb9d07a27534a146f3 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 0.
I20260809 15:45:49.736245 23528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ae14a74a26f434db08c652e097385c8" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
W20260809 15:45:49.736750 23351 leader_election.cc:341] T 0c29937a80724ddb9d07a27534a146f3 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:49.741068 23400 raft_consensus.cc:3037] T 5712f6b6fffc4cc5a8ddfbe63e4097df P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:49.742197 23648 raft_consensus.cc:482] T d2d4f749fcfd4fb3bfb8115af3471598 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:45:49.742597 23648 raft_consensus.cc:504] T d2d4f749fcfd4fb3bfb8115af3471598 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:49.749697 23282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae6d2c08a7a94f58ac8dd698e3fd456c" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:45:49.749764 23397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a51c0baaa6e432997c9d3552af4f254" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:45:49.750469 23282 raft_consensus.cc:2379] T ae6d2c08a7a94f58ac8dd698e3fd456c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c in current term 1: Already voted for candidate c0f38c1908414566aa9115bb2cbdd85d in this term.
I20260809 15:45:49.750517 23397 raft_consensus.cc:2379] T 0a51c0baaa6e432997c9d3552af4f254 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 in current term 1: Already voted for candidate c0f38c1908414566aa9115bb2cbdd85d in this term.
W20260809 15:45:49.747068 23353 leader_election.cc:341] T 0a51c0baaa6e432997c9d3552af4f254 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:49.751435 23470 leader_election.cc:302] T ae6d2c08a7a94f58ac8dd698e3fd456c P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:45:49.752125 23643 raft_consensus.cc:2726] T ae6d2c08a7a94f58ac8dd698e3fd456c P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 15:45:49.752238 23522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a51c0baaa6e432997c9d3552af4f254" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
W20260809 15:45:49.753026 23353 leader_election.cc:341] T 0a51c0baaa6e432997c9d3552af4f254 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:49.755251 23593 tablet_bootstrap.cc:492] T 6ae14a74a26f434db08c652e097385c8 P 65c36d164b0540f099d592f38307ee6c: Bootstrap starting.
I20260809 15:45:49.755446 23648 leader_election.cc:288] T d2d4f749fcfd4fb3bfb8115af3471598 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:45:49.759505 23722 raft_consensus.cc:482] T 0da4d5283c8a46878bd39b85e5c71f66 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:45:49.760016 23722 raft_consensus.cc:504] T 0da4d5283c8a46878bd39b85e5c71f66 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:49.760172 23117 catalog_manager.cc:5495] T 99bdb93503db4de4a1cfb0161ea0c07c P c0f38c1908414566aa9115bb2cbdd85d reported cstate change: term changed from 0 to 1, leader changed from <none> to c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66). New cstate: current_term: 1 leader_uuid: "c0f38c1908414566aa9115bb2cbdd85d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } health_report { overall_health: UNKNOWN } } }
I20260809 15:45:49.753425 23353 leader_election.cc:302] T 0a51c0baaa6e432997c9d3552af4f254 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:45:49.779835 23591 tablet_bootstrap.cc:492] T a88421dba80d4d3388aa5eb62e23853b P c0f38c1908414566aa9115bb2cbdd85d: No bootstrap required, opened a new log
I20260809 15:45:49.779843 23353 leader_election.cc:302] T 0c29937a80724ddb9d07a27534a146f3 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters: 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:45:49.780437 23591 ts_tablet_manager.cc:1397] T a88421dba80d4d3388aa5eb62e23853b P c0f38c1908414566aa9115bb2cbdd85d: Time spent bootstrapping tablet: real 0.084s user 0.015s sys 0.015s
I20260809 15:45:49.781312 23790 consensus_queue.cc:1035] T ae6d2c08a7a94f58ac8dd698e3fd456c P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:45:49.783324 23931 raft_consensus.cc:2781] T 0c29937a80724ddb9d07a27534a146f3 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 15:45:49.783752 23931 raft_consensus.cc:482] T 0c29937a80724ddb9d07a27534a146f3 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:45:49.784085 23931 raft_consensus.cc:3037] T 0c29937a80724ddb9d07a27534a146f3 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:49.784482 23522 raft_consensus.cc:3037] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:49.784420 23924 raft_consensus.cc:2726] T 0a51c0baaa6e432997c9d3552af4f254 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260809 15:45:49.766737 23228 leader_election.cc:302] T 6ae14a74a26f434db08c652e097385c8 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:45:49.785158 23722 leader_election.cc:288] T 0da4d5283c8a46878bd39b85e5c71f66 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:45:49.783001 23591 raft_consensus.cc:348] T a88421dba80d4d3388aa5eb62e23853b P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:49.786762 23591 raft_consensus.cc:374] T a88421dba80d4d3388aa5eb62e23853b P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:49.785566 23924 consensus_queue.cc:1035] T ae6d2c08a7a94f58ac8dd698e3fd456c P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260809 15:45:49.787775 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2d4f749fcfd4fb3bfb8115af3471598" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:45:49.787081 23591 raft_consensus.cc:729] T a88421dba80d4d3388aa5eb62e23853b P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Initialized, Role: FOLLOWER
I20260809 15:45:49.789004 23591 consensus_queue.cc:260] T a88421dba80d4d3388aa5eb62e23853b P c0f38c1908414566aa9115bb2cbdd85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:49.791079 23591 ts_tablet_manager.cc:1428] T a88421dba80d4d3388aa5eb62e23853b P c0f38c1908414566aa9115bb2cbdd85d: Time spent starting tablet: real 0.010s user 0.004s sys 0.000s
I20260809 15:45:49.794867 23591 tablet_bootstrap.cc:492] T 4bcac4b24e7a4448abdc934bba5a624e P c0f38c1908414566aa9115bb2cbdd85d: Bootstrap starting.
W20260809 15:45:49.795794 23353 leader_election.cc:341] T d2d4f749fcfd4fb3bfb8115af3471598 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553): Illegal state: must be running to vote when last-logged opid is not known
W20260809 15:45:49.803151 23228 leader_election.cc:341] T 6ae14a74a26f434db08c652e097385c8 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:49.810077 23528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0da4d5283c8a46878bd39b85e5c71f66" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:45:49.810871 23528 raft_consensus.cc:2454] T 0da4d5283c8a46878bd39b85e5c71f66 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 0.
I20260809 15:45:49.805881 23808 raft_consensus.cc:2781] T 6ae14a74a26f434db08c652e097385c8 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 15:45:49.811460 23808 raft_consensus.cc:482] T 6ae14a74a26f434db08c652e097385c8 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:45:49.811784 23808 raft_consensus.cc:3037] T 6ae14a74a26f434db08c652e097385c8 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:49.818403 23283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2d4f749fcfd4fb3bfb8115af3471598" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:45:49.819219 23283 raft_consensus.cc:2454] T d2d4f749fcfd4fb3bfb8115af3471598 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 0.
I20260809 15:45:49.820534 23400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0da4d5283c8a46878bd39b85e5c71f66" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
W20260809 15:45:49.823832 23228 leader_election.cc:341] T 0a51c0baaa6e432997c9d3552af4f254 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:49.824364 23228 leader_election.cc:302] T 0a51c0baaa6e432997c9d3552af4f254 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:45:49.825551 23802 raft_consensus.cc:2726] T 0a51c0baaa6e432997c9d3552af4f254 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260809 15:45:49.820564 23351 leader_election.cc:302] T d2d4f749fcfd4fb3bfb8115af3471598 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c
I20260809 15:45:49.830137 23924 raft_consensus.cc:2781] T d2d4f749fcfd4fb3bfb8115af3471598 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 15:45:49.831440 23924 raft_consensus.cc:482] T d2d4f749fcfd4fb3bfb8115af3471598 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:45:49.836174 23924 raft_consensus.cc:3037] T d2d4f749fcfd4fb3bfb8115af3471598 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:49.833302 23228 leader_election.cc:302] T 0da4d5283c8a46878bd39b85e5c71f66 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:45:49.840960 23931 raft_consensus.cc:504] T 0c29937a80724ddb9d07a27534a146f3 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:49.846453 23722 raft_consensus.cc:2781] T 0da4d5283c8a46878bd39b85e5c71f66 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 15:45:49.847813 23722 raft_consensus.cc:482] T 0da4d5283c8a46878bd39b85e5c71f66 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:45:49.848896 23722 raft_consensus.cc:3037] T 0da4d5283c8a46878bd39b85e5c71f66 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:49.854874 23594 tablet_bootstrap.cc:492] T 763e58d6ba644eea830a6c1664d39c9b P 67b3aeadfa7a432abe8af4d292e04ab9: No bootstrap required, opened a new log
I20260809 15:45:49.855295 23594 ts_tablet_manager.cc:1397] T 763e58d6ba644eea830a6c1664d39c9b P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent bootstrapping tablet: real 0.239s user 0.019s sys 0.027s
I20260809 15:45:49.857333 23593 tablet_bootstrap.cc:654] T 6ae14a74a26f434db08c652e097385c8 P 65c36d164b0540f099d592f38307ee6c: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:49.860517 23931 leader_election.cc:288] T 0c29937a80724ddb9d07a27534a146f3 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:45:49.860360 23285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c29937a80724ddb9d07a27534a146f3" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
W20260809 15:45:49.870577 23351 leader_election.cc:341] T 0c29937a80724ddb9d07a27534a146f3 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459): Illegal state: must be running to vote when last-logged opid is not known
W20260809 15:45:49.871343 23228 leader_election.cc:341] T 0da4d5283c8a46878bd39b85e5c71f66 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:49.872233 23594 raft_consensus.cc:348] T 763e58d6ba644eea830a6c1664d39c9b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:49.872704 23594 raft_consensus.cc:374] T 763e58d6ba644eea830a6c1664d39c9b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:49.872903 23594 raft_consensus.cc:729] T 763e58d6ba644eea830a6c1664d39c9b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Initialized, Role: FOLLOWER
I20260809 15:45:49.871863 23528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c29937a80724ddb9d07a27534a146f3" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:45:49.875917 23528 raft_consensus.cc:3037] T 0c29937a80724ddb9d07a27534a146f3 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:49.878599 23661 consensus_queue.cc:1035] T 3296ef50be254fc29cfc77d22a8f7a23 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:45:49.873390 23594 consensus_queue.cc:260] T 763e58d6ba644eea830a6c1664d39c9b P 67b3aeadfa7a432abe8af4d292e04ab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:49.882448 23528 raft_consensus.cc:2454] T 0c29937a80724ddb9d07a27534a146f3 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 1.
I20260809 15:45:49.882869 23808 raft_consensus.cc:504] T 6ae14a74a26f434db08c652e097385c8 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:49.883656 23353 leader_election.cc:302] T 0c29937a80724ddb9d07a27534a146f3 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters: 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:45:49.884603 23790 raft_consensus.cc:2781] T 0c29937a80724ddb9d07a27534a146f3 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:45:49.884778 23808 leader_election.cc:288] T 6ae14a74a26f434db08c652e097385c8 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:45:49.885080 23790 raft_consensus.cc:686] T 0c29937a80724ddb9d07a27534a146f3 P c0f38c1908414566aa9115bb2cbdd85d [term 1 LEADER]: Becoming Leader. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:45:49.885946 23790 consensus_queue.cc:237] T 0c29937a80724ddb9d07a27534a146f3 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:49.890954 23924 raft_consensus.cc:504] T d2d4f749fcfd4fb3bfb8115af3471598 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:49.894860 23594 ts_tablet_manager.cc:1428] T 763e58d6ba644eea830a6c1664d39c9b P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent starting tablet: real 0.039s user 0.005s sys 0.000s
I20260809 15:45:49.895649 23924 leader_election.cc:288] T d2d4f749fcfd4fb3bfb8115af3471598 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:45:49.896212 23285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2d4f749fcfd4fb3bfb8115af3471598" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:45:49.897171 23400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ae14a74a26f434db08c652e097385c8" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:45:49.900207 23285 raft_consensus.cc:3037] T d2d4f749fcfd4fb3bfb8115af3471598 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:49.900300 23400 raft_consensus.cc:3037] T 6ae14a74a26f434db08c652e097385c8 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:49.906230 23594 tablet_bootstrap.cc:492] T 0c29937a80724ddb9d07a27534a146f3 P 67b3aeadfa7a432abe8af4d292e04ab9: Bootstrap starting.
I20260809 15:45:49.897698 23528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ae14a74a26f434db08c652e097385c8" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:45:49.907490 23400 raft_consensus.cc:2454] T 6ae14a74a26f434db08c652e097385c8 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 1.
W20260809 15:45:49.908592 23228 leader_election.cc:341] T 6ae14a74a26f434db08c652e097385c8 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:49.899389 23591 tablet_bootstrap.cc:654] T 4bcac4b24e7a4448abdc934bba5a624e P c0f38c1908414566aa9115bb2cbdd85d: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:49.898161 23522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2d4f749fcfd4fb3bfb8115af3471598" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
W20260809 15:45:49.914608 23353 leader_election.cc:341] T d2d4f749fcfd4fb3bfb8115af3471598 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:49.917033 23722 raft_consensus.cc:504] T 0da4d5283c8a46878bd39b85e5c71f66 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:49.918218 23285 raft_consensus.cc:2454] T d2d4f749fcfd4fb3bfb8115af3471598 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 1.
I20260809 15:45:49.920123 23351 leader_election.cc:302] T d2d4f749fcfd4fb3bfb8115af3471598 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c
I20260809 15:45:49.923033 23931 raft_consensus.cc:2781] T d2d4f749fcfd4fb3bfb8115af3471598 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:45:49.923776 23931 raft_consensus.cc:686] T d2d4f749fcfd4fb3bfb8115af3471598 P c0f38c1908414566aa9115bb2cbdd85d [term 1 LEADER]: Becoming Leader. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:45:49.918027 23228 leader_election.cc:302] T 6ae14a74a26f434db08c652e097385c8 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c
I20260809 15:45:49.927773 23802 raft_consensus.cc:482] T 2424fd9bd7f14e508fd106cbfda786a4 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:45:49.929950 23931 consensus_queue.cc:237] T d2d4f749fcfd4fb3bfb8115af3471598 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:49.930287 23802 raft_consensus.cc:504] T 2424fd9bd7f14e508fd106cbfda786a4 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:49.932966 23802 leader_election.cc:288] T 2424fd9bd7f14e508fd106cbfda786a4 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:45:49.935689 23400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0da4d5283c8a46878bd39b85e5c71f66" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:45:49.927850 23648 consensus_queue.cc:1035] T e71a05ba1d4d42bd95c3f85f9313399e P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.008s
I20260809 15:45:49.938362 23661 raft_consensus.cc:2781] T 6ae14a74a26f434db08c652e097385c8 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:45:49.937995 23118 catalog_manager.cc:5495] T 0c29937a80724ddb9d07a27534a146f3 P c0f38c1908414566aa9115bb2cbdd85d reported cstate change: term changed from 0 to 1, leader changed from <none> to c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66). New cstate: current_term: 1 leader_uuid: "c0f38c1908414566aa9115bb2cbdd85d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } health_report { overall_health: UNKNOWN } } }
I20260809 15:45:49.938817 23661 raft_consensus.cc:686] T 6ae14a74a26f434db08c652e097385c8 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 LEADER]: Becoming Leader. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Running, Role: LEADER
I20260809 15:45:49.940837 23400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2424fd9bd7f14e508fd106cbfda786a4" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:45:49.940920 23661 consensus_queue.cc:237] T 6ae14a74a26f434db08c652e097385c8 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:49.941533 23400 raft_consensus.cc:2454] T 2424fd9bd7f14e508fd106cbfda786a4 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 1.
I20260809 15:45:49.948989 23594 tablet_bootstrap.cc:654] T 0c29937a80724ddb9d07a27534a146f3 P 67b3aeadfa7a432abe8af4d292e04ab9: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:49.949697 23522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2424fd9bd7f14e508fd106cbfda786a4" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:45:49.950515 23522 raft_consensus.cc:2454] T 2424fd9bd7f14e508fd106cbfda786a4 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 1.
I20260809 15:45:49.944105 23802 consensus_queue.cc:1035] T 3296ef50be254fc29cfc77d22a8f7a23 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20260809 15:45:49.944672 23722 leader_election.cc:288] T 0da4d5283c8a46878bd39b85e5c71f66 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:45:49.955588 23593 tablet_bootstrap.cc:492] T 6ae14a74a26f434db08c652e097385c8 P 65c36d164b0540f099d592f38307ee6c: No bootstrap required, opened a new log
I20260809 15:45:49.956022 23593 ts_tablet_manager.cc:1397] T 6ae14a74a26f434db08c652e097385c8 P 65c36d164b0540f099d592f38307ee6c: Time spent bootstrapping tablet: real 0.201s user 0.009s sys 0.005s
I20260809 15:45:49.957475 23591 tablet_bootstrap.cc:492] T 4bcac4b24e7a4448abdc934bba5a624e P c0f38c1908414566aa9115bb2cbdd85d: No bootstrap required, opened a new log
I20260809 15:45:49.957893 23591 ts_tablet_manager.cc:1397] T 4bcac4b24e7a4448abdc934bba5a624e P c0f38c1908414566aa9115bb2cbdd85d: Time spent bootstrapping tablet: real 0.163s user 0.007s sys 0.015s
W20260809 15:45:49.964171 23228 leader_election.cc:341] T 0da4d5283c8a46878bd39b85e5c71f66 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:49.973678 23228 leader_election.cc:302] T 2424fd9bd7f14e508fd106cbfda786a4 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:45:49.975792 23661 raft_consensus.cc:2781] T 2424fd9bd7f14e508fd106cbfda786a4 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260809 15:45:49.976279 23661 raft_consensus.cc:482] T 2424fd9bd7f14e508fd106cbfda786a4 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:45:49.976791 23661 raft_consensus.cc:3037] T 2424fd9bd7f14e508fd106cbfda786a4 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:45:49.983671 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0da4d5283c8a46878bd39b85e5c71f66" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:45:49.984225 23527 raft_consensus.cc:3037] T 0da4d5283c8a46878bd39b85e5c71f66 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:49.985458 23593 raft_consensus.cc:348] T 6ae14a74a26f434db08c652e097385c8 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:49.989132 23593 raft_consensus.cc:374] T 6ae14a74a26f434db08c652e097385c8 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:49.989416 23593 raft_consensus.cc:729] T 6ae14a74a26f434db08c652e097385c8 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Initialized, Role: FOLLOWER
I20260809 15:45:49.990058 23593 consensus_queue.cc:260] T 6ae14a74a26f434db08c652e097385c8 P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:49.999233 23661 raft_consensus.cc:504] T 2424fd9bd7f14e508fd106cbfda786a4 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:50.001511 23661 leader_election.cc:288] T 2424fd9bd7f14e508fd106cbfda786a4 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 2 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:45:50.001199 23118 catalog_manager.cc:5495] T d2d4f749fcfd4fb3bfb8115af3471598 P c0f38c1908414566aa9115bb2cbdd85d reported cstate change: term changed from 0 to 1, leader changed from <none> to c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66). New cstate: current_term: 1 leader_uuid: "c0f38c1908414566aa9115bb2cbdd85d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } health_report { overall_health: UNKNOWN } } }
I20260809 15:45:50.002554 23400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2424fd9bd7f14e508fd106cbfda786a4" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:45:50.002972 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2424fd9bd7f14e508fd106cbfda786a4" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:45:50.003078 23400 raft_consensus.cc:3037] T 2424fd9bd7f14e508fd106cbfda786a4 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:45:50.003595 23526 raft_consensus.cc:3037] T 2424fd9bd7f14e508fd106cbfda786a4 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:45:50.005949 23527 raft_consensus.cc:2454] T 0da4d5283c8a46878bd39b85e5c71f66 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 1.
I20260809 15:45:50.007138 23228 leader_election.cc:302] T 0da4d5283c8a46878bd39b85e5c71f66 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters: c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:45:50.008311 23593 ts_tablet_manager.cc:1428] T 6ae14a74a26f434db08c652e097385c8 P 65c36d164b0540f099d592f38307ee6c: Time spent starting tablet: real 0.052s user 0.002s sys 0.007s
I20260809 15:45:50.008625 23722 raft_consensus.cc:2781] T 0da4d5283c8a46878bd39b85e5c71f66 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:45:50.009049 23722 raft_consensus.cc:686] T 0da4d5283c8a46878bd39b85e5c71f66 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 LEADER]: Becoming Leader. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Running, Role: LEADER
I20260809 15:45:50.009245 23593 tablet_bootstrap.cc:492] T 7baa2492df0340c4a5ac241959ce1ac3 P 65c36d164b0540f099d592f38307ee6c: Bootstrap starting.
I20260809 15:45:50.009805 23722 consensus_queue.cc:237] T 0da4d5283c8a46878bd39b85e5c71f66 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:50.008351 23117 catalog_manager.cc:5495] T 6ae14a74a26f434db08c652e097385c8 P 67b3aeadfa7a432abe8af4d292e04ab9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65). New cstate: current_term: 1 leader_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } health_report { overall_health: UNKNOWN } } }
I20260809 15:45:49.990016 23591 raft_consensus.cc:348] T 4bcac4b24e7a4448abdc934bba5a624e P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:50.025840 23591 raft_consensus.cc:374] T 4bcac4b24e7a4448abdc934bba5a624e P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:50.026015 23591 raft_consensus.cc:729] T 4bcac4b24e7a4448abdc934bba5a624e P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Initialized, Role: FOLLOWER
I20260809 15:45:50.029639 23526 raft_consensus.cc:2454] T 2424fd9bd7f14e508fd106cbfda786a4 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 2.
I20260809 15:45:50.032147 23228 leader_election.cc:302] T 2424fd9bd7f14e508fd106cbfda786a4 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:45:50.033245 23802 raft_consensus.cc:2781] T 2424fd9bd7f14e508fd106cbfda786a4 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 15:45:50.033840 23802 raft_consensus.cc:686] T 2424fd9bd7f14e508fd106cbfda786a4 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 LEADER]: Becoming Leader. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Running, Role: LEADER
I20260809 15:45:50.040139 23802 consensus_queue.cc:237] T 2424fd9bd7f14e508fd106cbfda786a4 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:50.043798 23924 raft_consensus.cc:482] T a88421dba80d4d3388aa5eb62e23853b P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:45:50.044456 23924 raft_consensus.cc:504] T a88421dba80d4d3388aa5eb62e23853b P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:50.034976 23591 consensus_queue.cc:260] T 4bcac4b24e7a4448abdc934bba5a624e P c0f38c1908414566aa9115bb2cbdd85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:50.049285 23285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a88421dba80d4d3388aa5eb62e23853b" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:45:50.049883 23924 leader_election.cc:288] T a88421dba80d4d3388aa5eb62e23853b P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
W20260809 15:45:50.050734 23351 leader_election.cc:341] T a88421dba80d4d3388aa5eb62e23853b P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:50.053782 23591 ts_tablet_manager.cc:1428] T 4bcac4b24e7a4448abdc934bba5a624e P c0f38c1908414566aa9115bb2cbdd85d: Time spent starting tablet: real 0.096s user 0.012s sys 0.008s
I20260809 15:45:50.054080 23655 consensus_queue.cc:1035] T 73f6f44c259547f1b05a3a434616fa54 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:45:50.055856 23591 tablet_bootstrap.cc:492] T d1a134ba33ea4076bbaac0ae460c17ba P c0f38c1908414566aa9115bb2cbdd85d: Bootstrap starting.
I20260809 15:45:50.059599 23593 tablet_bootstrap.cc:654] T 7baa2492df0340c4a5ac241959ce1ac3 P 65c36d164b0540f099d592f38307ee6c: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:50.056089 23400 raft_consensus.cc:2454] T 2424fd9bd7f14e508fd106cbfda786a4 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 2.
I20260809 15:45:50.071573 23931 consensus_queue.cc:1035] T e71a05ba1d4d42bd95c3f85f9313399e P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:45:50.073706 23522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a88421dba80d4d3388aa5eb62e23853b" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
W20260809 15:45:50.075917 23353 leader_election.cc:341] T a88421dba80d4d3388aa5eb62e23853b P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:50.076385 23353 leader_election.cc:302] T a88421dba80d4d3388aa5eb62e23853b P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:45:50.077742 23931 raft_consensus.cc:2726] T a88421dba80d4d3388aa5eb62e23853b P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 15:45:50.096549 23643 raft_consensus.cc:482] T 6ae14a74a26f434db08c652e097385c8 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:45:50.097234 23643 raft_consensus.cc:504] T 6ae14a74a26f434db08c652e097385c8 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:50.098217 23594 tablet_bootstrap.cc:492] T 0c29937a80724ddb9d07a27534a146f3 P 67b3aeadfa7a432abe8af4d292e04ab9: No bootstrap required, opened a new log
I20260809 15:45:50.098626 23594 ts_tablet_manager.cc:1397] T 0c29937a80724ddb9d07a27534a146f3 P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent bootstrapping tablet: real 0.193s user 0.008s sys 0.049s
I20260809 15:45:50.101161 23282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ae14a74a26f434db08c652e097385c8" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:45:50.104389 23594 raft_consensus.cc:348] T 0c29937a80724ddb9d07a27534a146f3 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:50.105357 23594 raft_consensus.cc:374] T 0c29937a80724ddb9d07a27534a146f3 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:50.105674 23594 raft_consensus.cc:729] T 0c29937a80724ddb9d07a27534a146f3 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Initialized, Role: FOLLOWER
I20260809 15:45:50.106318 23594 consensus_queue.cc:260] T 0c29937a80724ddb9d07a27534a146f3 P 67b3aeadfa7a432abe8af4d292e04ab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:50.108773 23400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ae14a74a26f434db08c652e097385c8" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:45:50.109802 23400 raft_consensus.cc:2379] T 6ae14a74a26f434db08c652e097385c8 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c in current term 1: Already voted for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in this term.
I20260809 15:45:50.103348 23643 leader_election.cc:288] T 6ae14a74a26f434db08c652e097385c8 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:45:50.110386 23933 consensus_queue.cc:1035] T 73f6f44c259547f1b05a3a434616fa54 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260809 15:45:50.113384 23472 leader_election.cc:302] T 6ae14a74a26f434db08c652e097385c8 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:45:50.114741 23933 raft_consensus.cc:3037] T 6ae14a74a26f434db08c652e097385c8 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:50.116269 23594 ts_tablet_manager.cc:1428] T 0c29937a80724ddb9d07a27534a146f3 P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent starting tablet: real 0.017s user 0.008s sys 0.000s
I20260809 15:45:50.118479 23594 tablet_bootstrap.cc:492] T 297c82ed906d4b48b184985eed3de601 P 67b3aeadfa7a432abe8af4d292e04ab9: Bootstrap starting.
I20260809 15:45:50.134457 23933 raft_consensus.cc:2726] T 6ae14a74a26f434db08c652e097385c8 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 15:45:50.136704 23593 tablet_bootstrap.cc:492] T 7baa2492df0340c4a5ac241959ce1ac3 P 65c36d164b0540f099d592f38307ee6c: No bootstrap required, opened a new log
I20260809 15:45:50.141228 23593 ts_tablet_manager.cc:1397] T 7baa2492df0340c4a5ac241959ce1ac3 P 65c36d164b0540f099d592f38307ee6c: Time spent bootstrapping tablet: real 0.132s user 0.023s sys 0.013s
I20260809 15:45:50.140803 23591 tablet_bootstrap.cc:654] T d1a134ba33ea4076bbaac0ae460c17ba P c0f38c1908414566aa9115bb2cbdd85d: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:50.145844 23593 raft_consensus.cc:348] T 7baa2492df0340c4a5ac241959ce1ac3 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:50.146605 23593 raft_consensus.cc:374] T 7baa2492df0340c4a5ac241959ce1ac3 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:50.146895 23593 raft_consensus.cc:729] T 7baa2492df0340c4a5ac241959ce1ac3 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Initialized, Role: FOLLOWER
I20260809 15:45:50.147622 23593 consensus_queue.cc:260] T 7baa2492df0340c4a5ac241959ce1ac3 P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:50.150885 23593 ts_tablet_manager.cc:1428] T 7baa2492df0340c4a5ac241959ce1ac3 P 65c36d164b0540f099d592f38307ee6c: Time spent starting tablet: real 0.009s user 0.004s sys 0.000s
I20260809 15:45:50.151017 23594 tablet_bootstrap.cc:654] T 297c82ed906d4b48b184985eed3de601 P 67b3aeadfa7a432abe8af4d292e04ab9: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:50.151674 23593 tablet_bootstrap.cc:492] T d2d4f749fcfd4fb3bfb8115af3471598 P 65c36d164b0540f099d592f38307ee6c: Bootstrap starting.
I20260809 15:45:50.156519 23118 catalog_manager.cc:5495] T 0da4d5283c8a46878bd39b85e5c71f66 P 67b3aeadfa7a432abe8af4d292e04ab9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65). New cstate: current_term: 1 leader_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } health_report { overall_health: UNKNOWN } } }
I20260809 15:45:50.159891 23593 tablet_bootstrap.cc:654] T d2d4f749fcfd4fb3bfb8115af3471598 P 65c36d164b0540f099d592f38307ee6c: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:50.167423 23591 tablet_bootstrap.cc:492] T d1a134ba33ea4076bbaac0ae460c17ba P c0f38c1908414566aa9115bb2cbdd85d: No bootstrap required, opened a new log
I20260809 15:45:50.167814 23591 ts_tablet_manager.cc:1397] T d1a134ba33ea4076bbaac0ae460c17ba P c0f38c1908414566aa9115bb2cbdd85d: Time spent bootstrapping tablet: real 0.112s user 0.003s sys 0.008s
I20260809 15:45:50.167793 23933 raft_consensus.cc:482] T 7baa2492df0340c4a5ac241959ce1ac3 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:45:50.168321 23933 raft_consensus.cc:504] T 7baa2492df0340c4a5ac241959ce1ac3 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:50.169385 23593 tablet_bootstrap.cc:492] T d2d4f749fcfd4fb3bfb8115af3471598 P 65c36d164b0540f099d592f38307ee6c: No bootstrap required, opened a new log
I20260809 15:45:50.169840 23593 ts_tablet_manager.cc:1397] T d2d4f749fcfd4fb3bfb8115af3471598 P 65c36d164b0540f099d592f38307ee6c: Time spent bootstrapping tablet: real 0.018s user 0.015s sys 0.000s
I20260809 15:45:50.158046 23118 catalog_manager.cc:5495] T 2424fd9bd7f14e508fd106cbfda786a4 P 67b3aeadfa7a432abe8af4d292e04ab9 reported cstate change: term changed from 0 to 2, leader changed from <none> to 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65). New cstate: current_term: 2 leader_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } health_report { overall_health: UNKNOWN } } }
I20260809 15:45:50.170351 23933 leader_election.cc:288] T 7baa2492df0340c4a5ac241959ce1ac3 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:45:50.171726 23285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7baa2492df0340c4a5ac241959ce1ac3" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:45:50.172112 23397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7baa2492df0340c4a5ac241959ce1ac3" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:45:50.173343 23472 leader_election.cc:302] T 7baa2492df0340c4a5ac241959ce1ac3 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:45:50.174050 23933 raft_consensus.cc:2726] T 7baa2492df0340c4a5ac241959ce1ac3 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 15:45:50.174556 23593 raft_consensus.cc:348] T d2d4f749fcfd4fb3bfb8115af3471598 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:50.175201 23593 raft_consensus.cc:374] T d2d4f749fcfd4fb3bfb8115af3471598 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:50.175015 23591 raft_consensus.cc:348] T d1a134ba33ea4076bbaac0ae460c17ba P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:50.175485 23593 raft_consensus.cc:729] T d2d4f749fcfd4fb3bfb8115af3471598 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Initialized, Role: FOLLOWER
I20260809 15:45:50.175702 23591 raft_consensus.cc:374] T d1a134ba33ea4076bbaac0ae460c17ba P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:50.175964 23591 raft_consensus.cc:729] T d1a134ba33ea4076bbaac0ae460c17ba P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Initialized, Role: FOLLOWER
I20260809 15:45:50.176561 23591 consensus_queue.cc:260] T d1a134ba33ea4076bbaac0ae460c17ba P c0f38c1908414566aa9115bb2cbdd85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:50.178570 23591 ts_tablet_manager.cc:1428] T d1a134ba33ea4076bbaac0ae460c17ba P c0f38c1908414566aa9115bb2cbdd85d: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20260809 15:45:50.179497 23594 tablet_bootstrap.cc:492] T 297c82ed906d4b48b184985eed3de601 P 67b3aeadfa7a432abe8af4d292e04ab9: No bootstrap required, opened a new log
I20260809 15:45:50.179917 23591 tablet_bootstrap.cc:492] T 0da4d5283c8a46878bd39b85e5c71f66 P c0f38c1908414566aa9115bb2cbdd85d: Bootstrap starting.
I20260809 15:45:50.179924 23594 ts_tablet_manager.cc:1397] T 297c82ed906d4b48b184985eed3de601 P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent bootstrapping tablet: real 0.062s user 0.011s sys 0.000s
I20260809 15:45:50.182157 23594 raft_consensus.cc:348] T 297c82ed906d4b48b184985eed3de601 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:50.182772 23594 raft_consensus.cc:374] T 297c82ed906d4b48b184985eed3de601 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:50.183029 23594 raft_consensus.cc:729] T 297c82ed906d4b48b184985eed3de601 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Initialized, Role: FOLLOWER
I20260809 15:45:50.183568 23594 consensus_queue.cc:260] T 297c82ed906d4b48b184985eed3de601 P 67b3aeadfa7a432abe8af4d292e04ab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:50.185397 23591 tablet_bootstrap.cc:654] T 0da4d5283c8a46878bd39b85e5c71f66 P c0f38c1908414566aa9115bb2cbdd85d: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:50.178274 23593 consensus_queue.cc:260] T d2d4f749fcfd4fb3bfb8115af3471598 P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:50.187628 23593 ts_tablet_manager.cc:1428] T d2d4f749fcfd4fb3bfb8115af3471598 P 65c36d164b0540f099d592f38307ee6c: Time spent starting tablet: real 0.018s user 0.007s sys 0.000s
I20260809 15:45:50.188303 23594 ts_tablet_manager.cc:1428] T 297c82ed906d4b48b184985eed3de601 P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20260809 15:45:50.189698 23594 tablet_bootstrap.cc:492] T 4bcac4b24e7a4448abdc934bba5a624e P 67b3aeadfa7a432abe8af4d292e04ab9: Bootstrap starting.
I20260809 15:45:50.194644 23594 tablet_bootstrap.cc:654] T 4bcac4b24e7a4448abdc934bba5a624e P 67b3aeadfa7a432abe8af4d292e04ab9: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:50.195801 23591 tablet_bootstrap.cc:492] T 0da4d5283c8a46878bd39b85e5c71f66 P c0f38c1908414566aa9115bb2cbdd85d: No bootstrap required, opened a new log
I20260809 15:45:50.196218 23591 ts_tablet_manager.cc:1397] T 0da4d5283c8a46878bd39b85e5c71f66 P c0f38c1908414566aa9115bb2cbdd85d: Time spent bootstrapping tablet: real 0.016s user 0.010s sys 0.005s
I20260809 15:45:50.197255 23593 tablet_bootstrap.cc:492] T a88421dba80d4d3388aa5eb62e23853b P 65c36d164b0540f099d592f38307ee6c: Bootstrap starting.
I20260809 15:45:50.198591 23591 raft_consensus.cc:348] T 0da4d5283c8a46878bd39b85e5c71f66 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:50.199205 23591 raft_consensus.cc:374] T 0da4d5283c8a46878bd39b85e5c71f66 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:50.199489 23591 raft_consensus.cc:729] T 0da4d5283c8a46878bd39b85e5c71f66 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Initialized, Role: FOLLOWER
I20260809 15:45:50.200047 23591 consensus_queue.cc:260] T 0da4d5283c8a46878bd39b85e5c71f66 P c0f38c1908414566aa9115bb2cbdd85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:50.201967 23591 ts_tablet_manager.cc:1428] T 0da4d5283c8a46878bd39b85e5c71f66 P c0f38c1908414566aa9115bb2cbdd85d: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260809 15:45:50.202404 23593 tablet_bootstrap.cc:654] T a88421dba80d4d3388aa5eb62e23853b P 65c36d164b0540f099d592f38307ee6c: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:50.202781 23591 tablet_bootstrap.cc:492] T 763e58d6ba644eea830a6c1664d39c9b P c0f38c1908414566aa9115bb2cbdd85d: Bootstrap starting.
W20260809 15:45:50.203583 23351 consensus_peers.cc:487] T 99bdb93503db4de4a1cfb0161ea0c07c P c0f38c1908414566aa9115bb2cbdd85d -> Peer 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459): Couldn't send request to peer 67b3aeadfa7a432abe8af4d292e04ab9. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260809 15:45:50.210142 23591 tablet_bootstrap.cc:654] T 763e58d6ba644eea830a6c1664d39c9b P c0f38c1908414566aa9115bb2cbdd85d: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:50.210728 23594 tablet_bootstrap.cc:492] T 4bcac4b24e7a4448abdc934bba5a624e P 67b3aeadfa7a432abe8af4d292e04ab9: No bootstrap required, opened a new log
I20260809 15:45:50.211301 23594 ts_tablet_manager.cc:1397] T 4bcac4b24e7a4448abdc934bba5a624e P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent bootstrapping tablet: real 0.022s user 0.009s sys 0.000s
I20260809 15:45:50.213753 23594 raft_consensus.cc:348] T 4bcac4b24e7a4448abdc934bba5a624e P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:50.214383 23594 raft_consensus.cc:374] T 4bcac4b24e7a4448abdc934bba5a624e P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:50.214622 23594 raft_consensus.cc:729] T 4bcac4b24e7a4448abdc934bba5a624e P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Initialized, Role: FOLLOWER
I20260809 15:45:50.215170 23594 consensus_queue.cc:260] T 4bcac4b24e7a4448abdc934bba5a624e P 67b3aeadfa7a432abe8af4d292e04ab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:50.217389 23594 ts_tablet_manager.cc:1428] T 4bcac4b24e7a4448abdc934bba5a624e P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260809 15:45:50.218209 23594 tablet_bootstrap.cc:492] T d0721a4403a440f383318a3cea9d1442 P 67b3aeadfa7a432abe8af4d292e04ab9: Bootstrap starting.
I20260809 15:45:50.226852 23594 tablet_bootstrap.cc:654] T d0721a4403a440f383318a3cea9d1442 P 67b3aeadfa7a432abe8af4d292e04ab9: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:50.227365 23591 tablet_bootstrap.cc:492] T 763e58d6ba644eea830a6c1664d39c9b P c0f38c1908414566aa9115bb2cbdd85d: No bootstrap required, opened a new log
I20260809 15:45:50.227811 23591 ts_tablet_manager.cc:1397] T 763e58d6ba644eea830a6c1664d39c9b P c0f38c1908414566aa9115bb2cbdd85d: Time spent bootstrapping tablet: real 0.025s user 0.011s sys 0.005s
I20260809 15:45:50.228574 23593 tablet_bootstrap.cc:492] T a88421dba80d4d3388aa5eb62e23853b P 65c36d164b0540f099d592f38307ee6c: No bootstrap required, opened a new log
I20260809 15:45:50.228953 23593 ts_tablet_manager.cc:1397] T a88421dba80d4d3388aa5eb62e23853b P 65c36d164b0540f099d592f38307ee6c: Time spent bootstrapping tablet: real 0.032s user 0.014s sys 0.000s
I20260809 15:45:50.230154 23591 raft_consensus.cc:348] T 763e58d6ba644eea830a6c1664d39c9b P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:50.231235 23591 raft_consensus.cc:374] T 763e58d6ba644eea830a6c1664d39c9b P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:50.231688 23591 raft_consensus.cc:729] T 763e58d6ba644eea830a6c1664d39c9b P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Initialized, Role: FOLLOWER
I20260809 15:45:50.231282 23593 raft_consensus.cc:348] T a88421dba80d4d3388aa5eb62e23853b P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:50.232141 23593 raft_consensus.cc:374] T a88421dba80d4d3388aa5eb62e23853b P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:50.232404 23593 raft_consensus.cc:729] T a88421dba80d4d3388aa5eb62e23853b P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Initialized, Role: FOLLOWER
I20260809 15:45:50.232506 23591 consensus_queue.cc:260] T 763e58d6ba644eea830a6c1664d39c9b P c0f38c1908414566aa9115bb2cbdd85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:50.233057 23593 consensus_queue.cc:260] T a88421dba80d4d3388aa5eb62e23853b P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:50.234611 23591 ts_tablet_manager.cc:1428] T 763e58d6ba644eea830a6c1664d39c9b P c0f38c1908414566aa9115bb2cbdd85d: Time spent starting tablet: real 0.006s user 0.001s sys 0.003s
I20260809 15:45:50.235432 23591 tablet_bootstrap.cc:492] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P c0f38c1908414566aa9115bb2cbdd85d: Bootstrap starting.
I20260809 15:45:50.238701 23593 ts_tablet_manager.cc:1428] T a88421dba80d4d3388aa5eb62e23853b P 65c36d164b0540f099d592f38307ee6c: Time spent starting tablet: real 0.009s user 0.005s sys 0.001s
I20260809 15:45:50.239576 23593 tablet_bootstrap.cc:492] T c5c2e425a74b4e64ae70b5def750434c P 65c36d164b0540f099d592f38307ee6c: Bootstrap starting.
I20260809 15:45:50.240504 23591 tablet_bootstrap.cc:654] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P c0f38c1908414566aa9115bb2cbdd85d: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:50.245512 23591 tablet_bootstrap.cc:492] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P c0f38c1908414566aa9115bb2cbdd85d: No bootstrap required, opened a new log
I20260809 15:45:50.246011 23591 ts_tablet_manager.cc:1397] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P c0f38c1908414566aa9115bb2cbdd85d: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.003s
I20260809 15:45:50.246454 23594 tablet_bootstrap.cc:492] T d0721a4403a440f383318a3cea9d1442 P 67b3aeadfa7a432abe8af4d292e04ab9: No bootstrap required, opened a new log
I20260809 15:45:50.246834 23594 ts_tablet_manager.cc:1397] T d0721a4403a440f383318a3cea9d1442 P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent bootstrapping tablet: real 0.029s user 0.009s sys 0.005s
I20260809 15:45:50.248204 23591 raft_consensus.cc:348] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:50.248669 23591 raft_consensus.cc:374] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:50.248857 23591 raft_consensus.cc:729] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Initialized, Role: FOLLOWER
I20260809 15:45:50.248986 23594 raft_consensus.cc:348] T d0721a4403a440f383318a3cea9d1442 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:50.249610 23594 raft_consensus.cc:374] T d0721a4403a440f383318a3cea9d1442 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:50.249336 23591 consensus_queue.cc:260] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P c0f38c1908414566aa9115bb2cbdd85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:50.245828 23593 tablet_bootstrap.cc:654] T c5c2e425a74b4e64ae70b5def750434c P 65c36d164b0540f099d592f38307ee6c: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:50.249852 23594 raft_consensus.cc:729] T d0721a4403a440f383318a3cea9d1442 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Initialized, Role: FOLLOWER
I20260809 15:45:50.250470 23594 consensus_queue.cc:260] T d0721a4403a440f383318a3cea9d1442 P 67b3aeadfa7a432abe8af4d292e04ab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:50.252717 23933 raft_consensus.cc:482] T a88421dba80d4d3388aa5eb62e23853b P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:45:50.253135 23933 raft_consensus.cc:504] T a88421dba80d4d3388aa5eb62e23853b P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:50.254910 23933 leader_election.cc:288] T a88421dba80d4d3388aa5eb62e23853b P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:45:50.255088 23591 ts_tablet_manager.cc:1428] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P c0f38c1908414566aa9115bb2cbdd85d: Time spent starting tablet: real 0.009s user 0.004s sys 0.000s
I20260809 15:45:50.255918 23591 tablet_bootstrap.cc:492] T c5c2e425a74b4e64ae70b5def750434c P c0f38c1908414566aa9115bb2cbdd85d: Bootstrap starting.
I20260809 15:45:50.257047 23285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a88421dba80d4d3388aa5eb62e23853b" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:45:50.258116 23397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a88421dba80d4d3388aa5eb62e23853b" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:45:50.258733 23397 raft_consensus.cc:2454] T a88421dba80d4d3388aa5eb62e23853b P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 0.
W20260809 15:45:50.259836 23470 leader_election.cc:341] T a88421dba80d4d3388aa5eb62e23853b P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:50.260460 23594 ts_tablet_manager.cc:1428] T d0721a4403a440f383318a3cea9d1442 P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent starting tablet: real 0.013s user 0.007s sys 0.000s
I20260809 15:45:50.260665 23591 tablet_bootstrap.cc:654] T c5c2e425a74b4e64ae70b5def750434c P c0f38c1908414566aa9115bb2cbdd85d: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:50.262106 23472 leader_election.cc:302] T a88421dba80d4d3388aa5eb62e23853b P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters: 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:45:50.263895 23933 raft_consensus.cc:2781] T a88421dba80d4d3388aa5eb62e23853b P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 15:45:50.264317 23933 raft_consensus.cc:482] T a88421dba80d4d3388aa5eb62e23853b P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:45:50.264776 23933 raft_consensus.cc:3037] T a88421dba80d4d3388aa5eb62e23853b P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:50.268743 23593 tablet_bootstrap.cc:492] T c5c2e425a74b4e64ae70b5def750434c P 65c36d164b0540f099d592f38307ee6c: No bootstrap required, opened a new log
I20260809 15:45:50.268834 23594 tablet_bootstrap.cc:492] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 67b3aeadfa7a432abe8af4d292e04ab9: Bootstrap starting.
I20260809 15:45:50.269152 23593 ts_tablet_manager.cc:1397] T c5c2e425a74b4e64ae70b5def750434c P 65c36d164b0540f099d592f38307ee6c: Time spent bootstrapping tablet: real 0.030s user 0.006s sys 0.008s
I20260809 15:45:50.271323 23593 raft_consensus.cc:348] T c5c2e425a74b4e64ae70b5def750434c P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:50.271953 23593 raft_consensus.cc:374] T c5c2e425a74b4e64ae70b5def750434c P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:50.272200 23593 raft_consensus.cc:729] T c5c2e425a74b4e64ae70b5def750434c P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Initialized, Role: FOLLOWER
I20260809 15:45:50.272747 23593 consensus_queue.cc:260] T c5c2e425a74b4e64ae70b5def750434c P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:50.274809 23594 tablet_bootstrap.cc:654] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 67b3aeadfa7a432abe8af4d292e04ab9: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:50.276201 23924 raft_consensus.cc:482] T 4bcac4b24e7a4448abdc934bba5a624e P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:45:50.276386 23933 raft_consensus.cc:504] T a88421dba80d4d3388aa5eb62e23853b P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:50.276646 23924 raft_consensus.cc:504] T 4bcac4b24e7a4448abdc934bba5a624e P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:50.278378 23924 leader_election.cc:288] T 4bcac4b24e7a4448abdc934bba5a624e P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:45:50.279001 23285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a88421dba80d4d3388aa5eb62e23853b" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
W20260809 15:45:50.280606 23470 leader_election.cc:341] T a88421dba80d4d3388aa5eb62e23853b P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:50.280682 23522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bcac4b24e7a4448abdc934bba5a624e" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:45:50.278837 23924 consensus_queue.cc:1035] T 99bdb93503db4de4a1cfb0161ea0c07c P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260809 15:45:50.281934 23353 leader_election.cc:341] T 4bcac4b24e7a4448abdc934bba5a624e P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:50.282771 23591 tablet_bootstrap.cc:492] T c5c2e425a74b4e64ae70b5def750434c P c0f38c1908414566aa9115bb2cbdd85d: No bootstrap required, opened a new log
I20260809 15:45:50.283172 23591 ts_tablet_manager.cc:1397] T c5c2e425a74b4e64ae70b5def750434c P c0f38c1908414566aa9115bb2cbdd85d: Time spent bootstrapping tablet: real 0.027s user 0.011s sys 0.004s
I20260809 15:45:50.281373 23285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bcac4b24e7a4448abdc934bba5a624e" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:45:50.283813 23285 raft_consensus.cc:2454] T 4bcac4b24e7a4448abdc934bba5a624e P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 0.
I20260809 15:45:50.285811 23351 leader_election.cc:302] T 4bcac4b24e7a4448abdc934bba5a624e P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c
I20260809 15:45:50.286928 23924 raft_consensus.cc:2781] T 4bcac4b24e7a4448abdc934bba5a624e P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 15:45:50.287317 23924 raft_consensus.cc:482] T 4bcac4b24e7a4448abdc934bba5a624e P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:45:50.286648 23591 raft_consensus.cc:348] T c5c2e425a74b4e64ae70b5def750434c P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:50.288312 23591 raft_consensus.cc:374] T c5c2e425a74b4e64ae70b5def750434c P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:50.288617 23591 raft_consensus.cc:729] T c5c2e425a74b4e64ae70b5def750434c P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Initialized, Role: FOLLOWER
I20260809 15:45:50.288968 23397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a88421dba80d4d3388aa5eb62e23853b" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:45:50.287719 23924 raft_consensus.cc:3037] T 4bcac4b24e7a4448abdc934bba5a624e P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:50.289513 23397 raft_consensus.cc:3037] T a88421dba80d4d3388aa5eb62e23853b P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:50.289398 23591 consensus_queue.cc:260] T c5c2e425a74b4e64ae70b5def750434c P c0f38c1908414566aa9115bb2cbdd85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:50.294821 23591 ts_tablet_manager.cc:1428] T c5c2e425a74b4e64ae70b5def750434c P c0f38c1908414566aa9115bb2cbdd85d: Time spent starting tablet: real 0.011s user 0.005s sys 0.001s
I20260809 15:45:50.297649 23591 tablet_bootstrap.cc:492] T 0acc47ce21c3422698d1ef7277c3ed63 P c0f38c1908414566aa9115bb2cbdd85d: Bootstrap starting.
I20260809 15:45:50.289499 23593 ts_tablet_manager.cc:1428] T c5c2e425a74b4e64ae70b5def750434c P 65c36d164b0540f099d592f38307ee6c: Time spent starting tablet: real 0.020s user 0.005s sys 0.001s
I20260809 15:45:50.300658 23593 tablet_bootstrap.cc:492] T 0a51c0baaa6e432997c9d3552af4f254 P 65c36d164b0540f099d592f38307ee6c: Bootstrap starting.
I20260809 15:45:50.305819 23593 tablet_bootstrap.cc:654] T 0a51c0baaa6e432997c9d3552af4f254 P 65c36d164b0540f099d592f38307ee6c: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:50.307638 23526 raft_consensus.cc:3037] T 7baa2492df0340c4a5ac241959ce1ac3 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:50.308629 23591 tablet_bootstrap.cc:654] T 0acc47ce21c3422698d1ef7277c3ed63 P c0f38c1908414566aa9115bb2cbdd85d: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:50.309646 23963 raft_consensus.cc:482] T 763e58d6ba644eea830a6c1664d39c9b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:45:50.310084 23963 raft_consensus.cc:504] T 763e58d6ba644eea830a6c1664d39c9b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:50.312059 23963 leader_election.cc:288] T 763e58d6ba644eea830a6c1664d39c9b P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:45:50.313944 23399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "763e58d6ba644eea830a6c1664d39c9b" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:45:50.314594 23399 raft_consensus.cc:2454] T 763e58d6ba644eea830a6c1664d39c9b P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 0.
I20260809 15:45:50.317364 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "763e58d6ba644eea830a6c1664d39c9b" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
W20260809 15:45:50.318701 23228 leader_election.cc:341] T 763e58d6ba644eea830a6c1664d39c9b P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:50.287967 23933 leader_election.cc:288] T a88421dba80d4d3388aa5eb62e23853b P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 1 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:45:50.295838 23397 raft_consensus.cc:2454] T a88421dba80d4d3388aa5eb62e23853b P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 1.
I20260809 15:45:50.295316 23924 raft_consensus.cc:504] T 4bcac4b24e7a4448abdc934bba5a624e P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:50.325475 23924 leader_election.cc:288] T 4bcac4b24e7a4448abdc934bba5a624e P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:45:50.326433 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bcac4b24e7a4448abdc934bba5a624e" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:45:50.320640 23228 leader_election.cc:302] T 763e58d6ba644eea830a6c1664d39c9b P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c
I20260809 15:45:50.327653 23285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bcac4b24e7a4448abdc934bba5a624e" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:45:50.327711 23472 leader_election.cc:302] T a88421dba80d4d3388aa5eb62e23853b P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters: 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:45:50.328382 23285 raft_consensus.cc:3037] T 4bcac4b24e7a4448abdc934bba5a624e P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:50.328940 23933 raft_consensus.cc:2781] T a88421dba80d4d3388aa5eb62e23853b P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:45:50.329407 23933 raft_consensus.cc:686] T a88421dba80d4d3388aa5eb62e23853b P 65c36d164b0540f099d592f38307ee6c [term 1 LEADER]: Becoming Leader. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:45:50.331439 23963 raft_consensus.cc:2781] T 763e58d6ba644eea830a6c1664d39c9b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 15:45:50.330720 23933 consensus_queue.cc:237] T a88421dba80d4d3388aa5eb62e23853b P 65c36d164b0540f099d592f38307ee6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:50.331800 23963 raft_consensus.cc:482] T 763e58d6ba644eea830a6c1664d39c9b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:45:50.332187 23963 raft_consensus.cc:3037] T 763e58d6ba644eea830a6c1664d39c9b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:50.336146 23594 tablet_bootstrap.cc:492] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 67b3aeadfa7a432abe8af4d292e04ab9: No bootstrap required, opened a new log
I20260809 15:45:50.336604 23594 ts_tablet_manager.cc:1397] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent bootstrapping tablet: real 0.068s user 0.010s sys 0.021s
W20260809 15:45:50.337209 23353 leader_election.cc:341] T 4bcac4b24e7a4448abdc934bba5a624e P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:50.341396 23285 raft_consensus.cc:2454] T 4bcac4b24e7a4448abdc934bba5a624e P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 1.
I20260809 15:45:50.342803 23351 leader_election.cc:302] T 4bcac4b24e7a4448abdc934bba5a624e P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c
I20260809 15:45:50.344203 23924 raft_consensus.cc:2781] T 4bcac4b24e7a4448abdc934bba5a624e P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:45:50.344944 23924 raft_consensus.cc:686] T 4bcac4b24e7a4448abdc934bba5a624e P c0f38c1908414566aa9115bb2cbdd85d [term 1 LEADER]: Becoming Leader. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:45:50.349673 23664 raft_consensus.cc:482] T d1a134ba33ea4076bbaac0ae460c17ba P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:45:50.350723 23664 raft_consensus.cc:504] T d1a134ba33ea4076bbaac0ae460c17ba P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:50.349834 23594 raft_consensus.cc:348] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:50.351650 23594 raft_consensus.cc:374] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:50.351931 23594 raft_consensus.cc:729] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Initialized, Role: FOLLOWER
I20260809 15:45:50.352734 23594 consensus_queue.cc:260] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 67b3aeadfa7a432abe8af4d292e04ab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:50.355384 23664 leader_election.cc:288] T d1a134ba33ea4076bbaac0ae460c17ba P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:45:50.346438 23924 consensus_queue.cc:237] T 4bcac4b24e7a4448abdc934bba5a624e P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:50.357061 23664 consensus_queue.cc:1035] T 0c29937a80724ddb9d07a27534a146f3 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260809 15:45:50.360162 23522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1a134ba33ea4076bbaac0ae460c17ba" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:45:50.373137 23594 ts_tablet_manager.cc:1428] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent starting tablet: real 0.036s user 0.008s sys 0.000s
I20260809 15:45:50.374029 23594 tablet_bootstrap.cc:492] T 7660dfa2ef694927b8eda4100a2bcd9b P 67b3aeadfa7a432abe8af4d292e04ab9: Bootstrap starting.
W20260809 15:45:50.381285 23353 leader_election.cc:341] T d1a134ba33ea4076bbaac0ae460c17ba P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:50.385079 23285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1a134ba33ea4076bbaac0ae460c17ba" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:45:50.385705 23285 raft_consensus.cc:2454] T d1a134ba33ea4076bbaac0ae460c17ba P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 0.
I20260809 15:45:50.384987 23116 catalog_manager.cc:5495] T 4bcac4b24e7a4448abdc934bba5a624e P c0f38c1908414566aa9115bb2cbdd85d reported cstate change: term changed from 0 to 1, leader changed from <none> to c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66). New cstate: current_term: 1 leader_uuid: "c0f38c1908414566aa9115bb2cbdd85d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } health_report { overall_health: UNKNOWN } } }
I20260809 15:45:50.383854 23722 consensus_queue.cc:1035] T 6ae14a74a26f434db08c652e097385c8 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20260809 15:45:50.387795 23963 raft_consensus.cc:504] T 763e58d6ba644eea830a6c1664d39c9b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:50.389958 23963 leader_election.cc:288] T 763e58d6ba644eea830a6c1664d39c9b P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:45:50.367712 23808 raft_consensus.cc:482] T 297c82ed906d4b48b184985eed3de601 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:45:50.391570 23808 raft_consensus.cc:504] T 297c82ed906d4b48b184985eed3de601 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:50.393492 23808 leader_election.cc:288] T 297c82ed906d4b48b184985eed3de601 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:45:50.394936 23351 leader_election.cc:302] T d1a134ba33ea4076bbaac0ae460c17ba P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c
I20260809 15:45:50.397325 23924 raft_consensus.cc:2781] T d1a134ba33ea4076bbaac0ae460c17ba P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 15:45:50.397660 23924 raft_consensus.cc:482] T d1a134ba33ea4076bbaac0ae460c17ba P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:45:50.398000 23924 raft_consensus.cc:3037] T d1a134ba33ea4076bbaac0ae460c17ba P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:50.400862 23528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "297c82ed906d4b48b184985eed3de601" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:45:50.404206 23522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "763e58d6ba644eea830a6c1664d39c9b" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:45:50.408437 23118 catalog_manager.cc:5495] T a88421dba80d4d3388aa5eb62e23853b P 65c36d164b0540f099d592f38307ee6c reported cstate change: term changed from 0 to 1, leader changed from <none> to 65c36d164b0540f099d592f38307ee6c (127.20.202.67). New cstate: current_term: 1 leader_uuid: "65c36d164b0540f099d592f38307ee6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } health_report { overall_health: HEALTHY } } }
I20260809 15:45:50.415824 23400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "297c82ed906d4b48b184985eed3de601" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:45:50.415683 23924 raft_consensus.cc:504] T d1a134ba33ea4076bbaac0ae460c17ba P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:50.416061 23664 consensus_queue.cc:1035] T d2d4f749fcfd4fb3bfb8115af3471598 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:45:50.417519 23924 leader_election.cc:288] T d1a134ba33ea4076bbaac0ae460c17ba P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:45:50.418533 23522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1a134ba33ea4076bbaac0ae460c17ba" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:45:50.419234 23283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1a134ba33ea4076bbaac0ae460c17ba" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:45:50.419734 23283 raft_consensus.cc:3037] T d1a134ba33ea4076bbaac0ae460c17ba P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:50.419901 23397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "763e58d6ba644eea830a6c1664d39c9b" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:45:50.420498 23397 raft_consensus.cc:3037] T 763e58d6ba644eea830a6c1664d39c9b P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:50.429580 23808 raft_consensus.cc:482] T 0c29937a80724ddb9d07a27534a146f3 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:45:50.432252 23808 raft_consensus.cc:504] T 0c29937a80724ddb9d07a27534a146f3 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:50.434028 23808 leader_election.cc:288] T 0c29937a80724ddb9d07a27534a146f3 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
W20260809 15:45:50.429858 23353 leader_election.cc:341] T d1a134ba33ea4076bbaac0ae460c17ba P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553): Illegal state: must be running to vote when last-logged opid is not known
W20260809 15:45:50.428261 23228 leader_election.cc:341] T 297c82ed906d4b48b184985eed3de601 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:50.435731 23283 raft_consensus.cc:2454] T d1a134ba33ea4076bbaac0ae460c17ba P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 1.
I20260809 15:45:50.437863 23351 leader_election.cc:302] T d1a134ba33ea4076bbaac0ae460c17ba P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c
I20260809 15:45:50.439899 23593 tablet_bootstrap.cc:492] T 0a51c0baaa6e432997c9d3552af4f254 P 65c36d164b0540f099d592f38307ee6c: No bootstrap required, opened a new log
I20260809 15:45:50.440248 23593 ts_tablet_manager.cc:1397] T 0a51c0baaa6e432997c9d3552af4f254 P 65c36d164b0540f099d592f38307ee6c: Time spent bootstrapping tablet: real 0.140s user 0.025s sys 0.029s
I20260809 15:45:50.439519 23664 raft_consensus.cc:2781] T d1a134ba33ea4076bbaac0ae460c17ba P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:45:50.440937 23664 raft_consensus.cc:686] T d1a134ba33ea4076bbaac0ae460c17ba P c0f38c1908414566aa9115bb2cbdd85d [term 1 LEADER]: Becoming Leader. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:45:50.441177 23591 tablet_bootstrap.cc:492] T 0acc47ce21c3422698d1ef7277c3ed63 P c0f38c1908414566aa9115bb2cbdd85d: No bootstrap required, opened a new log
I20260809 15:45:50.441561 23591 ts_tablet_manager.cc:1397] T 0acc47ce21c3422698d1ef7277c3ed63 P c0f38c1908414566aa9115bb2cbdd85d: Time spent bootstrapping tablet: real 0.144s user 0.029s sys 0.049s
I20260809 15:45:50.443213 23285 raft_consensus.cc:3037] T 0c29937a80724ddb9d07a27534a146f3 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:50.445281 23400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c29937a80724ddb9d07a27534a146f3" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:45:50.445036 23664 consensus_queue.cc:237] T d1a134ba33ea4076bbaac0ae460c17ba P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:50.446125 23591 raft_consensus.cc:348] T 0acc47ce21c3422698d1ef7277c3ed63 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:50.446739 23591 raft_consensus.cc:374] T 0acc47ce21c3422698d1ef7277c3ed63 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:50.446980 23591 raft_consensus.cc:729] T 0acc47ce21c3422698d1ef7277c3ed63 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Initialized, Role: FOLLOWER
I20260809 15:45:50.447700 23655 raft_consensus.cc:482] T c5c2e425a74b4e64ae70b5def750434c P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:45:50.447422 23593 raft_consensus.cc:348] T 0a51c0baaa6e432997c9d3552af4f254 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:50.447548 23591 consensus_queue.cc:260] T 0acc47ce21c3422698d1ef7277c3ed63 P c0f38c1908414566aa9115bb2cbdd85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:50.448100 23593 raft_consensus.cc:374] T 0a51c0baaa6e432997c9d3552af4f254 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:50.448379 23593 raft_consensus.cc:729] T 0a51c0baaa6e432997c9d3552af4f254 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Initialized, Role: FOLLOWER
I20260809 15:45:50.449023 23591 ts_tablet_manager.cc:1428] T 0acc47ce21c3422698d1ef7277c3ed63 P c0f38c1908414566aa9115bb2cbdd85d: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20260809 15:45:50.450121 23591 tablet_bootstrap.cc:492] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P c0f38c1908414566aa9115bb2cbdd85d: Bootstrap starting.
I20260809 15:45:50.449543 23593 consensus_queue.cc:260] T 0a51c0baaa6e432997c9d3552af4f254 P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:50.448115 23655 raft_consensus.cc:504] T c5c2e425a74b4e64ae70b5def750434c P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:50.459570 23529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c29937a80724ddb9d07a27534a146f3" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
W20260809 15:45:50.463302 23228 leader_election.cc:341] T 297c82ed906d4b48b184985eed3de601 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:50.463760 23228 leader_election.cc:302] T 297c82ed906d4b48b184985eed3de601 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:45:50.465278 23963 raft_consensus.cc:2726] T 297c82ed906d4b48b184985eed3de601 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260809 15:45:50.465339 23228 leader_election.cc:341] T 763e58d6ba644eea830a6c1664d39c9b P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:50.468214 23593 ts_tablet_manager.cc:1428] T 0a51c0baaa6e432997c9d3552af4f254 P 65c36d164b0540f099d592f38307ee6c: Time spent starting tablet: real 0.028s user 0.006s sys 0.000s
I20260809 15:45:50.469085 23593 tablet_bootstrap.cc:492] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 65c36d164b0540f099d592f38307ee6c: Bootstrap starting.
I20260809 15:45:50.470532 23655 leader_election.cc:288] T c5c2e425a74b4e64ae70b5def750434c P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:45:50.468163 23228 leader_election.cc:302] T 0c29937a80724ddb9d07a27534a146f3 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:45:50.471609 23399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5c2e425a74b4e64ae70b5def750434c" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:45:50.472223 23399 raft_consensus.cc:2454] T c5c2e425a74b4e64ae70b5def750434c P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 0.
I20260809 15:45:50.473085 23594 tablet_bootstrap.cc:654] T 7660dfa2ef694927b8eda4100a2bcd9b P 67b3aeadfa7a432abe8af4d292e04ab9: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:50.472882 23722 raft_consensus.cc:2726] T 0c29937a80724ddb9d07a27534a146f3 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 15:45:50.477981 23280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5c2e425a74b4e64ae70b5def750434c" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:45:50.477896 23472 leader_election.cc:302] T c5c2e425a74b4e64ae70b5def750434c P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:45:50.479643 23933 raft_consensus.cc:2781] T c5c2e425a74b4e64ae70b5def750434c P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 15:45:50.479988 23933 raft_consensus.cc:482] T c5c2e425a74b4e64ae70b5def750434c P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:45:50.480304 23933 raft_consensus.cc:3037] T c5c2e425a74b4e64ae70b5def750434c P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Advancing to term 1
W20260809 15:45:50.482160 23470 leader_election.cc:341] T c5c2e425a74b4e64ae70b5def750434c P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:50.491389 23526 raft_consensus.cc:3037] T d2d4f749fcfd4fb3bfb8115af3471598 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:50.472999 23397 raft_consensus.cc:2454] T 763e58d6ba644eea830a6c1664d39c9b P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 1.
I20260809 15:45:50.499964 23593 tablet_bootstrap.cc:654] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 65c36d164b0540f099d592f38307ee6c: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:50.495851 23924 consensus_queue.cc:1035] T 0c29937a80724ddb9d07a27534a146f3 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.024s
I20260809 15:45:50.520684 23933 raft_consensus.cc:504] T c5c2e425a74b4e64ae70b5def750434c P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:50.522604 23933 leader_election.cc:288] T c5c2e425a74b4e64ae70b5def750434c P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 1 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:45:50.523918 23285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5c2e425a74b4e64ae70b5def750434c" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:45:50.526978 23228 leader_election.cc:302] T 763e58d6ba644eea830a6c1664d39c9b P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c
I20260809 15:45:50.528321 23722 raft_consensus.cc:2781] T 763e58d6ba644eea830a6c1664d39c9b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:45:50.528940 23722 raft_consensus.cc:686] T 763e58d6ba644eea830a6c1664d39c9b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 LEADER]: Becoming Leader. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Running, Role: LEADER
I20260809 15:45:50.530217 23722 consensus_queue.cc:237] T 763e58d6ba644eea830a6c1664d39c9b P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:45:50.532223 23470 leader_election.cc:341] T c5c2e425a74b4e64ae70b5def750434c P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:50.533311 23116 catalog_manager.cc:5495] T d1a134ba33ea4076bbaac0ae460c17ba P c0f38c1908414566aa9115bb2cbdd85d reported cstate change: term changed from 0 to 1, leader changed from <none> to c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66). New cstate: current_term: 1 leader_uuid: "c0f38c1908414566aa9115bb2cbdd85d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } health_report { overall_health: UNKNOWN } } }
I20260809 15:45:50.552115 23528 raft_consensus.cc:1261] T 2424fd9bd7f14e508fd106cbfda786a4 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Refusing update from remote peer 67b3aeadfa7a432abe8af4d292e04ab9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260809 15:45:50.567456 23963 consensus_queue.cc:1035] T 2424fd9bd7f14e508fd106cbfda786a4 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.015s
I20260809 15:45:50.589242 23924 consensus_queue.cc:1035] T d2d4f749fcfd4fb3bfb8115af3471598 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:45:50.595216 23397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5c2e425a74b4e64ae70b5def750434c" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:45:50.596068 23397 raft_consensus.cc:3037] T c5c2e425a74b4e64ae70b5def750434c P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:50.615744 23594 tablet_bootstrap.cc:492] T 7660dfa2ef694927b8eda4100a2bcd9b P 67b3aeadfa7a432abe8af4d292e04ab9: No bootstrap required, opened a new log
I20260809 15:45:50.616216 23594 ts_tablet_manager.cc:1397] T 7660dfa2ef694927b8eda4100a2bcd9b P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent bootstrapping tablet: real 0.242s user 0.015s sys 0.009s
I20260809 15:45:50.601446 23963 consensus_queue.cc:1035] T 6ae14a74a26f434db08c652e097385c8 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.016s
I20260809 15:45:50.619649 23924 raft_consensus.cc:482] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:45:50.620116 23924 raft_consensus.cc:504] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:50.622179 23924 leader_election.cc:288] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:45:50.622732 23118 catalog_manager.cc:5495] T 763e58d6ba644eea830a6c1664d39c9b P 67b3aeadfa7a432abe8af4d292e04ab9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65). New cstate: current_term: 1 leader_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } health_report { overall_health: UNKNOWN } } }
I20260809 15:45:50.623251 23963 consensus_queue.cc:1035] T 0da4d5283c8a46878bd39b85e5c71f66 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260809 15:45:50.625011 23522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3aa6a9a52804df0ba9b0ca3c82f38c3" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:45:50.626847 23404 raft_consensus.cc:3037] T 0da4d5283c8a46878bd39b85e5c71f66 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Advancing to term 1
W20260809 15:45:50.634968 23353 leader_election.cc:341] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:50.631603 23282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3aa6a9a52804df0ba9b0ca3c82f38c3" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:45:50.636096 23282 raft_consensus.cc:2454] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 0.
I20260809 15:45:50.638480 23591 tablet_bootstrap.cc:654] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P c0f38c1908414566aa9115bb2cbdd85d: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:50.638291 23351 leader_election.cc:302] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c
I20260809 15:45:50.639477 23664 raft_consensus.cc:2781] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 15:45:50.639986 23664 raft_consensus.cc:482] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:45:50.640457 23664 raft_consensus.cc:3037] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:50.635154 23399 raft_consensus.cc:1261] T 2424fd9bd7f14e508fd106cbfda786a4 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Refusing update from remote peer 67b3aeadfa7a432abe8af4d292e04ab9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260809 15:45:50.646163 23808 consensus_queue.cc:1035] T 2424fd9bd7f14e508fd106cbfda786a4 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:45:50.662165 23594 raft_consensus.cc:348] T 7660dfa2ef694927b8eda4100a2bcd9b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:50.662834 23594 raft_consensus.cc:374] T 7660dfa2ef694927b8eda4100a2bcd9b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:50.663081 23594 raft_consensus.cc:729] T 7660dfa2ef694927b8eda4100a2bcd9b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Initialized, Role: FOLLOWER
I20260809 15:45:50.669709 23691 raft_consensus.cc:482] T d2d4f749fcfd4fb3bfb8115af3471598 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:45:50.671480 23594 consensus_queue.cc:260] T 7660dfa2ef694927b8eda4100a2bcd9b P 67b3aeadfa7a432abe8af4d292e04ab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:50.675721 23722 raft_consensus.cc:482] T d0721a4403a440f383318a3cea9d1442 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:45:50.676326 23722 raft_consensus.cc:504] T d0721a4403a440f383318a3cea9d1442 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:50.678939 23722 leader_election.cc:288] T d0721a4403a440f383318a3cea9d1442 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:45:50.670457 23691 raft_consensus.cc:504] T d2d4f749fcfd4fb3bfb8115af3471598 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:50.690414 23400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0721a4403a440f383318a3cea9d1442" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:45:50.692075 23593 tablet_bootstrap.cc:492] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 65c36d164b0540f099d592f38307ee6c: No bootstrap required, opened a new log
I20260809 15:45:50.692534 23593 ts_tablet_manager.cc:1397] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 65c36d164b0540f099d592f38307ee6c: Time spent bootstrapping tablet: real 0.224s user 0.014s sys 0.012s
I20260809 15:45:50.674790 23924 raft_consensus.cc:482] T 0da4d5283c8a46878bd39b85e5c71f66 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:45:50.698081 24022 consensus_queue.cc:1035] T 0da4d5283c8a46878bd39b85e5c71f66 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260809 15:45:50.694517 23594 ts_tablet_manager.cc:1428] T 7660dfa2ef694927b8eda4100a2bcd9b P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent starting tablet: real 0.078s user 0.007s sys 0.001s
I20260809 15:45:50.701176 23594 tablet_bootstrap.cc:492] T 30c1740ef6874df0a332fbd6fd4b03ce P 67b3aeadfa7a432abe8af4d292e04ab9: Bootstrap starting.
I20260809 15:45:50.708737 23280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2d4f749fcfd4fb3bfb8115af3471598" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:45:50.726807 23397 raft_consensus.cc:2454] T c5c2e425a74b4e64ae70b5def750434c P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 1.
I20260809 15:45:50.742466 23664 raft_consensus.cc:504] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:50.703810 23924 raft_consensus.cc:504] T 0da4d5283c8a46878bd39b85e5c71f66 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:50.764183 23924 leader_election.cc:288] T 0da4d5283c8a46878bd39b85e5c71f66 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
W20260809 15:45:50.769110 23193 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
W20260809 15:45:50.789314 24031 log.cc:924] Time spent T 0da4d5283c8a46878bd39b85e5c71f66 P 65c36d164b0540f099d592f38307ee6c: Append to log took a long time: real 0.069s user 0.000s sys 0.000s
W20260809 15:45:50.790361 23228 leader_election.cc:341] T d0721a4403a440f383318a3cea9d1442 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:50.796733 23808 raft_consensus.cc:482] T 7660dfa2ef694927b8eda4100a2bcd9b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:45:50.797187 23808 raft_consensus.cc:504] T 7660dfa2ef694927b8eda4100a2bcd9b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:50.794301 23593 raft_consensus.cc:348] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:50.798520 23593 raft_consensus.cc:374] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:50.798799 23593 raft_consensus.cc:729] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Initialized, Role: FOLLOWER
W20260809 15:45:50.788074 23470 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 61663us
I20260809 15:45:50.799551 23593 consensus_queue.cc:260] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:50.802078 23593 ts_tablet_manager.cc:1428] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 65c36d164b0540f099d592f38307ee6c: Time spent starting tablet: real 0.109s user 0.004s sys 0.003s
I20260809 15:45:50.809624 23808 leader_election.cc:288] T 7660dfa2ef694927b8eda4100a2bcd9b P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:45:50.798074 23691 leader_election.cc:288] T d2d4f749fcfd4fb3bfb8115af3471598 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:45:50.811612 23282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0da4d5283c8a46878bd39b85e5c71f66" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:45:50.814713 23528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0721a4403a440f383318a3cea9d1442" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:45:50.815802 23591 tablet_bootstrap.cc:492] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P c0f38c1908414566aa9115bb2cbdd85d: No bootstrap required, opened a new log
I20260809 15:45:50.816247 23591 ts_tablet_manager.cc:1397] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P c0f38c1908414566aa9115bb2cbdd85d: Time spent bootstrapping tablet: real 0.367s user 0.025s sys 0.060s
I20260809 15:45:50.815639 23593 tablet_bootstrap.cc:492] T d1a134ba33ea4076bbaac0ae460c17ba P 65c36d164b0540f099d592f38307ee6c: Bootstrap starting.
I20260809 15:45:50.818341 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7660dfa2ef694927b8eda4100a2bcd9b" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:45:50.818928 23591 raft_consensus.cc:348] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:50.819628 23591 raft_consensus.cc:374] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:50.819885 23591 raft_consensus.cc:729] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Initialized, Role: FOLLOWER
I20260809 15:45:50.820995 23520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0da4d5283c8a46878bd39b85e5c71f66" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:45:50.826148 23520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7660dfa2ef694927b8eda4100a2bcd9b" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
W20260809 15:45:50.826763 23353 consensus_peers.cc:487] T 4bcac4b24e7a4448abdc934bba5a624e P c0f38c1908414566aa9115bb2cbdd85d -> Peer 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553): Couldn't send request to peer 65c36d164b0540f099d592f38307ee6c. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260809 15:45:50.827883 23528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3aa6a9a52804df0ba9b0ca3c82f38c3" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
W20260809 15:45:50.829538 23228 leader_election.cc:341] T d0721a4403a440f383318a3cea9d1442 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:50.829943 23228 leader_election.cc:302] T d0721a4403a440f383318a3cea9d1442 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
W20260809 15:45:50.830947 23228 leader_election.cc:341] T 7660dfa2ef694927b8eda4100a2bcd9b P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:50.831598 23591 consensus_queue.cc:260] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P c0f38c1908414566aa9115bb2cbdd85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:50.835585 24037 raft_consensus.cc:2726] T d0721a4403a440f383318a3cea9d1442 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 15:45:50.810454 23664 leader_election.cc:288] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:45:50.850414 23591 ts_tablet_manager.cc:1428] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P c0f38c1908414566aa9115bb2cbdd85d: Time spent starting tablet: real 0.034s user 0.006s sys 0.001s
I20260809 15:45:50.851243 23591 tablet_bootstrap.cc:492] T 30c1740ef6874df0a332fbd6fd4b03ce P c0f38c1908414566aa9115bb2cbdd85d: Bootstrap starting.
W20260809 15:45:50.852288 23470 consensus_peers.cc:487] T a88421dba80d4d3388aa5eb62e23853b P 65c36d164b0540f099d592f38307ee6c -> Peer 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459): Couldn't send request to peer 67b3aeadfa7a432abe8af4d292e04ab9. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20260809 15:45:50.854040 23228 leader_election.cc:341] T 7660dfa2ef694927b8eda4100a2bcd9b P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:50.854414 23228 leader_election.cc:302] T 7660dfa2ef694927b8eda4100a2bcd9b P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:45:50.854558 23282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3aa6a9a52804df0ba9b0ca3c82f38c3" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:45:50.859428 23282 raft_consensus.cc:3037] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:50.864593 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2d4f749fcfd4fb3bfb8115af3471598" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:45:50.865625 23594 tablet_bootstrap.cc:654] T 30c1740ef6874df0a332fbd6fd4b03ce P 67b3aeadfa7a432abe8af4d292e04ab9: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:50.855075 23722 raft_consensus.cc:2726] T 7660dfa2ef694927b8eda4100a2bcd9b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 15:45:50.866070 23353 leader_election.cc:302] T 0da4d5283c8a46878bd39b85e5c71f66 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:45:50.857668 23472 leader_election.cc:302] T c5c2e425a74b4e64ae70b5def750434c P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters: 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:45:50.872779 23691 raft_consensus.cc:2781] T c5c2e425a74b4e64ae70b5def750434c P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Leader election won for term 1
W20260809 15:45:50.874284 23353 leader_election.cc:341] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:50.873528 23691 raft_consensus.cc:686] T c5c2e425a74b4e64ae70b5def750434c P 65c36d164b0540f099d592f38307ee6c [term 1 LEADER]: Becoming Leader. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:45:50.875969 23691 consensus_queue.cc:237] T c5c2e425a74b4e64ae70b5def750434c P 65c36d164b0540f099d592f38307ee6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:50.878803 23664 raft_consensus.cc:2726] T 0da4d5283c8a46878bd39b85e5c71f66 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 15:45:50.887784 23664 raft_consensus.cc:482] T 0acc47ce21c3422698d1ef7277c3ed63 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:45:50.888064 23591 tablet_bootstrap.cc:654] T 30c1740ef6874df0a332fbd6fd4b03ce P c0f38c1908414566aa9115bb2cbdd85d: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:50.888362 23664 raft_consensus.cc:504] T 0acc47ce21c3422698d1ef7277c3ed63 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:50.894479 23664 leader_election.cc:288] T 0acc47ce21c3422698d1ef7277c3ed63 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:45:50.898451 23282 raft_consensus.cc:2454] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 1.
I20260809 15:45:50.898247 23472 leader_election.cc:302] T d2d4f749fcfd4fb3bfb8115af3471598 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d
W20260809 15:45:50.899255 23353 consensus_peers.cc:487] T d1a134ba33ea4076bbaac0ae460c17ba P c0f38c1908414566aa9115bb2cbdd85d -> Peer 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553): Couldn't send request to peer 65c36d164b0540f099d592f38307ee6c. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260809 15:45:50.900352 23933 raft_consensus.cc:2726] T d2d4f749fcfd4fb3bfb8115af3471598 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 15:45:50.903215 23286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0acc47ce21c3422698d1ef7277c3ed63" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:45:50.903715 23351 leader_election.cc:302] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c
I20260809 15:45:50.905496 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0acc47ce21c3422698d1ef7277c3ed63" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:45:50.905668 23664 raft_consensus.cc:2781] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Leader election won for term 1
W20260809 15:45:50.906004 23351 leader_election.cc:341] T 0acc47ce21c3422698d1ef7277c3ed63 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:50.906265 23664 raft_consensus.cc:686] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P c0f38c1908414566aa9115bb2cbdd85d [term 1 LEADER]: Becoming Leader. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:45:50.907788 23655 raft_consensus.cc:482] T 0a51c0baaa6e432997c9d3552af4f254 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:45:50.907292 23664 consensus_queue.cc:237] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:50.908363 23655 raft_consensus.cc:504] T 0a51c0baaa6e432997c9d3552af4f254 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:50.912251 23655 leader_election.cc:288] T 0a51c0baaa6e432997c9d3552af4f254 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
W20260809 15:45:50.911185 23353 leader_election.cc:341] T 0acc47ce21c3422698d1ef7277c3ed63 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:50.920451 23116 catalog_manager.cc:5495] T c5c2e425a74b4e64ae70b5def750434c P 65c36d164b0540f099d592f38307ee6c reported cstate change: term changed from 0 to 1, leader changed from <none> to 65c36d164b0540f099d592f38307ee6c (127.20.202.67). New cstate: current_term: 1 leader_uuid: "65c36d164b0540f099d592f38307ee6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } health_report { overall_health: HEALTHY } } }
W20260809 15:45:50.925251 23228 consensus_peers.cc:487] T 763e58d6ba644eea830a6c1664d39c9b P 67b3aeadfa7a432abe8af4d292e04ab9 -> Peer 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553): Couldn't send request to peer 65c36d164b0540f099d592f38307ee6c. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260809 15:45:50.917553 23353 leader_election.cc:302] T 0acc47ce21c3422698d1ef7277c3ed63 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:45:50.926647 23924 raft_consensus.cc:2726] T 0acc47ce21c3422698d1ef7277c3ed63 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 15:45:50.928493 23593 tablet_bootstrap.cc:654] T d1a134ba33ea4076bbaac0ae460c17ba P 65c36d164b0540f099d592f38307ee6c: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:50.929364 23286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a51c0baaa6e432997c9d3552af4f254" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:45:50.930586 23286 raft_consensus.cc:2379] T 0a51c0baaa6e432997c9d3552af4f254 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c in current term 1: Already voted for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in this term.
I20260809 15:45:50.933211 23924 raft_consensus.cc:482] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:45:50.933697 23924 raft_consensus.cc:504] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:50.937332 23924 leader_election.cc:288] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:45:50.942070 23591 tablet_bootstrap.cc:492] T 30c1740ef6874df0a332fbd6fd4b03ce P c0f38c1908414566aa9115bb2cbdd85d: No bootstrap required, opened a new log
I20260809 15:45:50.942548 23591 ts_tablet_manager.cc:1397] T 30c1740ef6874df0a332fbd6fd4b03ce P c0f38c1908414566aa9115bb2cbdd85d: Time spent bootstrapping tablet: real 0.091s user 0.006s sys 0.014s
I20260809 15:45:50.942991 23286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f9d6565aa4c45a3b3f01210d0eb71f7" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
W20260809 15:45:50.944957 23351 leader_election.cc:341] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:50.940632 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f9d6565aa4c45a3b3f01210d0eb71f7" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:45:50.946323 23526 raft_consensus.cc:2454] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 0.
I20260809 15:45:50.947649 23353 leader_election.cc:302] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters: 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:45:50.949302 23924 raft_consensus.cc:2781] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 15:45:50.949627 23924 raft_consensus.cc:482] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:45:50.950099 23924 raft_consensus.cc:3037] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:50.952292 23399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a51c0baaa6e432997c9d3552af4f254" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:45:50.952869 23594 tablet_bootstrap.cc:492] T 30c1740ef6874df0a332fbd6fd4b03ce P 67b3aeadfa7a432abe8af4d292e04ab9: No bootstrap required, opened a new log
I20260809 15:45:50.953294 23594 ts_tablet_manager.cc:1397] T 30c1740ef6874df0a332fbd6fd4b03ce P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent bootstrapping tablet: real 0.252s user 0.006s sys 0.014s
I20260809 15:45:50.953388 23399 raft_consensus.cc:2379] T 0a51c0baaa6e432997c9d3552af4f254 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c in current term 1: Already voted for candidate c0f38c1908414566aa9115bb2cbdd85d in this term.
I20260809 15:45:50.955586 23594 raft_consensus.cc:348] T 30c1740ef6874df0a332fbd6fd4b03ce P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:50.956941 23594 raft_consensus.cc:374] T 30c1740ef6874df0a332fbd6fd4b03ce P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:50.957257 23594 raft_consensus.cc:729] T 30c1740ef6874df0a332fbd6fd4b03ce P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Initialized, Role: FOLLOWER
I20260809 15:45:50.959932 23591 raft_consensus.cc:348] T 30c1740ef6874df0a332fbd6fd4b03ce P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:50.961018 23591 raft_consensus.cc:374] T 30c1740ef6874df0a332fbd6fd4b03ce P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:50.961216 23591 raft_consensus.cc:729] T 30c1740ef6874df0a332fbd6fd4b03ce P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Initialized, Role: FOLLOWER
I20260809 15:45:50.961633 23691 consensus_queue.cc:1035] T a88421dba80d4d3388aa5eb62e23853b P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260809 15:45:50.961658 23591 consensus_queue.cc:260] T 30c1740ef6874df0a332fbd6fd4b03ce P c0f38c1908414566aa9115bb2cbdd85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:50.964479 23591 ts_tablet_manager.cc:1428] T 30c1740ef6874df0a332fbd6fd4b03ce P c0f38c1908414566aa9115bb2cbdd85d: Time spent starting tablet: real 0.022s user 0.006s sys 0.001s
I20260809 15:45:50.965516 23591 tablet_bootstrap.cc:492] T 563daf42d7784db195ea99c3cb21b370 P c0f38c1908414566aa9115bb2cbdd85d: Bootstrap starting.
I20260809 15:45:50.971226 23472 leader_election.cc:302] T 0a51c0baaa6e432997c9d3552af4f254 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:45:50.972296 23691 raft_consensus.cc:3037] T 0a51c0baaa6e432997c9d3552af4f254 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:50.972972 23591 tablet_bootstrap.cc:654] T 563daf42d7784db195ea99c3cb21b370 P c0f38c1908414566aa9115bb2cbdd85d: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:50.974378 23924 raft_consensus.cc:504] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:50.961745 23594 consensus_queue.cc:260] T 30c1740ef6874df0a332fbd6fd4b03ce P 67b3aeadfa7a432abe8af4d292e04ab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:50.977979 23594 ts_tablet_manager.cc:1428] T 30c1740ef6874df0a332fbd6fd4b03ce P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent starting tablet: real 0.024s user 0.006s sys 0.000s
I20260809 15:45:50.978797 23594 tablet_bootstrap.cc:492] T c5c2e425a74b4e64ae70b5def750434c P 67b3aeadfa7a432abe8af4d292e04ab9: Bootstrap starting.
I20260809 15:45:50.978884 23691 raft_consensus.cc:2726] T 0a51c0baaa6e432997c9d3552af4f254 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 15:45:50.987051 23286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f9d6565aa4c45a3b3f01210d0eb71f7" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
W20260809 15:45:50.989521 23351 leader_election.cc:341] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:50.991504 23661 consensus_queue.cc:1035] T 763e58d6ba644eea830a6c1664d39c9b P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260809 15:45:50.995042 23594 tablet_bootstrap.cc:654] T c5c2e425a74b4e64ae70b5def750434c P 67b3aeadfa7a432abe8af4d292e04ab9: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:50.996425 23593 tablet_bootstrap.cc:492] T d1a134ba33ea4076bbaac0ae460c17ba P 65c36d164b0540f099d592f38307ee6c: No bootstrap required, opened a new log
I20260809 15:45:50.998356 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f9d6565aa4c45a3b3f01210d0eb71f7" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:45:51.005815 23593 ts_tablet_manager.cc:1397] T d1a134ba33ea4076bbaac0ae460c17ba P 65c36d164b0540f099d592f38307ee6c: Time spent bootstrapping tablet: real 0.190s user 0.015s sys 0.023s
I20260809 15:45:51.005798 23527 raft_consensus.cc:3037] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:50.999393 23924 leader_election.cc:288] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:45:51.000077 23664 consensus_queue.cc:1035] T 4bcac4b24e7a4448abdc934bba5a624e P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:45:51.010099 23593 raft_consensus.cc:348] T d1a134ba33ea4076bbaac0ae460c17ba P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:51.010748 23593 raft_consensus.cc:374] T d1a134ba33ea4076bbaac0ae460c17ba P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:51.010989 23593 raft_consensus.cc:729] T d1a134ba33ea4076bbaac0ae460c17ba P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Initialized, Role: FOLLOWER
I20260809 15:45:51.011700 23593 consensus_queue.cc:260] T d1a134ba33ea4076bbaac0ae460c17ba P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:51.002529 23116 catalog_manager.cc:5495] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P c0f38c1908414566aa9115bb2cbdd85d reported cstate change: term changed from 0 to 1, leader changed from <none> to c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66). New cstate: current_term: 1 leader_uuid: "c0f38c1908414566aa9115bb2cbdd85d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } health_report { overall_health: UNKNOWN } } }
I20260809 15:45:51.005270 23591 tablet_bootstrap.cc:492] T 563daf42d7784db195ea99c3cb21b370 P c0f38c1908414566aa9115bb2cbdd85d: No bootstrap required, opened a new log
I20260809 15:45:51.016942 23591 ts_tablet_manager.cc:1397] T 563daf42d7784db195ea99c3cb21b370 P c0f38c1908414566aa9115bb2cbdd85d: Time spent bootstrapping tablet: real 0.052s user 0.011s sys 0.017s
I20260809 15:45:51.019863 23591 raft_consensus.cc:348] T 563daf42d7784db195ea99c3cb21b370 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:51.019942 23664 consensus_queue.cc:1035] T d1a134ba33ea4076bbaac0ae460c17ba P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260809 15:45:51.020524 23591 raft_consensus.cc:374] T 563daf42d7784db195ea99c3cb21b370 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:51.020792 23591 raft_consensus.cc:729] T 563daf42d7784db195ea99c3cb21b370 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Initialized, Role: FOLLOWER
I20260809 15:45:51.021337 23591 consensus_queue.cc:260] T 563daf42d7784db195ea99c3cb21b370 P c0f38c1908414566aa9115bb2cbdd85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:51.025693 23593 ts_tablet_manager.cc:1428] T d1a134ba33ea4076bbaac0ae460c17ba P 65c36d164b0540f099d592f38307ee6c: Time spent starting tablet: real 0.019s user 0.001s sys 0.004s
I20260809 15:45:51.026628 23593 tablet_bootstrap.cc:492] T 7660dfa2ef694927b8eda4100a2bcd9b P 65c36d164b0540f099d592f38307ee6c: Bootstrap starting.
I20260809 15:45:51.028844 23591 ts_tablet_manager.cc:1428] T 563daf42d7784db195ea99c3cb21b370 P c0f38c1908414566aa9115bb2cbdd85d: Time spent starting tablet: real 0.012s user 0.005s sys 0.001s
I20260809 15:45:51.031144 23591 tablet_bootstrap.cc:492] T 297c82ed906d4b48b184985eed3de601 P c0f38c1908414566aa9115bb2cbdd85d: Bootstrap starting.
I20260809 15:45:51.037314 23527 raft_consensus.cc:2454] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 1.
I20260809 15:45:51.038491 23353 leader_election.cc:302] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters: 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:45:51.039983 23648 raft_consensus.cc:2781] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:45:51.040601 23648 raft_consensus.cc:686] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P c0f38c1908414566aa9115bb2cbdd85d [term 1 LEADER]: Becoming Leader. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:45:51.041509 23648 consensus_queue.cc:237] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:51.042651 23593 tablet_bootstrap.cc:654] T 7660dfa2ef694927b8eda4100a2bcd9b P 65c36d164b0540f099d592f38307ee6c: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:51.043877 23924 raft_consensus.cc:482] T 563daf42d7784db195ea99c3cb21b370 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:45:51.044342 23924 raft_consensus.cc:504] T 563daf42d7784db195ea99c3cb21b370 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:51.046965 23924 leader_election.cc:288] T 563daf42d7784db195ea99c3cb21b370 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:45:51.051713 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "563daf42d7784db195ea99c3cb21b370" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:45:51.055410 23591 tablet_bootstrap.cc:654] T 297c82ed906d4b48b184985eed3de601 P c0f38c1908414566aa9115bb2cbdd85d: Neither blocks nor log segments found. Creating new log.
W20260809 15:45:51.056466 23353 leader_election.cc:341] T 563daf42d7784db195ea99c3cb21b370 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:51.061918 23282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "563daf42d7784db195ea99c3cb21b370" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:45:51.069469 23664 raft_consensus.cc:482] T 30c1740ef6874df0a332fbd6fd4b03ce P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:45:51.070001 23664 raft_consensus.cc:504] T 30c1740ef6874df0a332fbd6fd4b03ce P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:51.072701 23664 leader_election.cc:288] T 30c1740ef6874df0a332fbd6fd4b03ce P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
W20260809 15:45:51.073612 23351 leader_election.cc:341] T 563daf42d7784db195ea99c3cb21b370 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:51.074177 23351 leader_election.cc:302] T 563daf42d7784db195ea99c3cb21b370 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:45:51.074880 23286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30c1740ef6874df0a332fbd6fd4b03ce" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:45:51.074927 23790 raft_consensus.cc:2726] T 563daf42d7784db195ea99c3cb21b370 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 15:45:51.075552 23286 raft_consensus.cc:2454] T 30c1740ef6874df0a332fbd6fd4b03ce P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 0.
I20260809 15:45:51.078879 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30c1740ef6874df0a332fbd6fd4b03ce" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
W20260809 15:45:51.080608 23353 leader_election.cc:341] T 30c1740ef6874df0a332fbd6fd4b03ce P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:51.084971 23594 tablet_bootstrap.cc:492] T c5c2e425a74b4e64ae70b5def750434c P 67b3aeadfa7a432abe8af4d292e04ab9: No bootstrap required, opened a new log
I20260809 15:45:51.085443 23594 ts_tablet_manager.cc:1397] T c5c2e425a74b4e64ae70b5def750434c P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent bootstrapping tablet: real 0.107s user 0.020s sys 0.040s
I20260809 15:45:51.087536 23351 leader_election.cc:302] T 30c1740ef6874df0a332fbd6fd4b03ce P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c
I20260809 15:45:51.089141 23924 raft_consensus.cc:2781] T 30c1740ef6874df0a332fbd6fd4b03ce P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 15:45:51.089582 23924 raft_consensus.cc:482] T 30c1740ef6874df0a332fbd6fd4b03ce P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:45:51.090022 23924 raft_consensus.cc:3037] T 30c1740ef6874df0a332fbd6fd4b03ce P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:51.095779 23594 raft_consensus.cc:348] T c5c2e425a74b4e64ae70b5def750434c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:51.102277 23594 raft_consensus.cc:374] T c5c2e425a74b4e64ae70b5def750434c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:51.102634 23594 raft_consensus.cc:729] T c5c2e425a74b4e64ae70b5def750434c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Initialized, Role: FOLLOWER
I20260809 15:45:51.106240 23594 consensus_queue.cc:260] T c5c2e425a74b4e64ae70b5def750434c P 67b3aeadfa7a432abe8af4d292e04ab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:51.108932 23594 ts_tablet_manager.cc:1428] T c5c2e425a74b4e64ae70b5def750434c P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent starting tablet: real 0.016s user 0.005s sys 0.001s
I20260809 15:45:51.109777 23594 tablet_bootstrap.cc:492] T 0acc47ce21c3422698d1ef7277c3ed63 P 67b3aeadfa7a432abe8af4d292e04ab9: Bootstrap starting.
I20260809 15:45:51.115079 23643 raft_consensus.cc:482] T d1a134ba33ea4076bbaac0ae460c17ba P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:45:51.115653 23594 tablet_bootstrap.cc:654] T 0acc47ce21c3422698d1ef7277c3ed63 P 67b3aeadfa7a432abe8af4d292e04ab9: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:51.115521 23643 raft_consensus.cc:504] T d1a134ba33ea4076bbaac0ae460c17ba P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:51.117919 23643 leader_election.cc:288] T d1a134ba33ea4076bbaac0ae460c17ba P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:45:51.121311 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1a134ba33ea4076bbaac0ae460c17ba" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:45:51.121194 23118 catalog_manager.cc:5495] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P c0f38c1908414566aa9115bb2cbdd85d reported cstate change: term changed from 0 to 1, leader changed from <none> to c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66). New cstate: current_term: 1 leader_uuid: "c0f38c1908414566aa9115bb2cbdd85d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } health_report { overall_health: UNKNOWN } } }
I20260809 15:45:51.119869 23593 tablet_bootstrap.cc:492] T 7660dfa2ef694927b8eda4100a2bcd9b P 65c36d164b0540f099d592f38307ee6c: No bootstrap required, opened a new log
I20260809 15:45:51.123905 23593 ts_tablet_manager.cc:1397] T 7660dfa2ef694927b8eda4100a2bcd9b P 65c36d164b0540f099d592f38307ee6c: Time spent bootstrapping tablet: real 0.097s user 0.007s sys 0.005s
I20260809 15:45:51.126612 23593 raft_consensus.cc:348] T 7660dfa2ef694927b8eda4100a2bcd9b P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:51.119901 23280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1a134ba33ea4076bbaac0ae460c17ba" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:45:51.127226 23924 raft_consensus.cc:504] T 30c1740ef6874df0a332fbd6fd4b03ce P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:51.129333 23472 leader_election.cc:302] T d1a134ba33ea4076bbaac0ae460c17ba P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:45:51.130306 23924 leader_election.cc:288] T 30c1740ef6874df0a332fbd6fd4b03ce P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:45:51.130568 23933 raft_consensus.cc:2726] T d1a134ba33ea4076bbaac0ae460c17ba P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 15:45:51.132741 23594 tablet_bootstrap.cc:492] T 0acc47ce21c3422698d1ef7277c3ed63 P 67b3aeadfa7a432abe8af4d292e04ab9: No bootstrap required, opened a new log
I20260809 15:45:51.133129 23594 ts_tablet_manager.cc:1397] T 0acc47ce21c3422698d1ef7277c3ed63 P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent bootstrapping tablet: real 0.024s user 0.009s sys 0.001s
I20260809 15:45:51.134073 23280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30c1740ef6874df0a332fbd6fd4b03ce" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:45:51.134755 23280 raft_consensus.cc:3037] T 30c1740ef6874df0a332fbd6fd4b03ce P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:51.136049 23594 raft_consensus.cc:348] T 0acc47ce21c3422698d1ef7277c3ed63 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:51.136758 23594 raft_consensus.cc:374] T 0acc47ce21c3422698d1ef7277c3ed63 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:51.137020 23594 raft_consensus.cc:729] T 0acc47ce21c3422698d1ef7277c3ed63 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Initialized, Role: FOLLOWER
I20260809 15:45:51.137621 23594 consensus_queue.cc:260] T 0acc47ce21c3422698d1ef7277c3ed63 P 67b3aeadfa7a432abe8af4d292e04ab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:51.139782 23594 ts_tablet_manager.cc:1428] T 0acc47ce21c3422698d1ef7277c3ed63 P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20260809 15:45:51.141150 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30c1740ef6874df0a332fbd6fd4b03ce" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:45:51.142658 23594 tablet_bootstrap.cc:492] T 99bdb93503db4de4a1cfb0161ea0c07c P 67b3aeadfa7a432abe8af4d292e04ab9: Bootstrap starting.
W20260809 15:45:51.142654 23353 leader_election.cc:341] T 30c1740ef6874df0a332fbd6fd4b03ce P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:51.150372 23594 tablet_bootstrap.cc:654] T 99bdb93503db4de4a1cfb0161ea0c07c P 67b3aeadfa7a432abe8af4d292e04ab9: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:51.127236 23593 raft_consensus.cc:374] T 7660dfa2ef694927b8eda4100a2bcd9b P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:51.166625 23593 raft_consensus.cc:729] T 7660dfa2ef694927b8eda4100a2bcd9b P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Initialized, Role: FOLLOWER
I20260809 15:45:51.167251 23593 consensus_queue.cc:260] T 7660dfa2ef694927b8eda4100a2bcd9b P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:51.168774 23593 ts_tablet_manager.cc:1428] T 7660dfa2ef694927b8eda4100a2bcd9b P 65c36d164b0540f099d592f38307ee6c: Time spent starting tablet: real 0.045s user 0.003s sys 0.001s
I20260809 15:45:51.170065 23593 tablet_bootstrap.cc:492] T 763e58d6ba644eea830a6c1664d39c9b P 65c36d164b0540f099d592f38307ee6c: Bootstrap starting.
I20260809 15:45:51.173498 23591 tablet_bootstrap.cc:492] T 297c82ed906d4b48b184985eed3de601 P c0f38c1908414566aa9115bb2cbdd85d: No bootstrap required, opened a new log
I20260809 15:45:51.173905 23591 ts_tablet_manager.cc:1397] T 297c82ed906d4b48b184985eed3de601 P c0f38c1908414566aa9115bb2cbdd85d: Time spent bootstrapping tablet: real 0.143s user 0.020s sys 0.028s
I20260809 15:45:51.181289 23280 raft_consensus.cc:2454] T 30c1740ef6874df0a332fbd6fd4b03ce P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 1.
I20260809 15:45:51.177028 23591 raft_consensus.cc:348] T 297c82ed906d4b48b184985eed3de601 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:51.182315 23591 raft_consensus.cc:374] T 297c82ed906d4b48b184985eed3de601 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:51.182637 23591 raft_consensus.cc:729] T 297c82ed906d4b48b184985eed3de601 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Initialized, Role: FOLLOWER
I20260809 15:45:51.182826 23351 leader_election.cc:302] T 30c1740ef6874df0a332fbd6fd4b03ce P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c
I20260809 15:45:51.183681 23593 tablet_bootstrap.cc:654] T 763e58d6ba644eea830a6c1664d39c9b P 65c36d164b0540f099d592f38307ee6c: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:51.184511 23924 raft_consensus.cc:2781] T 30c1740ef6874df0a332fbd6fd4b03ce P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:45:51.185163 23924 raft_consensus.cc:686] T 30c1740ef6874df0a332fbd6fd4b03ce P c0f38c1908414566aa9115bb2cbdd85d [term 1 LEADER]: Becoming Leader. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:45:51.186043 23591 consensus_queue.cc:260] T 297c82ed906d4b48b184985eed3de601 P c0f38c1908414566aa9115bb2cbdd85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:51.188917 23594 tablet_bootstrap.cc:492] T 99bdb93503db4de4a1cfb0161ea0c07c P 67b3aeadfa7a432abe8af4d292e04ab9: No bootstrap required, opened a new log
I20260809 15:45:51.186472 23924 consensus_queue.cc:237] T 30c1740ef6874df0a332fbd6fd4b03ce P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:51.189354 23594 ts_tablet_manager.cc:1397] T 99bdb93503db4de4a1cfb0161ea0c07c P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent bootstrapping tablet: real 0.047s user 0.018s sys 0.014s
I20260809 15:45:51.189662 23591 ts_tablet_manager.cc:1428] T 297c82ed906d4b48b184985eed3de601 P c0f38c1908414566aa9115bb2cbdd85d: Time spent starting tablet: real 0.016s user 0.002s sys 0.006s
I20260809 15:45:51.190490 23591 tablet_bootstrap.cc:492] T 7660dfa2ef694927b8eda4100a2bcd9b P c0f38c1908414566aa9115bb2cbdd85d: Bootstrap starting.
I20260809 15:45:51.196581 23594 raft_consensus.cc:348] T 99bdb93503db4de4a1cfb0161ea0c07c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:51.197274 23594 raft_consensus.cc:374] T 99bdb93503db4de4a1cfb0161ea0c07c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:51.197562 23594 raft_consensus.cc:729] T 99bdb93503db4de4a1cfb0161ea0c07c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Initialized, Role: FOLLOWER
I20260809 15:45:51.198195 23594 consensus_queue.cc:260] T 99bdb93503db4de4a1cfb0161ea0c07c P 67b3aeadfa7a432abe8af4d292e04ab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:51.202728 23593 tablet_bootstrap.cc:492] T 763e58d6ba644eea830a6c1664d39c9b P 65c36d164b0540f099d592f38307ee6c: No bootstrap required, opened a new log
I20260809 15:45:51.203140 23593 ts_tablet_manager.cc:1397] T 763e58d6ba644eea830a6c1664d39c9b P 65c36d164b0540f099d592f38307ee6c: Time spent bootstrapping tablet: real 0.033s user 0.013s sys 0.006s
I20260809 15:45:51.207228 23594 ts_tablet_manager.cc:1428] T 99bdb93503db4de4a1cfb0161ea0c07c P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent starting tablet: real 0.018s user 0.008s sys 0.000s
I20260809 15:45:51.208055 23594 tablet_bootstrap.cc:492] T a88421dba80d4d3388aa5eb62e23853b P 67b3aeadfa7a432abe8af4d292e04ab9: Bootstrap starting.
I20260809 15:45:51.209394 23593 raft_consensus.cc:348] T 763e58d6ba644eea830a6c1664d39c9b P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:51.210029 23593 raft_consensus.cc:374] T 763e58d6ba644eea830a6c1664d39c9b P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:51.210287 23593 raft_consensus.cc:729] T 763e58d6ba644eea830a6c1664d39c9b P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Initialized, Role: FOLLOWER
I20260809 15:45:51.213200 23593 consensus_queue.cc:260] T 763e58d6ba644eea830a6c1664d39c9b P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:51.215246 23593 ts_tablet_manager.cc:1428] T 763e58d6ba644eea830a6c1664d39c9b P 65c36d164b0540f099d592f38307ee6c: Time spent starting tablet: real 0.012s user 0.004s sys 0.000s
I20260809 15:45:51.220101 23591 tablet_bootstrap.cc:654] T 7660dfa2ef694927b8eda4100a2bcd9b P c0f38c1908414566aa9115bb2cbdd85d: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:51.220731 23593 tablet_bootstrap.cc:492] T 30c1740ef6874df0a332fbd6fd4b03ce P 65c36d164b0540f099d592f38307ee6c: Bootstrap starting.
I20260809 15:45:51.223800 23116 catalog_manager.cc:5495] T 30c1740ef6874df0a332fbd6fd4b03ce P c0f38c1908414566aa9115bb2cbdd85d reported cstate change: term changed from 0 to 1, leader changed from <none> to c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66). New cstate: current_term: 1 leader_uuid: "c0f38c1908414566aa9115bb2cbdd85d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } health_report { overall_health: UNKNOWN } } }
I20260809 15:45:51.231346 23594 tablet_bootstrap.cc:654] T a88421dba80d4d3388aa5eb62e23853b P 67b3aeadfa7a432abe8af4d292e04ab9: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:51.234261 23593 tablet_bootstrap.cc:654] T 30c1740ef6874df0a332fbd6fd4b03ce P 65c36d164b0540f099d592f38307ee6c: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:51.239553 23591 tablet_bootstrap.cc:492] T 7660dfa2ef694927b8eda4100a2bcd9b P c0f38c1908414566aa9115bb2cbdd85d: No bootstrap required, opened a new log
I20260809 15:45:51.239972 23591 ts_tablet_manager.cc:1397] T 7660dfa2ef694927b8eda4100a2bcd9b P c0f38c1908414566aa9115bb2cbdd85d: Time spent bootstrapping tablet: real 0.050s user 0.011s sys 0.010s
I20260809 15:45:51.244570 23591 raft_consensus.cc:348] T 7660dfa2ef694927b8eda4100a2bcd9b P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:51.245218 23591 raft_consensus.cc:374] T 7660dfa2ef694927b8eda4100a2bcd9b P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:51.245482 23591 raft_consensus.cc:729] T 7660dfa2ef694927b8eda4100a2bcd9b P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Initialized, Role: FOLLOWER
I20260809 15:45:51.246035 23591 consensus_queue.cc:260] T 7660dfa2ef694927b8eda4100a2bcd9b P c0f38c1908414566aa9115bb2cbdd85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:51.246665 23594 tablet_bootstrap.cc:492] T a88421dba80d4d3388aa5eb62e23853b P 67b3aeadfa7a432abe8af4d292e04ab9: No bootstrap required, opened a new log
I20260809 15:45:51.248219 23591 ts_tablet_manager.cc:1428] T 7660dfa2ef694927b8eda4100a2bcd9b P c0f38c1908414566aa9115bb2cbdd85d: Time spent starting tablet: real 0.008s user 0.003s sys 0.003s
I20260809 15:45:51.250552 23594 ts_tablet_manager.cc:1397] T a88421dba80d4d3388aa5eb62e23853b P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent bootstrapping tablet: real 0.043s user 0.016s sys 0.005s
I20260809 15:45:51.250774 23593 tablet_bootstrap.cc:492] T 30c1740ef6874df0a332fbd6fd4b03ce P 65c36d164b0540f099d592f38307ee6c: No bootstrap required, opened a new log
I20260809 15:45:51.251030 23591 tablet_bootstrap.cc:492] T 538a228ddfb54bc6a36528d5ff07597b P c0f38c1908414566aa9115bb2cbdd85d: Bootstrap starting.
I20260809 15:45:51.251736 23593 ts_tablet_manager.cc:1397] T 30c1740ef6874df0a332fbd6fd4b03ce P 65c36d164b0540f099d592f38307ee6c: Time spent bootstrapping tablet: real 0.031s user 0.004s sys 0.018s
I20260809 15:45:51.255587 23593 raft_consensus.cc:348] T 30c1740ef6874df0a332fbd6fd4b03ce P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:51.256022 23593 raft_consensus.cc:374] T 30c1740ef6874df0a332fbd6fd4b03ce P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:51.256263 23593 raft_consensus.cc:729] T 30c1740ef6874df0a332fbd6fd4b03ce P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Initialized, Role: FOLLOWER
I20260809 15:45:51.256851 23593 consensus_queue.cc:260] T 30c1740ef6874df0a332fbd6fd4b03ce P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:51.257001 23594 raft_consensus.cc:348] T a88421dba80d4d3388aa5eb62e23853b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:51.257702 23594 raft_consensus.cc:374] T a88421dba80d4d3388aa5eb62e23853b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:51.258030 23594 raft_consensus.cc:729] T a88421dba80d4d3388aa5eb62e23853b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Initialized, Role: FOLLOWER
I20260809 15:45:51.259406 23593 ts_tablet_manager.cc:1428] T 30c1740ef6874df0a332fbd6fd4b03ce P 65c36d164b0540f099d592f38307ee6c: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260809 15:45:51.258790 23594 consensus_queue.cc:260] T a88421dba80d4d3388aa5eb62e23853b P 67b3aeadfa7a432abe8af4d292e04ab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:51.261660 23594 ts_tablet_manager.cc:1428] T a88421dba80d4d3388aa5eb62e23853b P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent starting tablet: real 0.010s user 0.006s sys 0.002s
I20260809 15:45:51.262742 23594 tablet_bootstrap.cc:492] T 538a228ddfb54bc6a36528d5ff07597b P 67b3aeadfa7a432abe8af4d292e04ab9: Bootstrap starting.
I20260809 15:45:51.263823 23593 tablet_bootstrap.cc:492] T 538a228ddfb54bc6a36528d5ff07597b P 65c36d164b0540f099d592f38307ee6c: Bootstrap starting.
I20260809 15:45:51.265591 23280 raft_consensus.cc:3037] T 99bdb93503db4de4a1cfb0161ea0c07c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:51.268347 23594 tablet_bootstrap.cc:654] T 538a228ddfb54bc6a36528d5ff07597b P 67b3aeadfa7a432abe8af4d292e04ab9: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:51.269465 23593 tablet_bootstrap.cc:654] T 538a228ddfb54bc6a36528d5ff07597b P 65c36d164b0540f099d592f38307ee6c: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:51.274744 23591 tablet_bootstrap.cc:654] T 538a228ddfb54bc6a36528d5ff07597b P c0f38c1908414566aa9115bb2cbdd85d: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:51.279592 23594 tablet_bootstrap.cc:492] T 538a228ddfb54bc6a36528d5ff07597b P 67b3aeadfa7a432abe8af4d292e04ab9: No bootstrap required, opened a new log
I20260809 15:45:51.280215 23594 ts_tablet_manager.cc:1397] T 538a228ddfb54bc6a36528d5ff07597b P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent bootstrapping tablet: real 0.018s user 0.012s sys 0.003s
I20260809 15:45:51.282476 23594 raft_consensus.cc:348] T 538a228ddfb54bc6a36528d5ff07597b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:51.283250 23594 raft_consensus.cc:374] T 538a228ddfb54bc6a36528d5ff07597b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:51.283605 23594 raft_consensus.cc:729] T 538a228ddfb54bc6a36528d5ff07597b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Initialized, Role: FOLLOWER
I20260809 15:45:51.284258 23594 consensus_queue.cc:260] T 538a228ddfb54bc6a36528d5ff07597b P 67b3aeadfa7a432abe8af4d292e04ab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:51.286217 23594 ts_tablet_manager.cc:1428] T 538a228ddfb54bc6a36528d5ff07597b P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260809 15:45:51.287235 23594 tablet_bootstrap.cc:492] T 563daf42d7784db195ea99c3cb21b370 P 67b3aeadfa7a432abe8af4d292e04ab9: Bootstrap starting.
I20260809 15:45:51.289322 23593 tablet_bootstrap.cc:492] T 538a228ddfb54bc6a36528d5ff07597b P 65c36d164b0540f099d592f38307ee6c: No bootstrap required, opened a new log
I20260809 15:45:51.289731 23593 ts_tablet_manager.cc:1397] T 538a228ddfb54bc6a36528d5ff07597b P 65c36d164b0540f099d592f38307ee6c: Time spent bootstrapping tablet: real 0.026s user 0.015s sys 0.000s
I20260809 15:45:51.291209 23591 tablet_bootstrap.cc:492] T 538a228ddfb54bc6a36528d5ff07597b P c0f38c1908414566aa9115bb2cbdd85d: No bootstrap required, opened a new log
I20260809 15:45:51.291926 23591 ts_tablet_manager.cc:1397] T 538a228ddfb54bc6a36528d5ff07597b P c0f38c1908414566aa9115bb2cbdd85d: Time spent bootstrapping tablet: real 0.041s user 0.008s sys 0.006s
I20260809 15:45:51.292443 23594 tablet_bootstrap.cc:654] T 563daf42d7784db195ea99c3cb21b370 P 67b3aeadfa7a432abe8af4d292e04ab9: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:51.292367 23593 raft_consensus.cc:348] T 538a228ddfb54bc6a36528d5ff07597b P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:51.293051 23593 raft_consensus.cc:374] T 538a228ddfb54bc6a36528d5ff07597b P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:51.293300 23593 raft_consensus.cc:729] T 538a228ddfb54bc6a36528d5ff07597b P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Initialized, Role: FOLLOWER
I20260809 15:45:51.293859 23593 consensus_queue.cc:260] T 538a228ddfb54bc6a36528d5ff07597b P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:51.294199 23933 consensus_queue.cc:1035] T c5c2e425a74b4e64ae70b5def750434c P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:45:51.294067 23591 raft_consensus.cc:348] T 538a228ddfb54bc6a36528d5ff07597b P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:51.294839 23591 raft_consensus.cc:374] T 538a228ddfb54bc6a36528d5ff07597b P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:51.295096 23591 raft_consensus.cc:729] T 538a228ddfb54bc6a36528d5ff07597b P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Initialized, Role: FOLLOWER
I20260809 15:45:51.295746 23593 ts_tablet_manager.cc:1428] T 538a228ddfb54bc6a36528d5ff07597b P 65c36d164b0540f099d592f38307ee6c: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260809 15:45:51.295771 23591 consensus_queue.cc:260] T 538a228ddfb54bc6a36528d5ff07597b P c0f38c1908414566aa9115bb2cbdd85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:51.296898 23593 tablet_bootstrap.cc:492] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 65c36d164b0540f099d592f38307ee6c: Bootstrap starting.
I20260809 15:45:51.301524 23591 ts_tablet_manager.cc:1428] T 538a228ddfb54bc6a36528d5ff07597b P c0f38c1908414566aa9115bb2cbdd85d: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20260809 15:45:51.301899 23593 tablet_bootstrap.cc:654] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 65c36d164b0540f099d592f38307ee6c: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:51.302314 23591 tablet_bootstrap.cc:492] T d0721a4403a440f383318a3cea9d1442 P c0f38c1908414566aa9115bb2cbdd85d: Bootstrap starting.
I20260809 15:45:51.310333 23594 tablet_bootstrap.cc:492] T 563daf42d7784db195ea99c3cb21b370 P 67b3aeadfa7a432abe8af4d292e04ab9: No bootstrap required, opened a new log
I20260809 15:45:51.311060 23594 ts_tablet_manager.cc:1397] T 563daf42d7784db195ea99c3cb21b370 P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent bootstrapping tablet: real 0.024s user 0.008s sys 0.008s
I20260809 15:45:51.313881 23594 raft_consensus.cc:348] T 563daf42d7784db195ea99c3cb21b370 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:51.314679 23594 raft_consensus.cc:374] T 563daf42d7784db195ea99c3cb21b370 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:51.315013 23594 raft_consensus.cc:729] T 563daf42d7784db195ea99c3cb21b370 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Initialized, Role: FOLLOWER
I20260809 15:45:51.315742 23594 consensus_queue.cc:260] T 563daf42d7784db195ea99c3cb21b370 P 67b3aeadfa7a432abe8af4d292e04ab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:51.316880 23526 raft_consensus.cc:3037] T 763e58d6ba644eea830a6c1664d39c9b P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:51.318120 23594 ts_tablet_manager.cc:1428] T 563daf42d7784db195ea99c3cb21b370 P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent starting tablet: real 0.007s user 0.003s sys 0.002s
I20260809 15:45:51.321894 23594 tablet_bootstrap.cc:492] T 4564dfc6d4094c1b9240b367ccecf3b1 P 67b3aeadfa7a432abe8af4d292e04ab9: Bootstrap starting.
I20260809 15:45:51.323730 23593 tablet_bootstrap.cc:492] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 65c36d164b0540f099d592f38307ee6c: No bootstrap required, opened a new log
I20260809 15:45:51.324137 23593 ts_tablet_manager.cc:1397] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 65c36d164b0540f099d592f38307ee6c: Time spent bootstrapping tablet: real 0.027s user 0.017s sys 0.006s
I20260809 15:45:51.326259 23593 raft_consensus.cc:348] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:51.326857 23593 raft_consensus.cc:374] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:51.326982 23594 tablet_bootstrap.cc:654] T 4564dfc6d4094c1b9240b367ccecf3b1 P 67b3aeadfa7a432abe8af4d292e04ab9: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:51.327138 23593 raft_consensus.cc:729] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Initialized, Role: FOLLOWER
I20260809 15:45:51.327618 23591 tablet_bootstrap.cc:654] T d0721a4403a440f383318a3cea9d1442 P c0f38c1908414566aa9115bb2cbdd85d: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:51.327744 23593 consensus_queue.cc:260] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:51.329782 23593 ts_tablet_manager.cc:1428] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 65c36d164b0540f099d592f38307ee6c: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260809 15:45:51.330597 23593 tablet_bootstrap.cc:492] T 0acc47ce21c3422698d1ef7277c3ed63 P 65c36d164b0540f099d592f38307ee6c: Bootstrap starting.
I20260809 15:45:51.339419 23593 tablet_bootstrap.cc:654] T 0acc47ce21c3422698d1ef7277c3ed63 P 65c36d164b0540f099d592f38307ee6c: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:51.348232 23594 tablet_bootstrap.cc:492] T 4564dfc6d4094c1b9240b367ccecf3b1 P 67b3aeadfa7a432abe8af4d292e04ab9: No bootstrap required, opened a new log
I20260809 15:45:51.348731 23594 ts_tablet_manager.cc:1397] T 4564dfc6d4094c1b9240b367ccecf3b1 P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent bootstrapping tablet: real 0.027s user 0.005s sys 0.015s
I20260809 15:45:51.350976 23594 raft_consensus.cc:348] T 4564dfc6d4094c1b9240b367ccecf3b1 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:51.351881 23594 raft_consensus.cc:374] T 4564dfc6d4094c1b9240b367ccecf3b1 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:51.352190 23594 raft_consensus.cc:729] T 4564dfc6d4094c1b9240b367ccecf3b1 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Initialized, Role: FOLLOWER
I20260809 15:45:51.353466 23280 raft_consensus.cc:3037] T c5c2e425a74b4e64ae70b5def750434c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:51.356724 23591 tablet_bootstrap.cc:492] T d0721a4403a440f383318a3cea9d1442 P c0f38c1908414566aa9115bb2cbdd85d: No bootstrap required, opened a new log
I20260809 15:45:51.357105 23591 ts_tablet_manager.cc:1397] T d0721a4403a440f383318a3cea9d1442 P c0f38c1908414566aa9115bb2cbdd85d: Time spent bootstrapping tablet: real 0.055s user 0.009s sys 0.001s
I20260809 15:45:51.360731 23924 consensus_queue.cc:1035] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260809 15:45:51.362756 23591 raft_consensus.cc:348] T d0721a4403a440f383318a3cea9d1442 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:51.363420 23591 raft_consensus.cc:374] T d0721a4403a440f383318a3cea9d1442 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:51.363667 23591 raft_consensus.cc:729] T d0721a4403a440f383318a3cea9d1442 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Initialized, Role: FOLLOWER
I20260809 15:45:51.364217 23591 consensus_queue.cc:260] T d0721a4403a440f383318a3cea9d1442 P c0f38c1908414566aa9115bb2cbdd85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:51.366082 23591 ts_tablet_manager.cc:1428] T d0721a4403a440f383318a3cea9d1442 P c0f38c1908414566aa9115bb2cbdd85d: Time spent starting tablet: real 0.009s user 0.003s sys 0.000s
I20260809 15:45:51.366849 23591 tablet_bootstrap.cc:492] T 294d4bcc1e7a477cb56455581277a60f P c0f38c1908414566aa9115bb2cbdd85d: Bootstrap starting.
I20260809 15:45:51.367552 23286 raft_consensus.cc:3037] T a88421dba80d4d3388aa5eb62e23853b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:51.368289 23594 consensus_queue.cc:260] T 4564dfc6d4094c1b9240b367ccecf3b1 P 67b3aeadfa7a432abe8af4d292e04ab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:51.373392 23594 ts_tablet_manager.cc:1428] T 4564dfc6d4094c1b9240b367ccecf3b1 P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent starting tablet: real 0.024s user 0.005s sys 0.000s
I20260809 15:45:51.374223 23594 tablet_bootstrap.cc:492] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 67b3aeadfa7a432abe8af4d292e04ab9: Bootstrap starting.
I20260809 15:45:51.375558 23691 consensus_queue.cc:1035] T c5c2e425a74b4e64ae70b5def750434c P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:45:51.375924 23593 tablet_bootstrap.cc:492] T 0acc47ce21c3422698d1ef7277c3ed63 P 65c36d164b0540f099d592f38307ee6c: No bootstrap required, opened a new log
I20260809 15:45:51.376338 23593 ts_tablet_manager.cc:1397] T 0acc47ce21c3422698d1ef7277c3ed63 P 65c36d164b0540f099d592f38307ee6c: Time spent bootstrapping tablet: real 0.046s user 0.005s sys 0.018s
I20260809 15:45:51.381438 23593 raft_consensus.cc:348] T 0acc47ce21c3422698d1ef7277c3ed63 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:51.382061 23593 raft_consensus.cc:374] T 0acc47ce21c3422698d1ef7277c3ed63 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:51.382289 23593 raft_consensus.cc:729] T 0acc47ce21c3422698d1ef7277c3ed63 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Initialized, Role: FOLLOWER
I20260809 15:45:51.383114 23593 consensus_queue.cc:260] T 0acc47ce21c3422698d1ef7277c3ed63 P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:51.386231 23593 ts_tablet_manager.cc:1428] T 0acc47ce21c3422698d1ef7277c3ed63 P 65c36d164b0540f099d592f38307ee6c: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
I20260809 15:45:51.388351 23593 tablet_bootstrap.cc:492] T d0721a4403a440f383318a3cea9d1442 P 65c36d164b0540f099d592f38307ee6c: Bootstrap starting.
I20260809 15:45:51.390225 23591 tablet_bootstrap.cc:654] T 294d4bcc1e7a477cb56455581277a60f P c0f38c1908414566aa9115bb2cbdd85d: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:51.392020 23526 raft_consensus.cc:3037] T d1a134ba33ea4076bbaac0ae460c17ba P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:51.394379 23527 raft_consensus.cc:3037] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:51.399616 23593 tablet_bootstrap.cc:654] T d0721a4403a440f383318a3cea9d1442 P 65c36d164b0540f099d592f38307ee6c: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:51.399839 23691 raft_consensus.cc:482] T 7660dfa2ef694927b8eda4100a2bcd9b P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:45:51.400278 23691 raft_consensus.cc:504] T 7660dfa2ef694927b8eda4100a2bcd9b P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:51.407312 23691 leader_election.cc:288] T 7660dfa2ef694927b8eda4100a2bcd9b P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:45:51.408061 23283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7660dfa2ef694927b8eda4100a2bcd9b" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:45:51.408221 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7660dfa2ef694927b8eda4100a2bcd9b" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:45:51.408712 23283 raft_consensus.cc:2454] T 7660dfa2ef694927b8eda4100a2bcd9b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 0.
I20260809 15:45:51.408826 23402 raft_consensus.cc:2454] T 7660dfa2ef694927b8eda4100a2bcd9b P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 0.
I20260809 15:45:51.411029 23470 leader_election.cc:302] T 7660dfa2ef694927b8eda4100a2bcd9b P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:45:51.411880 23594 tablet_bootstrap.cc:654] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 67b3aeadfa7a432abe8af4d292e04ab9: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:51.413851 23933 raft_consensus.cc:2781] T 7660dfa2ef694927b8eda4100a2bcd9b P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 15:45:51.414207 23933 raft_consensus.cc:482] T 7660dfa2ef694927b8eda4100a2bcd9b P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:45:51.414528 23933 raft_consensus.cc:3037] T 7660dfa2ef694927b8eda4100a2bcd9b P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:51.421732 23933 raft_consensus.cc:504] T 7660dfa2ef694927b8eda4100a2bcd9b P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:51.425559 23648 consensus_queue.cc:1035] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:45:51.426550 23933 leader_election.cc:288] T 7660dfa2ef694927b8eda4100a2bcd9b P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 1 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:45:51.427951 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7660dfa2ef694927b8eda4100a2bcd9b" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:45:51.428514 23402 raft_consensus.cc:3037] T 7660dfa2ef694927b8eda4100a2bcd9b P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:51.433910 23285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7660dfa2ef694927b8eda4100a2bcd9b" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:45:51.434446 23285 raft_consensus.cc:3037] T 7660dfa2ef694927b8eda4100a2bcd9b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:51.434830 23691 raft_consensus.cc:482] T 30c1740ef6874df0a332fbd6fd4b03ce P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:45:51.435345 23691 raft_consensus.cc:504] T 30c1740ef6874df0a332fbd6fd4b03ce P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:51.437414 23691 leader_election.cc:288] T 30c1740ef6874df0a332fbd6fd4b03ce P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:45:51.438295 23399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30c1740ef6874df0a332fbd6fd4b03ce" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:45:51.438489 23661 raft_consensus.cc:482] T 0acc47ce21c3422698d1ef7277c3ed63 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:45:51.438885 23661 raft_consensus.cc:504] T 0acc47ce21c3422698d1ef7277c3ed63 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:51.440634 23661 leader_election.cc:288] T 0acc47ce21c3422698d1ef7277c3ed63 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:45:51.442906 23285 raft_consensus.cc:2454] T 7660dfa2ef694927b8eda4100a2bcd9b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 1.
I20260809 15:45:51.443039 23529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0acc47ce21c3422698d1ef7277c3ed63" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:45:51.446460 23529 raft_consensus.cc:2454] T 0acc47ce21c3422698d1ef7277c3ed63 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 0.
I20260809 15:45:51.446591 23282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30c1740ef6874df0a332fbd6fd4b03ce" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:45:51.443907 23399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0acc47ce21c3422698d1ef7277c3ed63" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:45:51.447453 23282 raft_consensus.cc:2379] T 30c1740ef6874df0a332fbd6fd4b03ce P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c in current term 1: Already voted for candidate c0f38c1908414566aa9115bb2cbdd85d in this term.
I20260809 15:45:51.447762 23399 raft_consensus.cc:2454] T 0acc47ce21c3422698d1ef7277c3ed63 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 0.
I20260809 15:45:51.448607 23470 leader_election.cc:302] T 30c1740ef6874df0a332fbd6fd4b03ce P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:45:51.450510 23470 leader_election.cc:302] T 7660dfa2ef694927b8eda4100a2bcd9b P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:45:51.448566 23228 leader_election.cc:302] T 0acc47ce21c3422698d1ef7277c3ed63 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:45:51.451977 23655 raft_consensus.cc:3037] T 30c1740ef6874df0a332fbd6fd4b03ce P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:51.452412 23661 raft_consensus.cc:2781] T 0acc47ce21c3422698d1ef7277c3ed63 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 15:45:51.452767 23661 raft_consensus.cc:482] T 0acc47ce21c3422698d1ef7277c3ed63 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:45:51.453088 23661 raft_consensus.cc:3037] T 0acc47ce21c3422698d1ef7277c3ed63 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:51.455163 23594 tablet_bootstrap.cc:492] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 67b3aeadfa7a432abe8af4d292e04ab9: No bootstrap required, opened a new log
I20260809 15:45:51.455209 23648 raft_consensus.cc:482] T 538a228ddfb54bc6a36528d5ff07597b P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:45:51.455704 23594 ts_tablet_manager.cc:1397] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent bootstrapping tablet: real 0.082s user 0.007s sys 0.005s
I20260809 15:45:51.455756 23648 raft_consensus.cc:504] T 538a228ddfb54bc6a36528d5ff07597b P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:51.458225 23648 leader_election.cc:288] T 538a228ddfb54bc6a36528d5ff07597b P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:45:51.458060 23594 raft_consensus.cc:348] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:51.458730 23594 raft_consensus.cc:374] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:51.458993 23594 raft_consensus.cc:729] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Initialized, Role: FOLLOWER
I20260809 15:45:51.459599 23594 consensus_queue.cc:260] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 67b3aeadfa7a432abe8af4d292e04ab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:51.460904 23402 raft_consensus.cc:2454] T 7660dfa2ef694927b8eda4100a2bcd9b P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 1.
I20260809 15:45:51.461416 23594 ts_tablet_manager.cc:1428] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260809 15:45:51.462752 23591 tablet_bootstrap.cc:492] T 294d4bcc1e7a477cb56455581277a60f P c0f38c1908414566aa9115bb2cbdd85d: No bootstrap required, opened a new log
I20260809 15:45:51.463591 23593 tablet_bootstrap.cc:492] T d0721a4403a440f383318a3cea9d1442 P 65c36d164b0540f099d592f38307ee6c: No bootstrap required, opened a new log
I20260809 15:45:51.465104 23594 tablet_bootstrap.cc:492] T 00a1881267a14ba8b78c81040f17a014 P 67b3aeadfa7a432abe8af4d292e04ab9: Bootstrap starting.
I20260809 15:45:51.465389 23591 ts_tablet_manager.cc:1397] T 294d4bcc1e7a477cb56455581277a60f P c0f38c1908414566aa9115bb2cbdd85d: Time spent bootstrapping tablet: real 0.099s user 0.009s sys 0.001s
I20260809 15:45:51.465492 23593 ts_tablet_manager.cc:1397] T d0721a4403a440f383318a3cea9d1442 P 65c36d164b0540f099d592f38307ee6c: Time spent bootstrapping tablet: real 0.077s user 0.009s sys 0.003s
I20260809 15:45:51.459695 23529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "538a228ddfb54bc6a36528d5ff07597b" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:45:51.466190 23529 raft_consensus.cc:2454] T 538a228ddfb54bc6a36528d5ff07597b P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 0.
I20260809 15:45:51.467252 23353 leader_election.cc:302] T 538a228ddfb54bc6a36528d5ff07597b P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:45:51.459555 23691 raft_consensus.cc:2781] T 7660dfa2ef694927b8eda4100a2bcd9b P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:45:51.467864 23593 raft_consensus.cc:348] T d0721a4403a440f383318a3cea9d1442 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:51.468160 23648 raft_consensus.cc:2781] T 538a228ddfb54bc6a36528d5ff07597b P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 15:45:51.468120 23691 raft_consensus.cc:686] T 7660dfa2ef694927b8eda4100a2bcd9b P 65c36d164b0540f099d592f38307ee6c [term 1 LEADER]: Becoming Leader. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:45:51.468524 23593 raft_consensus.cc:374] T d0721a4403a440f383318a3cea9d1442 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:51.468595 23648 raft_consensus.cc:482] T 538a228ddfb54bc6a36528d5ff07597b P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:45:51.468866 23593 raft_consensus.cc:729] T d0721a4403a440f383318a3cea9d1442 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Initialized, Role: FOLLOWER
I20260809 15:45:51.469359 23691 consensus_queue.cc:237] T 7660dfa2ef694927b8eda4100a2bcd9b P 65c36d164b0540f099d592f38307ee6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:51.469485 23593 consensus_queue.cc:260] T d0721a4403a440f383318a3cea9d1442 P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:51.471370 23722 raft_consensus.cc:482] T 4564dfc6d4094c1b9240b367ccecf3b1 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:45:51.472455 23722 raft_consensus.cc:504] T 4564dfc6d4094c1b9240b367ccecf3b1 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:51.473608 23593 ts_tablet_manager.cc:1428] T d0721a4403a440f383318a3cea9d1442 P 65c36d164b0540f099d592f38307ee6c: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20260809 15:45:51.474403 23593 tablet_bootstrap.cc:492] T 4bcac4b24e7a4448abdc934bba5a624e P 65c36d164b0540f099d592f38307ee6c: Bootstrap starting.
I20260809 15:45:51.474210 23591 raft_consensus.cc:348] T 294d4bcc1e7a477cb56455581277a60f P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:51.474218 23722 leader_election.cc:288] T 4564dfc6d4094c1b9240b367ccecf3b1 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:45:51.474859 23591 raft_consensus.cc:374] T 294d4bcc1e7a477cb56455581277a60f P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:51.475119 23591 raft_consensus.cc:729] T 294d4bcc1e7a477cb56455581277a60f P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Initialized, Role: FOLLOWER
I20260809 15:45:51.478484 23521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4564dfc6d4094c1b9240b367ccecf3b1" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:45:51.482671 23661 raft_consensus.cc:504] T 0acc47ce21c3422698d1ef7277c3ed63 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:51.486145 23593 tablet_bootstrap.cc:654] T 4bcac4b24e7a4448abdc934bba5a624e P 65c36d164b0540f099d592f38307ee6c: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:51.494500 23661 leader_election.cc:288] T 0acc47ce21c3422698d1ef7277c3ed63 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:45:51.495797 23594 tablet_bootstrap.cc:654] T 00a1881267a14ba8b78c81040f17a014 P 67b3aeadfa7a432abe8af4d292e04ab9: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:51.486433 23399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4564dfc6d4094c1b9240b367ccecf3b1" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:45:51.500897 23790 consensus_queue.cc:1035] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:45:51.504722 23593 tablet_bootstrap.cc:492] T 4bcac4b24e7a4448abdc934bba5a624e P 65c36d164b0540f099d592f38307ee6c: No bootstrap required, opened a new log
I20260809 15:45:51.505154 23593 ts_tablet_manager.cc:1397] T 4bcac4b24e7a4448abdc934bba5a624e P 65c36d164b0540f099d592f38307ee6c: Time spent bootstrapping tablet: real 0.031s user 0.002s sys 0.009s
W20260809 15:45:51.487396 23228 leader_election.cc:341] T 4564dfc6d4094c1b9240b367ccecf3b1 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:51.512319 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0acc47ce21c3422698d1ef7277c3ed63" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:45:51.512837 23402 raft_consensus.cc:3037] T 0acc47ce21c3422698d1ef7277c3ed63 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:51.518312 23931 raft_consensus.cc:482] T d0721a4403a440f383318a3cea9d1442 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:45:51.518726 23931 raft_consensus.cc:504] T d0721a4403a440f383318a3cea9d1442 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:51.518872 23593 raft_consensus.cc:348] T 4bcac4b24e7a4448abdc934bba5a624e P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:51.520262 23593 raft_consensus.cc:374] T 4bcac4b24e7a4448abdc934bba5a624e P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:51.518813 23402 raft_consensus.cc:2454] T 0acc47ce21c3422698d1ef7277c3ed63 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 1.
I20260809 15:45:51.520526 23593 raft_consensus.cc:729] T 4bcac4b24e7a4448abdc934bba5a624e P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Initialized, Role: FOLLOWER
I20260809 15:45:51.520932 23931 leader_election.cc:288] T d0721a4403a440f383318a3cea9d1442 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:45:51.523684 23528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0721a4403a440f383318a3cea9d1442" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:45:51.524171 23528 raft_consensus.cc:2454] T d0721a4403a440f383318a3cea9d1442 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 0.
I20260809 15:45:51.525121 23353 leader_election.cc:302] T d0721a4403a440f383318a3cea9d1442 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:45:51.526392 23594 tablet_bootstrap.cc:492] T 00a1881267a14ba8b78c81040f17a014 P 67b3aeadfa7a432abe8af4d292e04ab9: No bootstrap required, opened a new log
I20260809 15:45:51.526880 23594 ts_tablet_manager.cc:1397] T 00a1881267a14ba8b78c81040f17a014 P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent bootstrapping tablet: real 0.062s user 0.010s sys 0.005s
I20260809 15:45:51.528128 23283 raft_consensus.cc:3037] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:51.528401 23593 consensus_queue.cc:260] T 4bcac4b24e7a4448abdc934bba5a624e P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:51.532998 23593 ts_tablet_manager.cc:1428] T 4bcac4b24e7a4448abdc934bba5a624e P 65c36d164b0540f099d592f38307ee6c: Time spent starting tablet: real 0.028s user 0.001s sys 0.005s
I20260809 15:45:51.533856 23593 tablet_bootstrap.cc:492] T 563daf42d7784db195ea99c3cb21b370 P 65c36d164b0540f099d592f38307ee6c: Bootstrap starting.
I20260809 15:45:51.536087 23277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0721a4403a440f383318a3cea9d1442" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:45:51.536726 23277 raft_consensus.cc:2454] T d0721a4403a440f383318a3cea9d1442 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 0.
I20260809 15:45:51.541626 23594 raft_consensus.cc:348] T 00a1881267a14ba8b78c81040f17a014 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:51.542333 23594 raft_consensus.cc:374] T 00a1881267a14ba8b78c81040f17a014 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:51.542608 23594 raft_consensus.cc:729] T 00a1881267a14ba8b78c81040f17a014 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Initialized, Role: FOLLOWER
I20260809 15:45:51.526785 23790 raft_consensus.cc:2781] T d0721a4403a440f383318a3cea9d1442 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 15:45:51.543581 23790 raft_consensus.cc:482] T d0721a4403a440f383318a3cea9d1442 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:45:51.543963 23790 raft_consensus.cc:3037] T d0721a4403a440f383318a3cea9d1442 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:51.458529 23655 raft_consensus.cc:2726] T 30c1740ef6874df0a332fbd6fd4b03ce P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 15:45:51.490139 23648 raft_consensus.cc:3037] T 538a228ddfb54bc6a36528d5ff07597b P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:51.543164 23594 consensus_queue.cc:260] T 00a1881267a14ba8b78c81040f17a014 P 67b3aeadfa7a432abe8af4d292e04ab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:51.543094 23529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0acc47ce21c3422698d1ef7277c3ed63" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:45:51.551934 23529 raft_consensus.cc:3037] T 0acc47ce21c3422698d1ef7277c3ed63 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:51.551856 23594 ts_tablet_manager.cc:1428] T 00a1881267a14ba8b78c81040f17a014 P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent starting tablet: real 0.025s user 0.007s sys 0.000s
I20260809 15:45:51.555047 23594 tablet_bootstrap.cc:492] T df20671a7b874dbb98c0ef28371c1308 P 67b3aeadfa7a432abe8af4d292e04ab9: Bootstrap starting.
W20260809 15:45:51.556620 23228 leader_election.cc:341] T 4564dfc6d4094c1b9240b367ccecf3b1 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:51.557085 23228 leader_election.cc:302] T 4564dfc6d4094c1b9240b367ccecf3b1 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:45:51.557780 24022 raft_consensus.cc:2726] T 4564dfc6d4094c1b9240b367ccecf3b1 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 15:45:51.561717 23593 tablet_bootstrap.cc:654] T 563daf42d7784db195ea99c3cb21b370 P 65c36d164b0540f099d592f38307ee6c: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:51.569931 23648 raft_consensus.cc:504] T 538a228ddfb54bc6a36528d5ff07597b P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:51.571661 23931 consensus_queue.cc:1035] T 30c1740ef6874df0a332fbd6fd4b03ce P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:45:51.573074 23593 tablet_bootstrap.cc:492] T 563daf42d7784db195ea99c3cb21b370 P 65c36d164b0540f099d592f38307ee6c: No bootstrap required, opened a new log
I20260809 15:45:51.573890 23594 tablet_bootstrap.cc:654] T df20671a7b874dbb98c0ef28371c1308 P 67b3aeadfa7a432abe8af4d292e04ab9: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:51.574687 23591 consensus_queue.cc:260] T 294d4bcc1e7a477cb56455581277a60f P c0f38c1908414566aa9115bb2cbdd85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:51.600819 23593 ts_tablet_manager.cc:1397] T 563daf42d7784db195ea99c3cb21b370 P 65c36d164b0540f099d592f38307ee6c: Time spent bootstrapping tablet: real 0.067s user 0.009s sys 0.001s
I20260809 15:45:51.558809 23228 leader_election.cc:302] T 0acc47ce21c3422698d1ef7277c3ed63 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:45:51.577819 23285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "538a228ddfb54bc6a36528d5ff07597b" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:45:51.579551 24022 raft_consensus.cc:482] T 538a228ddfb54bc6a36528d5ff07597b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:45:51.586330 23790 raft_consensus.cc:504] T d0721a4403a440f383318a3cea9d1442 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:51.616405 23528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0721a4403a440f383318a3cea9d1442" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:45:51.617002 23528 raft_consensus.cc:3037] T d0721a4403a440f383318a3cea9d1442 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:51.595148 23529 raft_consensus.cc:2454] T 0acc47ce21c3422698d1ef7277c3ed63 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 1.
I20260809 15:45:51.625574 23931 raft_consensus.cc:482] T 0a51c0baaa6e432997c9d3552af4f254 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:45:51.626592 23593 raft_consensus.cc:348] T 563daf42d7784db195ea99c3cb21b370 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:51.627499 23593 raft_consensus.cc:374] T 563daf42d7784db195ea99c3cb21b370 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:51.627749 23593 raft_consensus.cc:729] T 563daf42d7784db195ea99c3cb21b370 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Initialized, Role: FOLLOWER
I20260809 15:45:51.628264 23591 ts_tablet_manager.cc:1428] T 294d4bcc1e7a477cb56455581277a60f P c0f38c1908414566aa9115bb2cbdd85d: Time spent starting tablet: real 0.156s user 0.006s sys 0.000s
I20260809 15:45:51.629086 23591 tablet_bootstrap.cc:492] T 4564dfc6d4094c1b9240b367ccecf3b1 P c0f38c1908414566aa9115bb2cbdd85d: Bootstrap starting.
I20260809 15:45:51.634186 23593 consensus_queue.cc:260] T 563daf42d7784db195ea99c3cb21b370 P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:51.602783 23924 raft_consensus.cc:482] T 294d4bcc1e7a477cb56455581277a60f P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:45:51.603912 23661 raft_consensus.cc:2781] T 0acc47ce21c3422698d1ef7277c3ed63 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:45:51.626034 23931 raft_consensus.cc:504] T 0a51c0baaa6e432997c9d3552af4f254 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:51.639106 23593 ts_tablet_manager.cc:1428] T 563daf42d7784db195ea99c3cb21b370 P 65c36d164b0540f099d592f38307ee6c: Time spent starting tablet: real 0.038s user 0.005s sys 0.001s
I20260809 15:45:51.640560 23593 tablet_bootstrap.cc:492] T 213543cdbe814c89a17a963469922c1b P 65c36d164b0540f099d592f38307ee6c: Bootstrap starting.
I20260809 15:45:51.638525 23790 leader_election.cc:288] T d0721a4403a440f383318a3cea9d1442 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:45:51.623931 24022 raft_consensus.cc:504] T 538a228ddfb54bc6a36528d5ff07597b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:45:51.638418 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 89926.5us
I20260809 15:45:51.648698 23528 raft_consensus.cc:2454] T d0721a4403a440f383318a3cea9d1442 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 1.
I20260809 15:45:51.667338 23285 raft_consensus.cc:2454] T 538a228ddfb54bc6a36528d5ff07597b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 0.
I20260809 15:45:51.641103 23924 raft_consensus.cc:504] T 294d4bcc1e7a477cb56455581277a60f P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:51.672937 23648 leader_election.cc:288] T 538a228ddfb54bc6a36528d5ff07597b P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:45:51.684127 23931 leader_election.cc:288] T 0a51c0baaa6e432997c9d3552af4f254 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:45:51.688022 24022 leader_election.cc:288] T 538a228ddfb54bc6a36528d5ff07597b P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:45:51.708104 23591 tablet_bootstrap.cc:654] T 4564dfc6d4094c1b9240b367ccecf3b1 P c0f38c1908414566aa9115bb2cbdd85d: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:51.691177 23924 leader_election.cc:288] T 294d4bcc1e7a477cb56455581277a60f P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:45:51.718173 23594 tablet_bootstrap.cc:492] T df20671a7b874dbb98c0ef28371c1308 P 67b3aeadfa7a432abe8af4d292e04ab9: No bootstrap required, opened a new log
I20260809 15:45:51.718585 23594 ts_tablet_manager.cc:1397] T df20671a7b874dbb98c0ef28371c1308 P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent bootstrapping tablet: real 0.164s user 0.010s sys 0.005s
I20260809 15:45:51.721156 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "538a228ddfb54bc6a36528d5ff07597b" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:45:51.721827 23402 raft_consensus.cc:2379] T 538a228ddfb54bc6a36528d5ff07597b P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 in current term 1: Already voted for candidate c0f38c1908414566aa9115bb2cbdd85d in this term.
I20260809 15:45:51.716681 24125 raft_consensus.cc:482] T 297c82ed906d4b48b184985eed3de601 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:45:51.724022 24125 raft_consensus.cc:504] T 297c82ed906d4b48b184985eed3de601 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:51.724766 23593 tablet_bootstrap.cc:654] T 213543cdbe814c89a17a963469922c1b P 65c36d164b0540f099d592f38307ee6c: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:51.715561 24137 raft_consensus.cc:482] T 0a51c0baaa6e432997c9d3552af4f254 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:45:51.713840 23661 raft_consensus.cc:686] T 0acc47ce21c3422698d1ef7277c3ed63 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 LEADER]: Becoming Leader. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Running, Role: LEADER
I20260809 15:45:51.728101 24137 raft_consensus.cc:504] T 0a51c0baaa6e432997c9d3552af4f254 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:51.730945 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "538a228ddfb54bc6a36528d5ff07597b" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:45:51.731539 23527 raft_consensus.cc:2454] T 538a228ddfb54bc6a36528d5ff07597b P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 0.
I20260809 15:45:51.726962 24136 raft_consensus.cc:482] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:45:51.745983 24137 leader_election.cc:288] T 0a51c0baaa6e432997c9d3552af4f254 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:45:51.746656 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a51c0baaa6e432997c9d3552af4f254" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:45:51.747648 23402 raft_consensus.cc:2454] T 0a51c0baaa6e432997c9d3552af4f254 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 1.
I20260809 15:45:51.753417 23228 leader_election.cc:302] T 0a51c0baaa6e432997c9d3552af4f254 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:45:51.727288 23924 consensus_queue.cc:1035] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.018s
I20260809 15:45:51.758304 23286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0721a4403a440f383318a3cea9d1442" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:45:51.745532 24125 leader_election.cc:288] T 297c82ed906d4b48b184985eed3de601 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:45:51.761936 23286 raft_consensus.cc:3037] T d0721a4403a440f383318a3cea9d1442 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:51.768985 23353 leader_election.cc:302] T d0721a4403a440f383318a3cea9d1442 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:45:51.769985 23283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a51c0baaa6e432997c9d3552af4f254" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:45:51.746718 24136 raft_consensus.cc:504] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:51.753957 23280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "538a228ddfb54bc6a36528d5ff07597b" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:45:51.772801 24123 raft_consensus.cc:2781] T d0721a4403a440f383318a3cea9d1442 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:45:51.773121 24134 raft_consensus.cc:2781] T 0a51c0baaa6e432997c9d3552af4f254 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260809 15:45:51.773268 24123 raft_consensus.cc:686] T d0721a4403a440f383318a3cea9d1442 P c0f38c1908414566aa9115bb2cbdd85d [term 1 LEADER]: Becoming Leader. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:45:51.773523 24134 raft_consensus.cc:482] T 0a51c0baaa6e432997c9d3552af4f254 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:45:51.773790 24136 leader_election.cc:288] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:45:51.773882 24134 raft_consensus.cc:3037] T 0a51c0baaa6e432997c9d3552af4f254 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:45:51.774266 23528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "538a228ddfb54bc6a36528d5ff07597b" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:45:51.778782 23528 raft_consensus.cc:3037] T 538a228ddfb54bc6a36528d5ff07597b P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:51.782805 23722 raft_consensus.cc:482] T 563daf42d7784db195ea99c3cb21b370 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:45:51.770416 23661 consensus_queue.cc:237] T 0acc47ce21c3422698d1ef7277c3ed63 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:51.783222 23722 raft_consensus.cc:504] T 563daf42d7784db195ea99c3cb21b370 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:51.784874 23722 leader_election.cc:288] T 563daf42d7784db195ea99c3cb21b370 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:45:51.785518 23594 raft_consensus.cc:348] T df20671a7b874dbb98c0ef28371c1308 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:51.786368 23594 raft_consensus.cc:374] T df20671a7b874dbb98c0ef28371c1308 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:51.787004 23594 raft_consensus.cc:729] T df20671a7b874dbb98c0ef28371c1308 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Initialized, Role: FOLLOWER
I20260809 15:45:51.778220 23280 raft_consensus.cc:3037] T 538a228ddfb54bc6a36528d5ff07597b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:51.788122 23594 consensus_queue.cc:260] T df20671a7b874dbb98c0ef28371c1308 P 67b3aeadfa7a432abe8af4d292e04ab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:51.789031 23529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a51c0baaa6e432997c9d3552af4f254" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:45:51.790051 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "563daf42d7784db195ea99c3cb21b370" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:45:51.790457 23529 raft_consensus.cc:2454] T 0a51c0baaa6e432997c9d3552af4f254 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 1.
I20260809 15:45:51.790591 23402 raft_consensus.cc:2454] T 563daf42d7784db195ea99c3cb21b370 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 0.
I20260809 15:45:51.785717 23643 raft_consensus.cc:482] T 563daf42d7784db195ea99c3cb21b370 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:45:51.791805 23228 leader_election.cc:302] T 563daf42d7784db195ea99c3cb21b370 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:45:51.791100 23399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f9d6565aa4c45a3b3f01210d0eb71f7" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:45:51.795562 24123 consensus_queue.cc:237] T d0721a4403a440f383318a3cea9d1442 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:51.799865 23520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "294d4bcc1e7a477cb56455581277a60f" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:45:51.806427 23591 tablet_bootstrap.cc:492] T 4564dfc6d4094c1b9240b367ccecf3b1 P c0f38c1908414566aa9115bb2cbdd85d: No bootstrap required, opened a new log
I20260809 15:45:51.806869 23591 ts_tablet_manager.cc:1397] T 4564dfc6d4094c1b9240b367ccecf3b1 P c0f38c1908414566aa9115bb2cbdd85d: Time spent bootstrapping tablet: real 0.178s user 0.014s sys 0.004s
I20260809 15:45:51.808228 23276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "294d4bcc1e7a477cb56455581277a60f" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:45:51.811411 23722 raft_consensus.cc:2781] T 563daf42d7784db195ea99c3cb21b370 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 15:45:51.811758 23722 raft_consensus.cc:482] T 563daf42d7784db195ea99c3cb21b370 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:45:51.812062 23722 raft_consensus.cc:3037] T 563daf42d7784db195ea99c3cb21b370 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:51.817847 23594 ts_tablet_manager.cc:1428] T df20671a7b874dbb98c0ef28371c1308 P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent starting tablet: real 0.099s user 0.013s sys 0.003s
I20260809 15:45:51.818730 23594 tablet_bootstrap.cc:492] T 28e17dcb5efe45b8b6a8c42da03b4819 P 67b3aeadfa7a432abe8af4d292e04ab9: Bootstrap starting.
I20260809 15:45:51.794214 23643 raft_consensus.cc:504] T 563daf42d7784db195ea99c3cb21b370 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:51.827555 23643 leader_election.cc:288] T 563daf42d7784db195ea99c3cb21b370 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:45:51.830941 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "297c82ed906d4b48b184985eed3de601" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:45:51.832305 23276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "297c82ed906d4b48b184985eed3de601" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:45:51.832911 23276 raft_consensus.cc:2454] T 297c82ed906d4b48b184985eed3de601 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 0.
I20260809 15:45:51.837750 23722 raft_consensus.cc:504] T 563daf42d7784db195ea99c3cb21b370 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:51.838833 23722 leader_election.cc:288] T 563daf42d7784db195ea99c3cb21b370 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
W20260809 15:45:51.836858 23472 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 90233.7us
I20260809 15:45:51.939144 23522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a51c0baaa6e432997c9d3552af4f254" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:45:51.831593 23286 raft_consensus.cc:2454] T d0721a4403a440f383318a3cea9d1442 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 1.
I20260809 15:45:51.861016 23593 tablet_bootstrap.cc:492] T 213543cdbe814c89a17a963469922c1b P 65c36d164b0540f099d592f38307ee6c: No bootstrap required, opened a new log
I20260809 15:45:51.861452 23722 raft_consensus.cc:482] T 00a1881267a14ba8b78c81040f17a014 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:45:51.903164 23594 tablet_bootstrap.cc:654] T 28e17dcb5efe45b8b6a8c42da03b4819 P 67b3aeadfa7a432abe8af4d292e04ab9: Neither blocks nor log segments found. Creating new log.
W20260809 15:45:51.861838 23351 leader_election.cc:341] T 294d4bcc1e7a477cb56455581277a60f P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:51.955693 23522 raft_consensus.cc:2454] T 0a51c0baaa6e432997c9d3552af4f254 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 1.
I20260809 15:45:51.956763 23722 raft_consensus.cc:504] T 00a1881267a14ba8b78c81040f17a014 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:51.956766 23593 ts_tablet_manager.cc:1397] T 213543cdbe814c89a17a963469922c1b P 65c36d164b0540f099d592f38307ee6c: Time spent bootstrapping tablet: real 0.316s user 0.011s sys 0.015s
I20260809 15:45:51.847522 23591 raft_consensus.cc:348] T 4564dfc6d4094c1b9240b367ccecf3b1 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:51.960187 23591 raft_consensus.cc:374] T 4564dfc6d4094c1b9240b367ccecf3b1 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260809 15:45:51.960253 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 118627us
I20260809 15:45:51.960480 23591 raft_consensus.cc:729] T 4564dfc6d4094c1b9240b367ccecf3b1 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Initialized, Role: FOLLOWER
I20260809 15:45:51.960798 23228 leader_election.cc:302] T 538a228ddfb54bc6a36528d5ff07597b P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters: c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:45:51.960997 23591 consensus_queue.cc:260] T 4564dfc6d4094c1b9240b367ccecf3b1 P c0f38c1908414566aa9115bb2cbdd85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:51.911850 23528 raft_consensus.cc:2454] T 538a228ddfb54bc6a36528d5ff07597b P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 1.
I20260809 15:45:51.965294 23722 leader_election.cc:288] T 00a1881267a14ba8b78c81040f17a014 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:45:51.968642 23591 ts_tablet_manager.cc:1428] T 4564dfc6d4094c1b9240b367ccecf3b1 P c0f38c1908414566aa9115bb2cbdd85d: Time spent starting tablet: real 0.161s user 0.006s sys 0.001s
I20260809 15:45:51.874963 23404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "563daf42d7784db195ea99c3cb21b370" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:45:51.975512 23404 raft_consensus.cc:3037] T 563daf42d7784db195ea99c3cb21b370 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:51.875164 23521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "563daf42d7784db195ea99c3cb21b370" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:45:51.977100 23521 raft_consensus.cc:2454] T 563daf42d7784db195ea99c3cb21b370 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 0.
I20260809 15:45:51.875617 23277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "563daf42d7784db195ea99c3cb21b370" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:45:51.875108 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f9d6565aa4c45a3b3f01210d0eb71f7" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
W20260809 15:45:51.967435 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 107300us
I20260809 15:45:51.978107 23400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00a1881267a14ba8b78c81040f17a014" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:45:51.978961 23351 leader_election.cc:302] T 297c82ed906d4b48b184985eed3de601 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:45:51.913995 23280 raft_consensus.cc:2454] T 538a228ddfb54bc6a36528d5ff07597b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 1.
I20260809 15:45:51.978140 23277 raft_consensus.cc:2379] T 563daf42d7784db195ea99c3cb21b370 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c in current term 1: Already voted for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in this term.
W20260809 15:45:51.985574 23228 leader_election.cc:341] T 00a1881267a14ba8b78c81040f17a014 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:51.983196 23117 catalog_manager.cc:5495] T 7660dfa2ef694927b8eda4100a2bcd9b P 65c36d164b0540f099d592f38307ee6c reported cstate change: term changed from 0 to 1, leader changed from <none> to 65c36d164b0540f099d592f38307ee6c (127.20.202.67). New cstate: current_term: 1 leader_uuid: "65c36d164b0540f099d592f38307ee6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } health_report { overall_health: HEALTHY } } }
I20260809 15:45:51.984968 24133 raft_consensus.cc:2781] T 297c82ed906d4b48b184985eed3de601 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 15:45:51.848100 23353 leader_election.cc:302] T 0a51c0baaa6e432997c9d3552af4f254 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:45:51.993727 23722 raft_consensus.cc:482] T df20671a7b874dbb98c0ef28371c1308 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:45:51.997303 23591 tablet_bootstrap.cc:492] T 00a1881267a14ba8b78c81040f17a014 P c0f38c1908414566aa9115bb2cbdd85d: Bootstrap starting.
I20260809 15:45:51.997156 23722 raft_consensus.cc:504] T df20671a7b874dbb98c0ef28371c1308 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:51.998065 23924 consensus_queue.cc:1035] T 30c1740ef6874df0a332fbd6fd4b03ce P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:45:51.996059 23400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "563daf42d7784db195ea99c3cb21b370" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:45:51.990511 24133 raft_consensus.cc:482] T 297c82ed906d4b48b184985eed3de601 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:45:51.999482 24133 raft_consensus.cc:3037] T 297c82ed906d4b48b184985eed3de601 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:52.000226 23722 leader_election.cc:288] T df20671a7b874dbb98c0ef28371c1308 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:45:52.000391 23593 raft_consensus.cc:348] T 213543cdbe814c89a17a963469922c1b P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:52.001049 23593 raft_consensus.cc:374] T 213543cdbe814c89a17a963469922c1b P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:52.001291 23593 raft_consensus.cc:729] T 213543cdbe814c89a17a963469922c1b P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Initialized, Role: FOLLOWER
I20260809 15:45:52.001837 23593 consensus_queue.cc:260] T 213543cdbe814c89a17a963469922c1b P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:52.002892 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df20671a7b874dbb98c0ef28371c1308" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:45:51.904546 24134 raft_consensus.cc:504] T 0a51c0baaa6e432997c9d3552af4f254 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:51.986071 23661 raft_consensus.cc:2741] T 538a228ddfb54bc6a36528d5ff07597b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260809 15:45:52.004361 23118 catalog_manager.cc:5495] T 0acc47ce21c3422698d1ef7277c3ed63 P 67b3aeadfa7a432abe8af4d292e04ab9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65). New cstate: current_term: 1 leader_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } health_report { overall_health: UNKNOWN } } }
W20260809 15:45:52.005487 23228 leader_election.cc:341] T df20671a7b874dbb98c0ef28371c1308 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:52.009835 23593 ts_tablet_manager.cc:1428] T 213543cdbe814c89a17a963469922c1b P 65c36d164b0540f099d592f38307ee6c: Time spent starting tablet: real 0.050s user 0.003s sys 0.002s
I20260809 15:45:52.010607 23593 tablet_bootstrap.cc:492] T 4564dfc6d4094c1b9240b367ccecf3b1 P 65c36d164b0540f099d592f38307ee6c: Bootstrap starting.
W20260809 15:45:52.006953 23353 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 180473us
I20260809 15:45:52.013368 24138 raft_consensus.cc:2781] T 0a51c0baaa6e432997c9d3552af4f254 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260809 15:45:52.013726 24138 raft_consensus.cc:482] T 0a51c0baaa6e432997c9d3552af4f254 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:45:52.014019 24138 raft_consensus.cc:3037] T 0a51c0baaa6e432997c9d3552af4f254 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:45:52.015352 23228 leader_election.cc:302] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:45:52.019677 23283 raft_consensus.cc:2379] T 0a51c0baaa6e432997c9d3552af4f254 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d in current term 2: Already voted for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in this term.
I20260809 15:45:52.025751 24134 leader_election.cc:288] T 0a51c0baaa6e432997c9d3552af4f254 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 2 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:45:52.026866 23404 raft_consensus.cc:2454] T 563daf42d7784db195ea99c3cb21b370 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 1.
I20260809 15:45:52.016952 24136 raft_consensus.cc:2726] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 15:45:52.028445 23228 leader_election.cc:302] T 563daf42d7784db195ea99c3cb21b370 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:45:52.038646 23591 tablet_bootstrap.cc:654] T 00a1881267a14ba8b78c81040f17a014 P c0f38c1908414566aa9115bb2cbdd85d: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:52.029019 23643 raft_consensus.cc:482] T 4bcac4b24e7a4448abdc934bba5a624e P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:45:52.049839 24130 raft_consensus.cc:2781] T 563daf42d7784db195ea99c3cb21b370 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:45:52.050288 24130 raft_consensus.cc:686] T 563daf42d7784db195ea99c3cb21b370 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 LEADER]: Becoming Leader. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Running, Role: LEADER
I20260809 15:45:52.051108 24130 consensus_queue.cc:237] T 563daf42d7784db195ea99c3cb21b370 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:52.048339 23404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a51c0baaa6e432997c9d3552af4f254" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:45:52.052929 23643 raft_consensus.cc:504] T 4bcac4b24e7a4448abdc934bba5a624e P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:52.071954 23594 tablet_bootstrap.cc:492] T 28e17dcb5efe45b8b6a8c42da03b4819 P 67b3aeadfa7a432abe8af4d292e04ab9: No bootstrap required, opened a new log
I20260809 15:45:52.072336 23594 ts_tablet_manager.cc:1397] T 28e17dcb5efe45b8b6a8c42da03b4819 P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent bootstrapping tablet: real 0.254s user 0.007s sys 0.025s
I20260809 15:45:52.081110 23643 leader_election.cc:288] T 4bcac4b24e7a4448abdc934bba5a624e P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:45:52.075141 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "563daf42d7784db195ea99c3cb21b370" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:45:52.081797 23527 raft_consensus.cc:3037] T 563daf42d7784db195ea99c3cb21b370 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:52.077401 23526 raft_consensus.cc:3037] T 4bcac4b24e7a4448abdc934bba5a624e P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Advancing to term 1
W20260809 15:45:52.085436 23193 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
W20260809 15:45:52.075989 23353 leader_election.cc:341] T 294d4bcc1e7a477cb56455581277a60f P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:52.132730 23472 leader_election.cc:302] T 563daf42d7784db195ea99c3cb21b370 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:45:52.098706 23353 leader_election.cc:302] T 294d4bcc1e7a477cb56455581277a60f P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
W20260809 15:45:52.158147 23353 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 82856.4us
I20260809 15:45:52.157944 24133 raft_consensus.cc:504] T 297c82ed906d4b48b184985eed3de601 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:45:52.158881 23353 leader_election.cc:341] T 297c82ed906d4b48b184985eed3de601 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:52.160244 23931 raft_consensus.cc:2726] T 294d4bcc1e7a477cb56455581277a60f P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 15:45:52.163090 23593 tablet_bootstrap.cc:654] T 4564dfc6d4094c1b9240b367ccecf3b1 P 65c36d164b0540f099d592f38307ee6c: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:52.168139 24159 consensus_queue.cc:1035] T 7660dfa2ef694927b8eda4100a2bcd9b P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.006s
I20260809 15:45:52.172552 23521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00a1881267a14ba8b78c81040f17a014" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:45:52.176188 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bcac4b24e7a4448abdc934bba5a624e" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:45:52.176201 24133 leader_election.cc:288] T 297c82ed906d4b48b184985eed3de601 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:45:52.186837 23528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a51c0baaa6e432997c9d3552af4f254" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:45:52.187440 23528 raft_consensus.cc:3037] T 0a51c0baaa6e432997c9d3552af4f254 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:45:52.189143 23521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df20671a7b874dbb98c0ef28371c1308" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
W20260809 15:45:52.191151 23228 leader_election.cc:341] T 00a1881267a14ba8b78c81040f17a014 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:52.191536 23228 leader_election.cc:302] T 00a1881267a14ba8b78c81040f17a014 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:45:52.192210 24134 raft_consensus.cc:2726] T 00a1881267a14ba8b78c81040f17a014 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260809 15:45:52.195533 23228 leader_election.cc:341] T df20671a7b874dbb98c0ef28371c1308 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:52.195945 23228 leader_election.cc:302] T df20671a7b874dbb98c0ef28371c1308 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:45:52.198977 23594 raft_consensus.cc:348] T 28e17dcb5efe45b8b6a8c42da03b4819 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:52.199628 23594 raft_consensus.cc:374] T 28e17dcb5efe45b8b6a8c42da03b4819 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:52.199832 23594 raft_consensus.cc:729] T 28e17dcb5efe45b8b6a8c42da03b4819 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Initialized, Role: FOLLOWER
I20260809 15:45:52.200394 23594 consensus_queue.cc:260] T 28e17dcb5efe45b8b6a8c42da03b4819 P 67b3aeadfa7a432abe8af4d292e04ab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:52.206601 23591 tablet_bootstrap.cc:492] T 00a1881267a14ba8b78c81040f17a014 P c0f38c1908414566aa9115bb2cbdd85d: No bootstrap required, opened a new log
I20260809 15:45:52.207056 23591 ts_tablet_manager.cc:1397] T 00a1881267a14ba8b78c81040f17a014 P c0f38c1908414566aa9115bb2cbdd85d: Time spent bootstrapping tablet: real 0.210s user 0.015s sys 0.012s
I20260809 15:45:52.197957 24136 raft_consensus.cc:2726] T df20671a7b874dbb98c0ef28371c1308 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 15:45:52.211987 23283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bcac4b24e7a4448abdc934bba5a624e" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:45:52.216526 23591 raft_consensus.cc:348] T 00a1881267a14ba8b78c81040f17a014 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:52.217254 23591 raft_consensus.cc:374] T 00a1881267a14ba8b78c81040f17a014 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:52.217571 23591 raft_consensus.cc:729] T 00a1881267a14ba8b78c81040f17a014 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Initialized, Role: FOLLOWER
I20260809 15:45:52.218205 23591 consensus_queue.cc:260] T 00a1881267a14ba8b78c81040f17a014 P c0f38c1908414566aa9115bb2cbdd85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:52.220187 23591 ts_tablet_manager.cc:1428] T 00a1881267a14ba8b78c81040f17a014 P c0f38c1908414566aa9115bb2cbdd85d: Time spent starting tablet: real 0.013s user 0.005s sys 0.000s
I20260809 15:45:52.198460 23351 leader_election.cc:302] T 538a228ddfb54bc6a36528d5ff07597b P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:45:52.230600 23594 ts_tablet_manager.cc:1428] T 28e17dcb5efe45b8b6a8c42da03b4819 P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent starting tablet: real 0.158s user 0.005s sys 0.001s
I20260809 15:45:52.232820 23594 tablet_bootstrap.cc:492] T d3e94b7e50754a81b3d72a07476cb7b9 P 67b3aeadfa7a432abe8af4d292e04ab9: Bootstrap starting.
I20260809 15:45:52.232577 23591 tablet_bootstrap.cc:492] T 213543cdbe814c89a17a963469922c1b P c0f38c1908414566aa9115bb2cbdd85d: Bootstrap starting.
I20260809 15:45:52.240084 23527 raft_consensus.cc:2454] T 563daf42d7784db195ea99c3cb21b370 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 1.
I20260809 15:45:52.240598 23933 raft_consensus.cc:2726] T 563daf42d7784db195ea99c3cb21b370 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 15:45:52.245716 23931 raft_consensus.cc:2781] T 538a228ddfb54bc6a36528d5ff07597b P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:45:52.247534 23931 raft_consensus.cc:686] T 538a228ddfb54bc6a36528d5ff07597b P c0f38c1908414566aa9115bb2cbdd85d [term 1 LEADER]: Becoming Leader. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:45:52.249948 23931 consensus_queue.cc:237] T 538a228ddfb54bc6a36528d5ff07597b P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:52.254175 23470 leader_election.cc:302] T 4bcac4b24e7a4448abdc934bba5a624e P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d
W20260809 15:45:52.256750 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 77042us
I20260809 15:45:52.263080 23790 raft_consensus.cc:482] T 4564dfc6d4094c1b9240b367ccecf3b1 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:45:52.235814 23528 raft_consensus.cc:2454] T 0a51c0baaa6e432997c9d3552af4f254 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 2.
I20260809 15:45:52.252907 24138 raft_consensus.cc:504] T 0a51c0baaa6e432997c9d3552af4f254 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:52.272313 23521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "297c82ed906d4b48b184985eed3de601" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:45:52.274526 23404 raft_consensus.cc:2379] T 0a51c0baaa6e432997c9d3552af4f254 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 in current term 2: Already voted for candidate c0f38c1908414566aa9115bb2cbdd85d in this term.
I20260809 15:45:52.276665 24138 leader_election.cc:288] T 0a51c0baaa6e432997c9d3552af4f254 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 2 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:45:52.271414 23790 raft_consensus.cc:504] T 4564dfc6d4094c1b9240b367ccecf3b1 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:52.278875 23790 leader_election.cc:288] T 4564dfc6d4094c1b9240b367ccecf3b1 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:45:52.282469 23228 leader_election.cc:302] T 0a51c0baaa6e432997c9d3552af4f254 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:45:52.283624 23518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4564dfc6d4094c1b9240b367ccecf3b1" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:45:52.284169 24130 raft_consensus.cc:2781] T 0a51c0baaa6e432997c9d3552af4f254 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 15:45:52.284618 24130 raft_consensus.cc:686] T 0a51c0baaa6e432997c9d3552af4f254 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 LEADER]: Becoming Leader. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Running, Role: LEADER
I20260809 15:45:52.285130 24159 raft_consensus.cc:482] T 213543cdbe814c89a17a963469922c1b P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:45:52.285387 24130 consensus_queue.cc:237] T 0a51c0baaa6e432997c9d3552af4f254 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:52.285545 24159 raft_consensus.cc:504] T 213543cdbe814c89a17a963469922c1b P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:52.291923 23529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a51c0baaa6e432997c9d3552af4f254" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:45:52.293512 23529 raft_consensus.cc:2379] T 0a51c0baaa6e432997c9d3552af4f254 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d in current term 2: Already voted for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in this term.
W20260809 15:45:52.296391 23353 leader_election.cc:341] T 297c82ed906d4b48b184985eed3de601 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553): Illegal state: must be running to vote when last-logged opid is not known
W20260809 15:45:52.307565 23353 leader_election.cc:341] T 4564dfc6d4094c1b9240b367ccecf3b1 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:52.316437 24159 leader_election.cc:288] T 213543cdbe814c89a17a963469922c1b P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:45:52.316313 24136 consensus_queue.cc:1035] T 0acc47ce21c3422698d1ef7277c3ed63 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:45:52.335119 23397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "213543cdbe814c89a17a963469922c1b" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:45:52.361255 23282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "297c82ed906d4b48b184985eed3de601" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:45:52.361802 23282 raft_consensus.cc:3037] T 297c82ed906d4b48b184985eed3de601 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:52.362778 23276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4564dfc6d4094c1b9240b367ccecf3b1" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:45:52.363471 23276 raft_consensus.cc:2454] T 4564dfc6d4094c1b9240b367ccecf3b1 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 0.
I20260809 15:45:52.366304 23594 tablet_bootstrap.cc:654] T d3e94b7e50754a81b3d72a07476cb7b9 P 67b3aeadfa7a432abe8af4d292e04ab9: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:52.377570 23273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a51c0baaa6e432997c9d3552af4f254" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:45:52.379068 23933 raft_consensus.cc:2726] T 4bcac4b24e7a4448abdc934bba5a624e P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 15:45:52.383876 23282 raft_consensus.cc:2454] T 297c82ed906d4b48b184985eed3de601 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 1.
W20260809 15:45:52.370514 23472 leader_election.cc:341] T 213543cdbe814c89a17a963469922c1b P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:52.386998 23593 tablet_bootstrap.cc:492] T 4564dfc6d4094c1b9240b367ccecf3b1 P 65c36d164b0540f099d592f38307ee6c: No bootstrap required, opened a new log
I20260809 15:45:52.387567 23593 ts_tablet_manager.cc:1397] T 4564dfc6d4094c1b9240b367ccecf3b1 P 65c36d164b0540f099d592f38307ee6c: Time spent bootstrapping tablet: real 0.377s user 0.007s sys 0.014s
I20260809 15:45:52.387838 23790 raft_consensus.cc:482] T 00a1881267a14ba8b78c81040f17a014 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:45:52.386619 24159 consensus_queue.cc:1035] T 7660dfa2ef694927b8eda4100a2bcd9b P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
I20260809 15:45:52.391077 23790 raft_consensus.cc:504] T 00a1881267a14ba8b78c81040f17a014 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:52.391806 23591 tablet_bootstrap.cc:654] T 213543cdbe814c89a17a963469922c1b P c0f38c1908414566aa9115bb2cbdd85d: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:52.399233 23790 leader_election.cc:288] T 00a1881267a14ba8b78c81040f17a014 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:45:52.513701 23594 tablet_bootstrap.cc:492] T d3e94b7e50754a81b3d72a07476cb7b9 P 67b3aeadfa7a432abe8af4d292e04ab9: No bootstrap required, opened a new log
I20260809 15:45:52.400194 23283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "213543cdbe814c89a17a963469922c1b" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:45:52.521009 23594 ts_tablet_manager.cc:1397] T d3e94b7e50754a81b3d72a07476cb7b9 P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent bootstrapping tablet: real 0.290s user 0.026s sys 0.070s
I20260809 15:45:52.510721 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00a1881267a14ba8b78c81040f17a014" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:45:52.468362 23116 catalog_manager.cc:5495] T d0721a4403a440f383318a3cea9d1442 P c0f38c1908414566aa9115bb2cbdd85d reported cstate change: term changed from 0 to 1, leader changed from <none> to c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66). New cstate: current_term: 1 leader_uuid: "c0f38c1908414566aa9115bb2cbdd85d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } health_report { overall_health: UNKNOWN } } }
I20260809 15:45:52.518162 23351 leader_election.cc:302] T 4564dfc6d4094c1b9240b367ccecf3b1 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c
I20260809 15:45:52.524427 24123 raft_consensus.cc:2781] T 4564dfc6d4094c1b9240b367ccecf3b1 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 15:45:52.524760 24123 raft_consensus.cc:482] T 4564dfc6d4094c1b9240b367ccecf3b1 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:45:52.525070 24123 raft_consensus.cc:3037] T 4564dfc6d4094c1b9240b367ccecf3b1 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Advancing to term 1
W20260809 15:45:52.507735 24197 log.cc:924] Time spent T 30c1740ef6874df0a332fbd6fd4b03ce P c0f38c1908414566aa9115bb2cbdd85d: Append to log took a long time: real 0.098s user 0.002s sys 0.000s
W20260809 15:45:52.463714 24193 log.cc:924] Time spent T 538a228ddfb54bc6a36528d5ff07597b P c0f38c1908414566aa9115bb2cbdd85d: Append to log took a long time: real 0.097s user 0.004s sys 0.000s
I20260809 15:45:52.511257 23593 raft_consensus.cc:348] T 4564dfc6d4094c1b9240b367ccecf3b1 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:52.530330 23593 raft_consensus.cc:374] T 4564dfc6d4094c1b9240b367ccecf3b1 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:52.530648 23593 raft_consensus.cc:729] T 4564dfc6d4094c1b9240b367ccecf3b1 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Initialized, Role: FOLLOWER
I20260809 15:45:52.531358 23593 consensus_queue.cc:260] T 4564dfc6d4094c1b9240b367ccecf3b1 P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:52.538313 23594 raft_consensus.cc:348] T d3e94b7e50754a81b3d72a07476cb7b9 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:52.538949 23594 raft_consensus.cc:374] T d3e94b7e50754a81b3d72a07476cb7b9 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:52.539201 23594 raft_consensus.cc:729] T d3e94b7e50754a81b3d72a07476cb7b9 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Initialized, Role: FOLLOWER
I20260809 15:45:52.539783 23594 consensus_queue.cc:260] T d3e94b7e50754a81b3d72a07476cb7b9 P 67b3aeadfa7a432abe8af4d292e04ab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:45:52.542744 23353 leader_election.cc:341] T 00a1881267a14ba8b78c81040f17a014 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:52.548704 23593 ts_tablet_manager.cc:1428] T 4564dfc6d4094c1b9240b367ccecf3b1 P 65c36d164b0540f099d592f38307ee6c: Time spent starting tablet: real 0.161s user 0.005s sys 0.000s
I20260809 15:45:52.549623 23593 tablet_bootstrap.cc:492] T 297c82ed906d4b48b184985eed3de601 P 65c36d164b0540f099d592f38307ee6c: Bootstrap starting.
I20260809 15:45:52.543560 23661 consensus_queue.cc:1035] T 563daf42d7784db195ea99c3cb21b370 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
W20260809 15:45:52.551899 23470 leader_election.cc:341] T 213543cdbe814c89a17a963469922c1b P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:52.554332 23470 leader_election.cc:302] T 213543cdbe814c89a17a963469922c1b P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:45:52.555503 23691 raft_consensus.cc:2726] T 213543cdbe814c89a17a963469922c1b P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 15:45:52.560572 24123 raft_consensus.cc:504] T 4564dfc6d4094c1b9240b367ccecf3b1 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:52.563089 24123 leader_election.cc:288] T 4564dfc6d4094c1b9240b367ccecf3b1 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:45:52.564446 23528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4564dfc6d4094c1b9240b367ccecf3b1" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:45:52.565080 23528 raft_consensus.cc:3037] T 4564dfc6d4094c1b9240b367ccecf3b1 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:52.570289 24129 consensus_queue.cc:1035] T d0721a4403a440f383318a3cea9d1442 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260809 15:45:52.578418 23594 ts_tablet_manager.cc:1428] T d3e94b7e50754a81b3d72a07476cb7b9 P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent starting tablet: real 0.057s user 0.006s sys 0.000s
I20260809 15:45:52.579439 23594 tablet_bootstrap.cc:492] T cee02fff2c9c48a19f43eb358a89e7b0 P 67b3aeadfa7a432abe8af4d292e04ab9: Bootstrap starting.
I20260809 15:45:52.590399 23116 catalog_manager.cc:5495] T 563daf42d7784db195ea99c3cb21b370 P 67b3aeadfa7a432abe8af4d292e04ab9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65). New cstate: current_term: 1 leader_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } health_report { overall_health: UNKNOWN } } }
I20260809 15:45:52.596650 23274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00a1881267a14ba8b78c81040f17a014" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:45:52.597362 23661 consensus_queue.cc:1035] T 0acc47ce21c3422698d1ef7277c3ed63 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:45:52.600342 23116 catalog_manager.cc:5495] T 0a51c0baaa6e432997c9d3552af4f254 P 67b3aeadfa7a432abe8af4d292e04ab9 reported cstate change: term changed from 0 to 2, leader changed from <none> to 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65). New cstate: current_term: 2 leader_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } health_report { overall_health: UNKNOWN } } }
I20260809 15:45:52.597318 23274 raft_consensus.cc:2454] T 00a1881267a14ba8b78c81040f17a014 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 0.
I20260809 15:45:52.609315 23591 tablet_bootstrap.cc:492] T 213543cdbe814c89a17a963469922c1b P c0f38c1908414566aa9115bb2cbdd85d: No bootstrap required, opened a new log
I20260809 15:45:52.609807 23591 ts_tablet_manager.cc:1397] T 213543cdbe814c89a17a963469922c1b P c0f38c1908414566aa9115bb2cbdd85d: Time spent bootstrapping tablet: real 0.377s user 0.023s sys 0.002s
I20260809 15:45:52.611132 23528 raft_consensus.cc:2454] T 4564dfc6d4094c1b9240b367ccecf3b1 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 1.
I20260809 15:45:52.578840 23351 leader_election.cc:302] T 0a51c0baaa6e432997c9d3552af4f254 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
W20260809 15:45:52.631942 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 55720.4us
I20260809 15:45:52.635646 24138 raft_consensus.cc:2726] T 0a51c0baaa6e432997c9d3552af4f254 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260809 15:45:52.638497 23351 leader_election.cc:302] T 297c82ed906d4b48b184985eed3de601 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c
I20260809 15:45:52.639727 24123 raft_consensus.cc:2781] T 297c82ed906d4b48b184985eed3de601 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:45:52.640658 23353 leader_election.cc:302] T 4564dfc6d4094c1b9240b367ccecf3b1 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:45:52.641119 23351 leader_election.cc:302] T 00a1881267a14ba8b78c81040f17a014 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c
I20260809 15:45:52.646899 23648 raft_consensus.cc:2781] T 4564dfc6d4094c1b9240b367ccecf3b1 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:45:52.656911 24022 raft_consensus.cc:482] T 28e17dcb5efe45b8b6a8c42da03b4819 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:45:52.659861 24022 raft_consensus.cc:504] T 28e17dcb5efe45b8b6a8c42da03b4819 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:52.662441 24022 leader_election.cc:288] T 28e17dcb5efe45b8b6a8c42da03b4819 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:45:52.645088 24133 consensus_queue.cc:1035] T d0721a4403a440f383318a3cea9d1442 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.013s
I20260809 15:45:52.654527 23648 raft_consensus.cc:686] T 4564dfc6d4094c1b9240b367ccecf3b1 P c0f38c1908414566aa9115bb2cbdd85d [term 1 LEADER]: Becoming Leader. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:45:52.668797 23648 consensus_queue.cc:237] T 4564dfc6d4094c1b9240b367ccecf3b1 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:52.674196 23397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e17dcb5efe45b8b6a8c42da03b4819" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:45:52.675050 23529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e17dcb5efe45b8b6a8c42da03b4819" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
W20260809 15:45:52.678606 23228 leader_election.cc:341] T 28e17dcb5efe45b8b6a8c42da03b4819 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:52.650857 23790 raft_consensus.cc:2781] T 00a1881267a14ba8b78c81040f17a014 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 15:45:52.676627 23117 catalog_manager.cc:5495] T 538a228ddfb54bc6a36528d5ff07597b P c0f38c1908414566aa9115bb2cbdd85d reported cstate change: term changed from 0 to 1, leader changed from <none> to c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66). New cstate: current_term: 1 leader_uuid: "c0f38c1908414566aa9115bb2cbdd85d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } health_report { overall_health: UNKNOWN } } }
W20260809 15:45:52.680833 23228 leader_election.cc:341] T 28e17dcb5efe45b8b6a8c42da03b4819 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:52.681223 23228 leader_election.cc:302] T 28e17dcb5efe45b8b6a8c42da03b4819 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:45:52.682740 23790 raft_consensus.cc:482] T 00a1881267a14ba8b78c81040f17a014 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:45:52.683393 24130 raft_consensus.cc:2726] T 28e17dcb5efe45b8b6a8c42da03b4819 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260809 15:45:52.686353 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 45296.3us
I20260809 15:45:52.694397 23274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4564dfc6d4094c1b9240b367ccecf3b1" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:45:52.695304 23274 raft_consensus.cc:3037] T 4564dfc6d4094c1b9240b367ccecf3b1 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:52.684556 23790 raft_consensus.cc:3037] T 00a1881267a14ba8b78c81040f17a014 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:52.719722 23591 raft_consensus.cc:348] T 213543cdbe814c89a17a963469922c1b P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:52.720407 23591 raft_consensus.cc:374] T 213543cdbe814c89a17a963469922c1b P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:52.720669 23591 raft_consensus.cc:729] T 213543cdbe814c89a17a963469922c1b P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Initialized, Role: FOLLOWER
I20260809 15:45:52.733600 24022 consensus_queue.cc:1035] T 563daf42d7784db195ea99c3cb21b370 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.016s
I20260809 15:45:52.745430 23591 consensus_queue.cc:260] T 213543cdbe814c89a17a963469922c1b P c0f38c1908414566aa9115bb2cbdd85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:52.755772 23593 tablet_bootstrap.cc:654] T 297c82ed906d4b48b184985eed3de601 P 65c36d164b0540f099d592f38307ee6c: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:52.752640 24123 raft_consensus.cc:686] T 297c82ed906d4b48b184985eed3de601 P c0f38c1908414566aa9115bb2cbdd85d [term 1 LEADER]: Becoming Leader. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:45:52.758543 24123 consensus_queue.cc:237] T 297c82ed906d4b48b184985eed3de601 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:45:52.751188 23472 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 50620.1us
I20260809 15:45:52.767521 23591 ts_tablet_manager.cc:1428] T 213543cdbe814c89a17a963469922c1b P c0f38c1908414566aa9115bb2cbdd85d: Time spent starting tablet: real 0.157s user 0.008s sys 0.008s
I20260809 15:45:52.776700 23594 tablet_bootstrap.cc:654] T cee02fff2c9c48a19f43eb358a89e7b0 P 67b3aeadfa7a432abe8af4d292e04ab9: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:52.787364 23396 raft_consensus.cc:1261] T 0a51c0baaa6e432997c9d3552af4f254 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Refusing update from remote peer 67b3aeadfa7a432abe8af4d292e04ab9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260809 15:45:52.793721 23274 raft_consensus.cc:2454] T 4564dfc6d4094c1b9240b367ccecf3b1 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 1.
I20260809 15:45:52.816056 23591 tablet_bootstrap.cc:492] T f9e7cadcd24d45fcbfa84209225e8f2d P c0f38c1908414566aa9115bb2cbdd85d: Bootstrap starting.
I20260809 15:45:52.818171 24142 consensus_queue.cc:1035] T 0a51c0baaa6e432997c9d3552af4f254 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
I20260809 15:45:52.817915 23790 raft_consensus.cc:504] T 00a1881267a14ba8b78c81040f17a014 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:52.832388 23790 leader_election.cc:288] T 00a1881267a14ba8b78c81040f17a014 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:45:52.833470 23521 raft_consensus.cc:1261] T 0a51c0baaa6e432997c9d3552af4f254 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Refusing update from remote peer 67b3aeadfa7a432abe8af4d292e04ab9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260809 15:45:52.836871 24142 consensus_queue.cc:1035] T 0a51c0baaa6e432997c9d3552af4f254 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:45:52.840267 23529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00a1881267a14ba8b78c81040f17a014" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:45:52.840832 23594 tablet_bootstrap.cc:492] T cee02fff2c9c48a19f43eb358a89e7b0 P 67b3aeadfa7a432abe8af4d292e04ab9: No bootstrap required, opened a new log
I20260809 15:45:52.841248 23594 ts_tablet_manager.cc:1397] T cee02fff2c9c48a19f43eb358a89e7b0 P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent bootstrapping tablet: real 0.262s user 0.013s sys 0.020s
I20260809 15:45:52.840451 24138 consensus_queue.cc:1035] T 538a228ddfb54bc6a36528d5ff07597b P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260809 15:45:52.842394 23353 leader_election.cc:341] T 00a1881267a14ba8b78c81040f17a014 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:52.844617 23594 raft_consensus.cc:348] T cee02fff2c9c48a19f43eb358a89e7b0 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:52.846580 23594 raft_consensus.cc:374] T cee02fff2c9c48a19f43eb358a89e7b0 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:52.846865 23594 raft_consensus.cc:729] T cee02fff2c9c48a19f43eb358a89e7b0 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Initialized, Role: FOLLOWER
I20260809 15:45:52.849080 23594 consensus_queue.cc:260] T cee02fff2c9c48a19f43eb358a89e7b0 P 67b3aeadfa7a432abe8af4d292e04ab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:52.862404 23594 ts_tablet_manager.cc:1428] T cee02fff2c9c48a19f43eb358a89e7b0 P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent starting tablet: real 0.021s user 0.006s sys 0.000s
I20260809 15:45:52.863229 23594 tablet_bootstrap.cc:492] T 213543cdbe814c89a17a963469922c1b P 67b3aeadfa7a432abe8af4d292e04ab9: Bootstrap starting.
I20260809 15:45:52.874305 23273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00a1881267a14ba8b78c81040f17a014" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:45:52.875033 23273 raft_consensus.cc:3037] T 00a1881267a14ba8b78c81040f17a014 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Advancing to term 1
W20260809 15:45:52.878474 24217 log.cc:924] Time spent T 563daf42d7784db195ea99c3cb21b370 P 65c36d164b0540f099d592f38307ee6c: Append to log took a long time: real 0.061s user 0.004s sys 0.000s
I20260809 15:45:52.879860 23648 raft_consensus.cc:482] T 213543cdbe814c89a17a963469922c1b P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:45:52.880311 23648 raft_consensus.cc:504] T 213543cdbe814c89a17a963469922c1b P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:52.883843 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "213543cdbe814c89a17a963469922c1b" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:45:52.884699 23527 raft_consensus.cc:2454] T 213543cdbe814c89a17a963469922c1b P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 0.
I20260809 15:45:52.886137 23353 leader_election.cc:302] T 213543cdbe814c89a17a963469922c1b P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:45:52.887197 23648 leader_election.cc:288] T 213543cdbe814c89a17a963469922c1b P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:45:52.887199 24211 raft_consensus.cc:2781] T 213543cdbe814c89a17a963469922c1b P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 15:45:52.887795 24211 raft_consensus.cc:482] T 213543cdbe814c89a17a963469922c1b P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:45:52.888156 24211 raft_consensus.cc:3037] T 213543cdbe814c89a17a963469922c1b P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:52.890573 23280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "213543cdbe814c89a17a963469922c1b" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:45:52.895207 23594 tablet_bootstrap.cc:654] T 213543cdbe814c89a17a963469922c1b P 67b3aeadfa7a432abe8af4d292e04ab9: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:52.909927 24212 consensus_queue.cc:1035] T 538a228ddfb54bc6a36528d5ff07597b P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
W20260809 15:45:52.913908 23351 leader_election.cc:341] T 213543cdbe814c89a17a963469922c1b P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:52.923879 23593 tablet_bootstrap.cc:492] T 297c82ed906d4b48b184985eed3de601 P 65c36d164b0540f099d592f38307ee6c: No bootstrap required, opened a new log
I20260809 15:45:52.924299 23593 ts_tablet_manager.cc:1397] T 297c82ed906d4b48b184985eed3de601 P 65c36d164b0540f099d592f38307ee6c: Time spent bootstrapping tablet: real 0.375s user 0.010s sys 0.011s
I20260809 15:45:52.917613 24211 raft_consensus.cc:504] T 213543cdbe814c89a17a963469922c1b P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:52.929478 24211 leader_election.cc:288] T 213543cdbe814c89a17a963469922c1b P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:45:52.932583 23521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "213543cdbe814c89a17a963469922c1b" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:45:52.933442 23521 raft_consensus.cc:3037] T 213543cdbe814c89a17a963469922c1b P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:52.938241 23591 tablet_bootstrap.cc:654] T f9e7cadcd24d45fcbfa84209225e8f2d P c0f38c1908414566aa9115bb2cbdd85d: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:52.942080 23593 raft_consensus.cc:348] T 297c82ed906d4b48b184985eed3de601 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:52.942731 23593 raft_consensus.cc:374] T 297c82ed906d4b48b184985eed3de601 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:52.943001 23593 raft_consensus.cc:729] T 297c82ed906d4b48b184985eed3de601 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Initialized, Role: FOLLOWER
I20260809 15:45:52.944473 23280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "213543cdbe814c89a17a963469922c1b" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:45:52.954787 23593 consensus_queue.cc:260] T 297c82ed906d4b48b184985eed3de601 P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:45:52.955705 23351 leader_election.cc:341] T 213543cdbe814c89a17a963469922c1b P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:52.956866 23593 ts_tablet_manager.cc:1428] T 297c82ed906d4b48b184985eed3de601 P 65c36d164b0540f099d592f38307ee6c: Time spent starting tablet: real 0.032s user 0.007s sys 0.000s
I20260809 15:45:52.957644 23593 tablet_bootstrap.cc:492] T 294d4bcc1e7a477cb56455581277a60f P 65c36d164b0540f099d592f38307ee6c: Bootstrap starting.
I20260809 15:45:52.960522 23116 catalog_manager.cc:5495] T 297c82ed906d4b48b184985eed3de601 P c0f38c1908414566aa9115bb2cbdd85d reported cstate change: term changed from 0 to 1, leader changed from <none> to c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66). New cstate: current_term: 1 leader_uuid: "c0f38c1908414566aa9115bb2cbdd85d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } health_report { overall_health: UNKNOWN } } }
I20260809 15:45:52.962198 23116 catalog_manager.cc:5495] T 4564dfc6d4094c1b9240b367ccecf3b1 P c0f38c1908414566aa9115bb2cbdd85d reported cstate change: term changed from 0 to 1, leader changed from <none> to c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66). New cstate: current_term: 1 leader_uuid: "c0f38c1908414566aa9115bb2cbdd85d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } health_report { overall_health: UNKNOWN } } }
I20260809 15:45:52.956751 23521 raft_consensus.cc:2454] T 213543cdbe814c89a17a963469922c1b P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 1.
I20260809 15:45:52.965657 23273 raft_consensus.cc:2454] T 00a1881267a14ba8b78c81040f17a014 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 1.
I20260809 15:45:52.966012 23353 leader_election.cc:302] T 213543cdbe814c89a17a963469922c1b P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters: 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:45:52.967070 23351 leader_election.cc:302] T 00a1881267a14ba8b78c81040f17a014 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c
I20260809 15:45:52.967753 24212 raft_consensus.cc:2781] T 213543cdbe814c89a17a963469922c1b P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:45:52.968380 24212 raft_consensus.cc:686] T 213543cdbe814c89a17a963469922c1b P c0f38c1908414566aa9115bb2cbdd85d [term 1 LEADER]: Becoming Leader. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:45:52.969835 24129 raft_consensus.cc:2781] T 00a1881267a14ba8b78c81040f17a014 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:45:52.977632 24129 raft_consensus.cc:686] T 00a1881267a14ba8b78c81040f17a014 P c0f38c1908414566aa9115bb2cbdd85d [term 1 LEADER]: Becoming Leader. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:45:52.978855 24129 consensus_queue.cc:237] T 00a1881267a14ba8b78c81040f17a014 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:52.970698 24212 consensus_queue.cc:237] T 213543cdbe814c89a17a963469922c1b P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:52.991050 24022 raft_consensus.cc:482] T d3e94b7e50754a81b3d72a07476cb7b9 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:45:52.991624 24022 raft_consensus.cc:504] T d3e94b7e50754a81b3d72a07476cb7b9 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:52.994014 24022 leader_election.cc:288] T d3e94b7e50754a81b3d72a07476cb7b9 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:45:52.994580 23593 tablet_bootstrap.cc:654] T 294d4bcc1e7a477cb56455581277a60f P 65c36d164b0540f099d592f38307ee6c: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:52.999423 23397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3e94b7e50754a81b3d72a07476cb7b9" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:45:53.010664 23521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3e94b7e50754a81b3d72a07476cb7b9" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
W20260809 15:45:53.013748 23228 leader_election.cc:341] T d3e94b7e50754a81b3d72a07476cb7b9 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263): Illegal state: must be running to vote when last-logged opid is not known
W20260809 15:45:53.014375 23228 leader_election.cc:341] T d3e94b7e50754a81b3d72a07476cb7b9 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:53.014717 23228 leader_election.cc:302] T d3e94b7e50754a81b3d72a07476cb7b9 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:45:53.015389 24130 raft_consensus.cc:2726] T d3e94b7e50754a81b3d72a07476cb7b9 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 15:45:53.020196 23594 tablet_bootstrap.cc:492] T 213543cdbe814c89a17a963469922c1b P 67b3aeadfa7a432abe8af4d292e04ab9: No bootstrap required, opened a new log
I20260809 15:45:53.020614 23594 ts_tablet_manager.cc:1397] T 213543cdbe814c89a17a963469922c1b P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent bootstrapping tablet: real 0.158s user 0.040s sys 0.021s
I20260809 15:45:53.024402 23591 tablet_bootstrap.cc:492] T f9e7cadcd24d45fcbfa84209225e8f2d P c0f38c1908414566aa9115bb2cbdd85d: No bootstrap required, opened a new log
I20260809 15:45:53.024880 23591 ts_tablet_manager.cc:1397] T f9e7cadcd24d45fcbfa84209225e8f2d P c0f38c1908414566aa9115bb2cbdd85d: Time spent bootstrapping tablet: real 0.209s user 0.026s sys 0.009s
I20260809 15:45:53.025978 23594 raft_consensus.cc:348] T 213543cdbe814c89a17a963469922c1b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:53.026785 23594 raft_consensus.cc:374] T 213543cdbe814c89a17a963469922c1b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:53.030532 24130 raft_consensus.cc:482] T cee02fff2c9c48a19f43eb358a89e7b0 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:45:53.030969 24130 raft_consensus.cc:504] T cee02fff2c9c48a19f43eb358a89e7b0 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:53.032963 23591 raft_consensus.cc:348] T f9e7cadcd24d45fcbfa84209225e8f2d P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:53.033648 23591 raft_consensus.cc:374] T f9e7cadcd24d45fcbfa84209225e8f2d P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:53.033895 23591 raft_consensus.cc:729] T f9e7cadcd24d45fcbfa84209225e8f2d P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Initialized, Role: FOLLOWER
I20260809 15:45:53.035141 23591 consensus_queue.cc:260] T f9e7cadcd24d45fcbfa84209225e8f2d P c0f38c1908414566aa9115bb2cbdd85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:53.036587 24130 leader_election.cc:288] T cee02fff2c9c48a19f43eb358a89e7b0 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:45:53.037142 23591 ts_tablet_manager.cc:1428] T f9e7cadcd24d45fcbfa84209225e8f2d P c0f38c1908414566aa9115bb2cbdd85d: Time spent starting tablet: real 0.012s user 0.006s sys 0.000s
I20260809 15:45:53.037966 23591 tablet_bootstrap.cc:492] T 07fcabb569414bddbd39df6cedd24e64 P c0f38c1908414566aa9115bb2cbdd85d: Bootstrap starting.
I20260809 15:45:53.038233 23594 raft_consensus.cc:729] T 213543cdbe814c89a17a963469922c1b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Initialized, Role: FOLLOWER
I20260809 15:45:53.038920 23594 consensus_queue.cc:260] T 213543cdbe814c89a17a963469922c1b P 67b3aeadfa7a432abe8af4d292e04ab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:53.039399 23521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee02fff2c9c48a19f43eb358a89e7b0" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:45:53.042713 23397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee02fff2c9c48a19f43eb358a89e7b0" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:45:53.043987 23594 ts_tablet_manager.cc:1428] T 213543cdbe814c89a17a963469922c1b P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent starting tablet: real 0.023s user 0.006s sys 0.000s
I20260809 15:45:53.044870 23594 tablet_bootstrap.cc:492] T d3be1e356b254d22b995be32f1c12f6f P 67b3aeadfa7a432abe8af4d292e04ab9: Bootstrap starting.
W20260809 15:45:53.041838 23228 leader_election.cc:341] T cee02fff2c9c48a19f43eb358a89e7b0 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553): Illegal state: must be running to vote when last-logged opid is not known
W20260809 15:45:53.046638 23228 leader_election.cc:341] T cee02fff2c9c48a19f43eb358a89e7b0 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:53.047883 24129 raft_consensus.cc:482] T f9e7cadcd24d45fcbfa84209225e8f2d P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:45:53.050793 23593 tablet_bootstrap.cc:492] T 294d4bcc1e7a477cb56455581277a60f P 65c36d164b0540f099d592f38307ee6c: No bootstrap required, opened a new log
I20260809 15:45:53.051188 23593 ts_tablet_manager.cc:1397] T 294d4bcc1e7a477cb56455581277a60f P 65c36d164b0540f099d592f38307ee6c: Time spent bootstrapping tablet: real 0.094s user 0.024s sys 0.023s
I20260809 15:45:53.051370 24129 raft_consensus.cc:504] T f9e7cadcd24d45fcbfa84209225e8f2d P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:53.051858 23591 tablet_bootstrap.cc:654] T 07fcabb569414bddbd39df6cedd24e64 P c0f38c1908414566aa9115bb2cbdd85d: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:53.053802 23273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9e7cadcd24d45fcbfa84209225e8f2d" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
W20260809 15:45:53.055037 23351 leader_election.cc:341] T f9e7cadcd24d45fcbfa84209225e8f2d P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:53.055769 23593 raft_consensus.cc:348] T 294d4bcc1e7a477cb56455581277a60f P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:53.056406 23593 raft_consensus.cc:374] T 294d4bcc1e7a477cb56455581277a60f P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:53.056669 23593 raft_consensus.cc:729] T 294d4bcc1e7a477cb56455581277a60f P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Initialized, Role: FOLLOWER
I20260809 15:45:53.047045 23228 leader_election.cc:302] T cee02fff2c9c48a19f43eb358a89e7b0 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:45:53.057246 23593 consensus_queue.cc:260] T 294d4bcc1e7a477cb56455581277a60f P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:53.058495 23521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9e7cadcd24d45fcbfa84209225e8f2d" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:45:53.059798 24130 raft_consensus.cc:2726] T cee02fff2c9c48a19f43eb358a89e7b0 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260809 15:45:53.060514 23353 leader_election.cc:341] T f9e7cadcd24d45fcbfa84209225e8f2d P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:53.060969 23353 leader_election.cc:302] T f9e7cadcd24d45fcbfa84209225e8f2d P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:45:53.061748 24212 raft_consensus.cc:2726] T f9e7cadcd24d45fcbfa84209225e8f2d P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 15:45:53.062644 24129 leader_election.cc:288] T f9e7cadcd24d45fcbfa84209225e8f2d P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:45:53.063632 23594 tablet_bootstrap.cc:654] T d3be1e356b254d22b995be32f1c12f6f P 67b3aeadfa7a432abe8af4d292e04ab9: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:53.066841 23116 catalog_manager.cc:5495] T 213543cdbe814c89a17a963469922c1b P c0f38c1908414566aa9115bb2cbdd85d reported cstate change: term changed from 0 to 1, leader changed from <none> to c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66). New cstate: current_term: 1 leader_uuid: "c0f38c1908414566aa9115bb2cbdd85d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } health_report { overall_health: UNKNOWN } } }
I20260809 15:45:53.069128 23594 tablet_bootstrap.cc:492] T d3be1e356b254d22b995be32f1c12f6f P 67b3aeadfa7a432abe8af4d292e04ab9: No bootstrap required, opened a new log
I20260809 15:45:53.068334 23116 catalog_manager.cc:5495] T 00a1881267a14ba8b78c81040f17a014 P c0f38c1908414566aa9115bb2cbdd85d reported cstate change: term changed from 0 to 1, leader changed from <none> to c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66). New cstate: current_term: 1 leader_uuid: "c0f38c1908414566aa9115bb2cbdd85d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } health_report { overall_health: UNKNOWN } } }
I20260809 15:45:53.069590 23594 ts_tablet_manager.cc:1397] T d3be1e356b254d22b995be32f1c12f6f P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent bootstrapping tablet: real 0.025s user 0.008s sys 0.000s
I20260809 15:45:53.072346 23594 raft_consensus.cc:348] T d3be1e356b254d22b995be32f1c12f6f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:53.072958 23594 raft_consensus.cc:374] T d3be1e356b254d22b995be32f1c12f6f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:53.073199 23594 raft_consensus.cc:729] T d3be1e356b254d22b995be32f1c12f6f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Initialized, Role: FOLLOWER
I20260809 15:45:53.073925 23594 consensus_queue.cc:260] T d3be1e356b254d22b995be32f1c12f6f P 67b3aeadfa7a432abe8af4d292e04ab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:53.075354 23593 ts_tablet_manager.cc:1428] T 294d4bcc1e7a477cb56455581277a60f P 65c36d164b0540f099d592f38307ee6c: Time spent starting tablet: real 0.024s user 0.005s sys 0.000s
I20260809 15:45:53.076306 23594 ts_tablet_manager.cc:1428] T d3be1e356b254d22b995be32f1c12f6f P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260809 15:45:53.076573 23593 tablet_bootstrap.cc:492] T df20671a7b874dbb98c0ef28371c1308 P 65c36d164b0540f099d592f38307ee6c: Bootstrap starting.
I20260809 15:45:53.077136 23594 tablet_bootstrap.cc:492] T 07fcabb569414bddbd39df6cedd24e64 P 67b3aeadfa7a432abe8af4d292e04ab9: Bootstrap starting.
I20260809 15:45:53.086560 23594 tablet_bootstrap.cc:654] T 07fcabb569414bddbd39df6cedd24e64 P 67b3aeadfa7a432abe8af4d292e04ab9: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:53.086879 23593 tablet_bootstrap.cc:654] T df20671a7b874dbb98c0ef28371c1308 P 65c36d164b0540f099d592f38307ee6c: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:53.088979 24160 raft_consensus.cc:482] T 297c82ed906d4b48b184985eed3de601 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:45:53.095293 24160 raft_consensus.cc:504] T 297c82ed906d4b48b184985eed3de601 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:53.099118 24160 leader_election.cc:288] T 297c82ed906d4b48b184985eed3de601 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:45:53.102252 23397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "297c82ed906d4b48b184985eed3de601" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:45:53.121253 23643 raft_consensus.cc:482] T 294d4bcc1e7a477cb56455581277a60f P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:45:53.122138 23591 tablet_bootstrap.cc:492] T 07fcabb569414bddbd39df6cedd24e64 P c0f38c1908414566aa9115bb2cbdd85d: No bootstrap required, opened a new log
I20260809 15:45:53.121846 23643 raft_consensus.cc:504] T 294d4bcc1e7a477cb56455581277a60f P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:53.122555 23591 ts_tablet_manager.cc:1397] T 07fcabb569414bddbd39df6cedd24e64 P c0f38c1908414566aa9115bb2cbdd85d: Time spent bootstrapping tablet: real 0.085s user 0.010s sys 0.005s
I20260809 15:45:53.125111 23643 leader_election.cc:288] T 294d4bcc1e7a477cb56455581277a60f P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:45:53.127794 23591 raft_consensus.cc:348] T 07fcabb569414bddbd39df6cedd24e64 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:53.128511 23591 raft_consensus.cc:374] T 07fcabb569414bddbd39df6cedd24e64 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:53.128754 23591 raft_consensus.cc:729] T 07fcabb569414bddbd39df6cedd24e64 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Initialized, Role: FOLLOWER
I20260809 15:45:53.129293 23591 consensus_queue.cc:260] T 07fcabb569414bddbd39df6cedd24e64 P c0f38c1908414566aa9115bb2cbdd85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:53.137693 23396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "294d4bcc1e7a477cb56455581277a60f" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:45:53.138437 23396 raft_consensus.cc:2454] T 294d4bcc1e7a477cb56455581277a60f P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 0.
I20260809 15:45:53.141247 23472 leader_election.cc:302] T 294d4bcc1e7a477cb56455581277a60f P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:45:53.142268 23691 raft_consensus.cc:2781] T 294d4bcc1e7a477cb56455581277a60f P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 15:45:53.142603 23691 raft_consensus.cc:482] T 294d4bcc1e7a477cb56455581277a60f P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:45:53.142985 23691 raft_consensus.cc:3037] T 294d4bcc1e7a477cb56455581277a60f P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:53.143644 23274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "297c82ed906d4b48b184985eed3de601" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:45:53.144699 23274 raft_consensus.cc:2379] T 297c82ed906d4b48b184985eed3de601 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c in current term 1: Already voted for candidate c0f38c1908414566aa9115bb2cbdd85d in this term.
I20260809 15:45:53.146384 23470 leader_election.cc:302] T 297c82ed906d4b48b184985eed3de601 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:45:53.149941 23643 raft_consensus.cc:3037] T 297c82ed906d4b48b184985eed3de601 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:53.151808 23591 ts_tablet_manager.cc:1428] T 07fcabb569414bddbd39df6cedd24e64 P c0f38c1908414566aa9115bb2cbdd85d: Time spent starting tablet: real 0.029s user 0.004s sys 0.004s
I20260809 15:45:53.150810 23273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "294d4bcc1e7a477cb56455581277a60f" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:45:53.152664 23591 tablet_bootstrap.cc:492] T 4b30f28311964a0b9149a50495e83d8f P c0f38c1908414566aa9115bb2cbdd85d: Bootstrap starting.
I20260809 15:45:53.168020 23593 tablet_bootstrap.cc:492] T df20671a7b874dbb98c0ef28371c1308 P 65c36d164b0540f099d592f38307ee6c: No bootstrap required, opened a new log
I20260809 15:45:53.168485 23593 ts_tablet_manager.cc:1397] T df20671a7b874dbb98c0ef28371c1308 P 65c36d164b0540f099d592f38307ee6c: Time spent bootstrapping tablet: real 0.092s user 0.014s sys 0.020s
I20260809 15:45:53.175877 23691 raft_consensus.cc:504] T 294d4bcc1e7a477cb56455581277a60f P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:53.182440 23691 leader_election.cc:288] T 294d4bcc1e7a477cb56455581277a60f P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 1 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:45:53.182642 23643 raft_consensus.cc:2726] T 297c82ed906d4b48b184985eed3de601 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 15:45:53.184952 23397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "294d4bcc1e7a477cb56455581277a60f" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:45:53.185715 23397 raft_consensus.cc:3037] T 294d4bcc1e7a477cb56455581277a60f P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:53.189359 24212 consensus_queue.cc:1035] T 297c82ed906d4b48b184985eed3de601 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:45:53.190179 24130 raft_consensus.cc:482] T d3be1e356b254d22b995be32f1c12f6f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:45:53.190757 24130 raft_consensus.cc:504] T d3be1e356b254d22b995be32f1c12f6f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:53.193403 24130 leader_election.cc:288] T d3be1e356b254d22b995be32f1c12f6f P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
W20260809 15:45:53.196810 23470 leader_election.cc:341] T 294d4bcc1e7a477cb56455581277a60f P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:53.195874 23396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3be1e356b254d22b995be32f1c12f6f" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:45:53.200213 24129 consensus_queue.cc:1035] T 297c82ed906d4b48b184985eed3de601 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:45:53.196671 23593 raft_consensus.cc:348] T df20671a7b874dbb98c0ef28371c1308 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:53.201906 23593 raft_consensus.cc:374] T df20671a7b874dbb98c0ef28371c1308 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:53.202158 23593 raft_consensus.cc:729] T df20671a7b874dbb98c0ef28371c1308 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Initialized, Role: FOLLOWER
I20260809 15:45:53.195065 23397 raft_consensus.cc:2454] T 294d4bcc1e7a477cb56455581277a60f P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 1.
I20260809 15:45:53.202800 24142 raft_consensus.cc:482] T 213543cdbe814c89a17a963469922c1b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:45:53.203527 24142 raft_consensus.cc:504] T 213543cdbe814c89a17a963469922c1b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:45:53.205830 23228 leader_election.cc:341] T d3be1e356b254d22b995be32f1c12f6f P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:53.206557 24142 leader_election.cc:288] T 213543cdbe814c89a17a963469922c1b P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:45:53.203507 23593 consensus_queue.cc:260] T df20671a7b874dbb98c0ef28371c1308 P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:53.208230 23521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "213543cdbe814c89a17a963469922c1b" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:45:53.209200 23521 raft_consensus.cc:2379] T 213543cdbe814c89a17a963469922c1b P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 in current term 1: Already voted for candidate c0f38c1908414566aa9115bb2cbdd85d in this term.
I20260809 15:45:53.209331 23594 tablet_bootstrap.cc:492] T 07fcabb569414bddbd39df6cedd24e64 P 67b3aeadfa7a432abe8af4d292e04ab9: No bootstrap required, opened a new log
I20260809 15:45:53.209812 23594 ts_tablet_manager.cc:1397] T 07fcabb569414bddbd39df6cedd24e64 P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent bootstrapping tablet: real 0.133s user 0.018s sys 0.007s
I20260809 15:45:53.210229 23593 ts_tablet_manager.cc:1428] T df20671a7b874dbb98c0ef28371c1308 P 65c36d164b0540f099d592f38307ee6c: Time spent starting tablet: real 0.041s user 0.006s sys 0.000s
I20260809 15:45:53.211097 23593 tablet_bootstrap.cc:492] T f9e7cadcd24d45fcbfa84209225e8f2d P 65c36d164b0540f099d592f38307ee6c: Bootstrap starting.
I20260809 15:45:53.194471 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3be1e356b254d22b995be32f1c12f6f" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:45:53.212957 23594 raft_consensus.cc:348] T 07fcabb569414bddbd39df6cedd24e64 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:53.213601 23594 raft_consensus.cc:374] T 07fcabb569414bddbd39df6cedd24e64 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:53.213850 23594 raft_consensus.cc:729] T 07fcabb569414bddbd39df6cedd24e64 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Initialized, Role: FOLLOWER
W20260809 15:45:53.214092 23228 leader_election.cc:341] T d3be1e356b254d22b995be32f1c12f6f P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:53.217262 23228 leader_election.cc:302] T d3be1e356b254d22b995be32f1c12f6f P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:45:53.218058 24130 raft_consensus.cc:2726] T d3be1e356b254d22b995be32f1c12f6f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 15:45:53.219026 23593 tablet_bootstrap.cc:654] T f9e7cadcd24d45fcbfa84209225e8f2d P 65c36d164b0540f099d592f38307ee6c: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:53.215052 23472 leader_election.cc:302] T 294d4bcc1e7a477cb56455581277a60f P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:45:53.222106 23591 tablet_bootstrap.cc:654] T 4b30f28311964a0b9149a50495e83d8f P c0f38c1908414566aa9115bb2cbdd85d: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:53.215108 23397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "213543cdbe814c89a17a963469922c1b" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:45:53.222538 24160 raft_consensus.cc:2781] T 294d4bcc1e7a477cb56455581277a60f P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:45:53.223345 24160 raft_consensus.cc:686] T 294d4bcc1e7a477cb56455581277a60f P 65c36d164b0540f099d592f38307ee6c [term 1 LEADER]: Becoming Leader. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:45:53.224364 24160 consensus_queue.cc:237] T 294d4bcc1e7a477cb56455581277a60f P 65c36d164b0540f099d592f38307ee6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:45:53.227145 23193 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260809 15:45:53.231786 23228 leader_election.cc:302] T 213543cdbe814c89a17a963469922c1b P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:45:53.232553 24022 raft_consensus.cc:3037] T 213543cdbe814c89a17a963469922c1b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:53.214404 23594 consensus_queue.cc:260] T 07fcabb569414bddbd39df6cedd24e64 P 67b3aeadfa7a432abe8af4d292e04ab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:53.240994 23594 ts_tablet_manager.cc:1428] T 07fcabb569414bddbd39df6cedd24e64 P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent starting tablet: real 0.031s user 0.007s sys 0.000s
I20260809 15:45:53.241891 23594 tablet_bootstrap.cc:492] T f9e7cadcd24d45fcbfa84209225e8f2d P 67b3aeadfa7a432abe8af4d292e04ab9: Bootstrap starting.
I20260809 15:45:53.249332 23282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "294d4bcc1e7a477cb56455581277a60f" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
W20260809 15:45:53.269152 23470 leader_election.cc:341] T 294d4bcc1e7a477cb56455581277a60f P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:53.266263 24022 raft_consensus.cc:2726] T 213543cdbe814c89a17a963469922c1b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 15:45:53.281669 24212 consensus_queue.cc:1035] T 4564dfc6d4094c1b9240b367ccecf3b1 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260809 15:45:53.284806 23648 raft_consensus.cc:482] T 07fcabb569414bddbd39df6cedd24e64 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:45:53.285199 23648 raft_consensus.cc:504] T 07fcabb569414bddbd39df6cedd24e64 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:53.286868 23648 leader_election.cc:288] T 07fcabb569414bddbd39df6cedd24e64 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:45:53.288213 23591 tablet_bootstrap.cc:492] T 4b30f28311964a0b9149a50495e83d8f P c0f38c1908414566aa9115bb2cbdd85d: No bootstrap required, opened a new log
I20260809 15:45:53.288645 23591 ts_tablet_manager.cc:1397] T 4b30f28311964a0b9149a50495e83d8f P c0f38c1908414566aa9115bb2cbdd85d: Time spent bootstrapping tablet: real 0.136s user 0.014s sys 0.007s
I20260809 15:45:53.291371 23282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07fcabb569414bddbd39df6cedd24e64" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:45:53.291960 23282 raft_consensus.cc:2454] T 07fcabb569414bddbd39df6cedd24e64 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 0.
I20260809 15:45:53.292336 23593 tablet_bootstrap.cc:492] T f9e7cadcd24d45fcbfa84209225e8f2d P 65c36d164b0540f099d592f38307ee6c: No bootstrap required, opened a new log
I20260809 15:45:53.292753 23593 ts_tablet_manager.cc:1397] T f9e7cadcd24d45fcbfa84209225e8f2d P 65c36d164b0540f099d592f38307ee6c: Time spent bootstrapping tablet: real 0.082s user 0.009s sys 0.012s
I20260809 15:45:53.293046 23521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07fcabb569414bddbd39df6cedd24e64" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
W20260809 15:45:53.294813 23353 leader_election.cc:341] T 07fcabb569414bddbd39df6cedd24e64 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:53.297022 23591 raft_consensus.cc:348] T 4b30f28311964a0b9149a50495e83d8f P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:53.297680 23591 raft_consensus.cc:374] T 4b30f28311964a0b9149a50495e83d8f P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:53.297948 23591 raft_consensus.cc:729] T 4b30f28311964a0b9149a50495e83d8f P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Initialized, Role: FOLLOWER
I20260809 15:45:53.298970 23591 consensus_queue.cc:260] T 4b30f28311964a0b9149a50495e83d8f P c0f38c1908414566aa9115bb2cbdd85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:53.300244 23593 raft_consensus.cc:348] T f9e7cadcd24d45fcbfa84209225e8f2d P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:53.300899 23593 raft_consensus.cc:374] T f9e7cadcd24d45fcbfa84209225e8f2d P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:53.301155 23593 raft_consensus.cc:729] T f9e7cadcd24d45fcbfa84209225e8f2d P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Initialized, Role: FOLLOWER
I20260809 15:45:53.301426 23351 leader_election.cc:302] T 07fcabb569414bddbd39df6cedd24e64 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c
I20260809 15:45:53.302857 23593 consensus_queue.cc:260] T f9e7cadcd24d45fcbfa84209225e8f2d P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:53.304884 23593 ts_tablet_manager.cc:1428] T f9e7cadcd24d45fcbfa84209225e8f2d P 65c36d164b0540f099d592f38307ee6c: Time spent starting tablet: real 0.012s user 0.007s sys 0.000s
I20260809 15:45:53.305646 23593 tablet_bootstrap.cc:492] T 3924b648461e463b9d7925e5671d68ae P 65c36d164b0540f099d592f38307ee6c: Bootstrap starting.
I20260809 15:45:53.306306 23591 ts_tablet_manager.cc:1428] T 4b30f28311964a0b9149a50495e83d8f P c0f38c1908414566aa9115bb2cbdd85d: Time spent starting tablet: real 0.017s user 0.007s sys 0.000s
I20260809 15:45:53.308429 23594 tablet_bootstrap.cc:654] T f9e7cadcd24d45fcbfa84209225e8f2d P 67b3aeadfa7a432abe8af4d292e04ab9: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:53.309317 23591 tablet_bootstrap.cc:492] T 28e17dcb5efe45b8b6a8c42da03b4819 P c0f38c1908414566aa9115bb2cbdd85d: Bootstrap starting.
I20260809 15:45:53.312660 23593 tablet_bootstrap.cc:654] T 3924b648461e463b9d7925e5671d68ae P 65c36d164b0540f099d592f38307ee6c: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:53.317169 24123 raft_consensus.cc:2781] T 07fcabb569414bddbd39df6cedd24e64 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 15:45:53.317569 24123 raft_consensus.cc:482] T 07fcabb569414bddbd39df6cedd24e64 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:45:53.317889 24123 raft_consensus.cc:3037] T 07fcabb569414bddbd39df6cedd24e64 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:53.318547 24211 consensus_queue.cc:1035] T 4564dfc6d4094c1b9240b367ccecf3b1 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:45:53.322553 23594 tablet_bootstrap.cc:492] T f9e7cadcd24d45fcbfa84209225e8f2d P 67b3aeadfa7a432abe8af4d292e04ab9: No bootstrap required, opened a new log
I20260809 15:45:53.322677 23591 tablet_bootstrap.cc:654] T 28e17dcb5efe45b8b6a8c42da03b4819 P c0f38c1908414566aa9115bb2cbdd85d: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:53.323086 23594 ts_tablet_manager.cc:1397] T f9e7cadcd24d45fcbfa84209225e8f2d P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent bootstrapping tablet: real 0.081s user 0.016s sys 0.001s
I20260809 15:45:53.324470 24123 raft_consensus.cc:504] T 07fcabb569414bddbd39df6cedd24e64 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:53.325552 23594 raft_consensus.cc:348] T f9e7cadcd24d45fcbfa84209225e8f2d P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:53.326292 23594 raft_consensus.cc:374] T f9e7cadcd24d45fcbfa84209225e8f2d P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:53.326601 23594 raft_consensus.cc:729] T f9e7cadcd24d45fcbfa84209225e8f2d P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Initialized, Role: FOLLOWER
I20260809 15:45:53.327170 24123 leader_election.cc:288] T 07fcabb569414bddbd39df6cedd24e64 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:45:53.327219 23594 consensus_queue.cc:260] T f9e7cadcd24d45fcbfa84209225e8f2d P 67b3aeadfa7a432abe8af4d292e04ab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:53.328647 23643 raft_consensus.cc:482] T f9e7cadcd24d45fcbfa84209225e8f2d P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:45:53.329046 23643 raft_consensus.cc:504] T f9e7cadcd24d45fcbfa84209225e8f2d P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:53.329253 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07fcabb569414bddbd39df6cedd24e64" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:45:53.329661 23594 ts_tablet_manager.cc:1428] T f9e7cadcd24d45fcbfa84209225e8f2d P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20260809 15:45:53.330674 23643 leader_election.cc:288] T f9e7cadcd24d45fcbfa84209225e8f2d P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:45:53.331248 23594 tablet_bootstrap.cc:492] T 4b30f28311964a0b9149a50495e83d8f P 67b3aeadfa7a432abe8af4d292e04ab9: Bootstrap starting.
I20260809 15:45:53.331411 23593 tablet_bootstrap.cc:492] T 3924b648461e463b9d7925e5671d68ae P 65c36d164b0540f099d592f38307ee6c: No bootstrap required, opened a new log
I20260809 15:45:53.331836 23593 ts_tablet_manager.cc:1397] T 3924b648461e463b9d7925e5671d68ae P 65c36d164b0540f099d592f38307ee6c: Time spent bootstrapping tablet: real 0.026s user 0.007s sys 0.007s
I20260809 15:45:53.332262 23397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9e7cadcd24d45fcbfa84209225e8f2d" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:45:53.332868 23397 raft_consensus.cc:2454] T f9e7cadcd24d45fcbfa84209225e8f2d P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 0.
I20260809 15:45:53.333937 23593 raft_consensus.cc:348] T 3924b648461e463b9d7925e5671d68ae P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:53.334578 23593 raft_consensus.cc:374] T 3924b648461e463b9d7925e5671d68ae P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:53.334824 23593 raft_consensus.cc:729] T 3924b648461e463b9d7925e5671d68ae P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Initialized, Role: FOLLOWER
I20260809 15:45:53.335382 23472 leader_election.cc:302] T f9e7cadcd24d45fcbfa84209225e8f2d P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:45:53.336156 23933 raft_consensus.cc:2781] T f9e7cadcd24d45fcbfa84209225e8f2d P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 15:45:53.335916 23593 consensus_queue.cc:260] T 3924b648461e463b9d7925e5671d68ae P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:53.336511 23933 raft_consensus.cc:482] T f9e7cadcd24d45fcbfa84209225e8f2d P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:45:53.336843 23933 raft_consensus.cc:3037] T f9e7cadcd24d45fcbfa84209225e8f2d P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:53.338291 23593 ts_tablet_manager.cc:1428] T 3924b648461e463b9d7925e5671d68ae P 65c36d164b0540f099d592f38307ee6c: Time spent starting tablet: real 0.006s user 0.002s sys 0.003s
W20260809 15:45:53.338848 23353 leader_election.cc:341] T 07fcabb569414bddbd39df6cedd24e64 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:53.339008 23593 tablet_bootstrap.cc:492] T 00a1881267a14ba8b78c81040f17a014 P 65c36d164b0540f099d592f38307ee6c: Bootstrap starting.
I20260809 15:45:53.343077 23933 raft_consensus.cc:504] T f9e7cadcd24d45fcbfa84209225e8f2d P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:53.344869 23591 tablet_bootstrap.cc:492] T 28e17dcb5efe45b8b6a8c42da03b4819 P c0f38c1908414566aa9115bb2cbdd85d: No bootstrap required, opened a new log
I20260809 15:45:53.345264 23591 ts_tablet_manager.cc:1397] T 28e17dcb5efe45b8b6a8c42da03b4819 P c0f38c1908414566aa9115bb2cbdd85d: Time spent bootstrapping tablet: real 0.036s user 0.004s sys 0.007s
I20260809 15:45:53.346228 23933 leader_election.cc:288] T f9e7cadcd24d45fcbfa84209225e8f2d P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 1 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:45:53.347477 23591 raft_consensus.cc:348] T 28e17dcb5efe45b8b6a8c42da03b4819 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:53.348114 23591 raft_consensus.cc:374] T 28e17dcb5efe45b8b6a8c42da03b4819 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:53.348363 23591 raft_consensus.cc:729] T 28e17dcb5efe45b8b6a8c42da03b4819 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Initialized, Role: FOLLOWER
I20260809 15:45:53.349026 23594 tablet_bootstrap.cc:654] T 4b30f28311964a0b9149a50495e83d8f P 67b3aeadfa7a432abe8af4d292e04ab9: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:53.350589 23282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07fcabb569414bddbd39df6cedd24e64" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:45:53.351143 23282 raft_consensus.cc:3037] T 07fcabb569414bddbd39df6cedd24e64 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:53.352295 23593 tablet_bootstrap.cc:654] T 00a1881267a14ba8b78c81040f17a014 P 65c36d164b0540f099d592f38307ee6c: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:53.355502 23591 consensus_queue.cc:260] T 28e17dcb5efe45b8b6a8c42da03b4819 P c0f38c1908414566aa9115bb2cbdd85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:53.356850 23280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9e7cadcd24d45fcbfa84209225e8f2d" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:45:53.357501 23280 raft_consensus.cc:2454] T f9e7cadcd24d45fcbfa84209225e8f2d P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 0.
I20260809 15:45:53.358115 23591 ts_tablet_manager.cc:1428] T 28e17dcb5efe45b8b6a8c42da03b4819 P c0f38c1908414566aa9115bb2cbdd85d: Time spent starting tablet: real 0.013s user 0.006s sys 0.000s
I20260809 15:45:53.358392 23273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9e7cadcd24d45fcbfa84209225e8f2d" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:45:53.358930 23591 tablet_bootstrap.cc:492] T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d: Bootstrap starting.
I20260809 15:45:53.358945 23273 raft_consensus.cc:3037] T f9e7cadcd24d45fcbfa84209225e8f2d P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:53.362319 23404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9e7cadcd24d45fcbfa84209225e8f2d" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:45:53.362864 23404 raft_consensus.cc:3037] T f9e7cadcd24d45fcbfa84209225e8f2d P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:53.364157 23591 tablet_bootstrap.cc:654] T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:53.366307 23273 raft_consensus.cc:2454] T f9e7cadcd24d45fcbfa84209225e8f2d P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 1.
I20260809 15:45:53.370343 23282 raft_consensus.cc:2454] T 07fcabb569414bddbd39df6cedd24e64 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 1.
I20260809 15:45:53.375344 23404 raft_consensus.cc:2454] T f9e7cadcd24d45fcbfa84209225e8f2d P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 1.
I20260809 15:45:53.379724 23472 leader_election.cc:302] T f9e7cadcd24d45fcbfa84209225e8f2d P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:45:53.381453 23351 leader_election.cc:302] T 07fcabb569414bddbd39df6cedd24e64 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c
I20260809 15:45:53.382542 24123 raft_consensus.cc:2781] T 07fcabb569414bddbd39df6cedd24e64 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:45:53.380532 23933 raft_consensus.cc:2781] T f9e7cadcd24d45fcbfa84209225e8f2d P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:45:53.383178 24123 raft_consensus.cc:686] T 07fcabb569414bddbd39df6cedd24e64 P c0f38c1908414566aa9115bb2cbdd85d [term 1 LEADER]: Becoming Leader. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:45:53.383601 23933 raft_consensus.cc:686] T f9e7cadcd24d45fcbfa84209225e8f2d P 65c36d164b0540f099d592f38307ee6c [term 1 LEADER]: Becoming Leader. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:45:53.384038 24123 consensus_queue.cc:237] T 07fcabb569414bddbd39df6cedd24e64 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:53.384435 23933 consensus_queue.cc:237] T f9e7cadcd24d45fcbfa84209225e8f2d P 65c36d164b0540f099d592f38307ee6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:53.407793 23593 tablet_bootstrap.cc:492] T 00a1881267a14ba8b78c81040f17a014 P 65c36d164b0540f099d592f38307ee6c: No bootstrap required, opened a new log
I20260809 15:45:53.408195 23593 ts_tablet_manager.cc:1397] T 00a1881267a14ba8b78c81040f17a014 P 65c36d164b0540f099d592f38307ee6c: Time spent bootstrapping tablet: real 0.069s user 0.015s sys 0.011s
I20260809 15:45:53.436524 23591 tablet_bootstrap.cc:492] T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d: No bootstrap required, opened a new log
I20260809 15:45:53.436947 23591 ts_tablet_manager.cc:1397] T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d: Time spent bootstrapping tablet: real 0.078s user 0.017s sys 0.025s
I20260809 15:45:53.438738 23593 raft_consensus.cc:348] T 00a1881267a14ba8b78c81040f17a014 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:53.439447 23593 raft_consensus.cc:374] T 00a1881267a14ba8b78c81040f17a014 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:53.439186 23591 raft_consensus.cc:348] T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:53.439692 23593 raft_consensus.cc:729] T 00a1881267a14ba8b78c81040f17a014 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Initialized, Role: FOLLOWER
I20260809 15:45:53.439853 23591 raft_consensus.cc:374] T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:53.440100 23591 raft_consensus.cc:729] T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Initialized, Role: FOLLOWER
I20260809 15:45:53.440258 23593 consensus_queue.cc:260] T 00a1881267a14ba8b78c81040f17a014 P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:53.440728 23591 consensus_queue.cc:260] T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:53.442282 23593 ts_tablet_manager.cc:1428] T 00a1881267a14ba8b78c81040f17a014 P 65c36d164b0540f099d592f38307ee6c: Time spent starting tablet: real 0.034s user 0.004s sys 0.000s
I20260809 15:45:53.442849 23591 ts_tablet_manager.cc:1428] T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260809 15:45:53.443112 23593 tablet_bootstrap.cc:492] T cee02fff2c9c48a19f43eb358a89e7b0 P 65c36d164b0540f099d592f38307ee6c: Bootstrap starting.
I20260809 15:45:53.443657 23591 tablet_bootstrap.cc:492] T cee02fff2c9c48a19f43eb358a89e7b0 P c0f38c1908414566aa9115bb2cbdd85d: Bootstrap starting.
I20260809 15:45:53.452492 23790 raft_consensus.cc:482] T 28e17dcb5efe45b8b6a8c42da03b4819 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:45:53.452926 23790 raft_consensus.cc:504] T 28e17dcb5efe45b8b6a8c42da03b4819 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:53.454757 23790 leader_election.cc:288] T 28e17dcb5efe45b8b6a8c42da03b4819 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:45:53.456934 23283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e17dcb5efe45b8b6a8c42da03b4819" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:45:53.457525 23283 raft_consensus.cc:2454] T 28e17dcb5efe45b8b6a8c42da03b4819 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 0.
I20260809 15:45:53.458630 23351 leader_election.cc:302] T 28e17dcb5efe45b8b6a8c42da03b4819 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:45:53.459107 23529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e17dcb5efe45b8b6a8c42da03b4819" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:45:53.459990 24212 raft_consensus.cc:2781] T 28e17dcb5efe45b8b6a8c42da03b4819 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 15:45:53.460331 24212 raft_consensus.cc:482] T 28e17dcb5efe45b8b6a8c42da03b4819 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:45:53.460652 24212 raft_consensus.cc:3037] T 28e17dcb5efe45b8b6a8c42da03b4819 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Advancing to term 1
W20260809 15:45:53.461337 23353 leader_election.cc:341] T 28e17dcb5efe45b8b6a8c42da03b4819 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:53.468789 23593 tablet_bootstrap.cc:654] T cee02fff2c9c48a19f43eb358a89e7b0 P 65c36d164b0540f099d592f38307ee6c: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:53.470124 23790 consensus_queue.cc:1035] T 213543cdbe814c89a17a963469922c1b P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:45:53.472599 23594 tablet_bootstrap.cc:492] T 4b30f28311964a0b9149a50495e83d8f P 67b3aeadfa7a432abe8af4d292e04ab9: No bootstrap required, opened a new log
I20260809 15:45:53.473022 23594 ts_tablet_manager.cc:1397] T 4b30f28311964a0b9149a50495e83d8f P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent bootstrapping tablet: real 0.143s user 0.015s sys 0.016s
I20260809 15:45:53.469492 23116 catalog_manager.cc:5495] T 294d4bcc1e7a477cb56455581277a60f P 65c36d164b0540f099d592f38307ee6c reported cstate change: term changed from 0 to 1, leader changed from <none> to 65c36d164b0540f099d592f38307ee6c (127.20.202.67). New cstate: current_term: 1 leader_uuid: "65c36d164b0540f099d592f38307ee6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } health_report { overall_health: HEALTHY } } }
I20260809 15:45:53.476464 24212 raft_consensus.cc:504] T 28e17dcb5efe45b8b6a8c42da03b4819 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:53.477387 23594 raft_consensus.cc:348] T 4b30f28311964a0b9149a50495e83d8f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:53.478056 23594 raft_consensus.cc:374] T 4b30f28311964a0b9149a50495e83d8f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:53.478303 23594 raft_consensus.cc:729] T 4b30f28311964a0b9149a50495e83d8f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Initialized, Role: FOLLOWER
I20260809 15:45:53.478868 23594 consensus_queue.cc:260] T 4b30f28311964a0b9149a50495e83d8f P 67b3aeadfa7a432abe8af4d292e04ab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:53.479874 23274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e17dcb5efe45b8b6a8c42da03b4819" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:45:53.480475 23274 raft_consensus.cc:3037] T 28e17dcb5efe45b8b6a8c42da03b4819 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:53.482687 23594 ts_tablet_manager.cc:1428] T 4b30f28311964a0b9149a50495e83d8f P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent starting tablet: real 0.009s user 0.006s sys 0.001s
I20260809 15:45:53.483579 23117 catalog_manager.cc:5495] T 07fcabb569414bddbd39df6cedd24e64 P c0f38c1908414566aa9115bb2cbdd85d reported cstate change: term changed from 0 to 1, leader changed from <none> to c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66). New cstate: current_term: 1 leader_uuid: "c0f38c1908414566aa9115bb2cbdd85d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } health_report { overall_health: UNKNOWN } } }
I20260809 15:45:53.486289 24123 consensus_queue.cc:1035] T 213543cdbe814c89a17a963469922c1b P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:45:53.488652 23594 tablet_bootstrap.cc:492] T 3924b648461e463b9d7925e5671d68ae P 67b3aeadfa7a432abe8af4d292e04ab9: Bootstrap starting.
I20260809 15:45:53.492738 23274 raft_consensus.cc:2454] T 28e17dcb5efe45b8b6a8c42da03b4819 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 1.
I20260809 15:45:53.494433 23351 leader_election.cc:302] T 28e17dcb5efe45b8b6a8c42da03b4819 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:45:53.495213 23790 raft_consensus.cc:2781] T 28e17dcb5efe45b8b6a8c42da03b4819 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:45:53.495882 23790 raft_consensus.cc:686] T 28e17dcb5efe45b8b6a8c42da03b4819 P c0f38c1908414566aa9115bb2cbdd85d [term 1 LEADER]: Becoming Leader. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:45:53.496888 23790 consensus_queue.cc:237] T 28e17dcb5efe45b8b6a8c42da03b4819 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:53.505846 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e17dcb5efe45b8b6a8c42da03b4819" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:45:53.499676 24212 leader_election.cc:288] T 28e17dcb5efe45b8b6a8c42da03b4819 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
W20260809 15:45:53.509855 23353 leader_election.cc:341] T 28e17dcb5efe45b8b6a8c42da03b4819 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:53.512282 23591 tablet_bootstrap.cc:654] T cee02fff2c9c48a19f43eb358a89e7b0 P c0f38c1908414566aa9115bb2cbdd85d: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:53.512646 23593 tablet_bootstrap.cc:492] T cee02fff2c9c48a19f43eb358a89e7b0 P 65c36d164b0540f099d592f38307ee6c: No bootstrap required, opened a new log
I20260809 15:45:53.513105 23593 ts_tablet_manager.cc:1397] T cee02fff2c9c48a19f43eb358a89e7b0 P 65c36d164b0540f099d592f38307ee6c: Time spent bootstrapping tablet: real 0.070s user 0.012s sys 0.007s
I20260809 15:45:53.515218 23526 raft_consensus.cc:3037] T 00a1881267a14ba8b78c81040f17a014 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:53.523725 23594 tablet_bootstrap.cc:654] T 3924b648461e463b9d7925e5671d68ae P 67b3aeadfa7a432abe8af4d292e04ab9: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:53.542069 23593 raft_consensus.cc:348] T cee02fff2c9c48a19f43eb358a89e7b0 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:53.542726 23593 raft_consensus.cc:374] T cee02fff2c9c48a19f43eb358a89e7b0 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:53.542976 23593 raft_consensus.cc:729] T cee02fff2c9c48a19f43eb358a89e7b0 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Initialized, Role: FOLLOWER
I20260809 15:45:53.543591 23593 consensus_queue.cc:260] T cee02fff2c9c48a19f43eb358a89e7b0 P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:53.543251 23117 catalog_manager.cc:5495] T f9e7cadcd24d45fcbfa84209225e8f2d P 65c36d164b0540f099d592f38307ee6c reported cstate change: term changed from 0 to 1, leader changed from <none> to 65c36d164b0540f099d592f38307ee6c (127.20.202.67). New cstate: current_term: 1 leader_uuid: "65c36d164b0540f099d592f38307ee6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } health_report { overall_health: HEALTHY } } }
I20260809 15:45:53.546512 23593 ts_tablet_manager.cc:1428] T cee02fff2c9c48a19f43eb358a89e7b0 P 65c36d164b0540f099d592f38307ee6c: Time spent starting tablet: real 0.033s user 0.007s sys 0.000s
I20260809 15:45:53.547346 23593 tablet_bootstrap.cc:492] T 07fcabb569414bddbd39df6cedd24e64 P 65c36d164b0540f099d592f38307ee6c: Bootstrap starting.
I20260809 15:45:53.551580 24211 consensus_queue.cc:1035] T 00a1881267a14ba8b78c81040f17a014 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:45:53.555732 24211 consensus_queue.cc:1035] T 00a1881267a14ba8b78c81040f17a014 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:45:53.562491 23591 tablet_bootstrap.cc:492] T cee02fff2c9c48a19f43eb358a89e7b0 P c0f38c1908414566aa9115bb2cbdd85d: No bootstrap required, opened a new log
I20260809 15:45:53.563319 23591 ts_tablet_manager.cc:1397] T cee02fff2c9c48a19f43eb358a89e7b0 P c0f38c1908414566aa9115bb2cbdd85d: Time spent bootstrapping tablet: real 0.120s user 0.015s sys 0.002s
I20260809 15:45:53.563983 23933 raft_consensus.cc:482] T 3924b648461e463b9d7925e5671d68ae P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:45:53.564538 23933 raft_consensus.cc:504] T 3924b648461e463b9d7925e5671d68ae P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:53.566152 23593 tablet_bootstrap.cc:654] T 07fcabb569414bddbd39df6cedd24e64 P 65c36d164b0540f099d592f38307ee6c: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:53.567582 23591 raft_consensus.cc:348] T cee02fff2c9c48a19f43eb358a89e7b0 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:53.568260 23591 raft_consensus.cc:374] T cee02fff2c9c48a19f43eb358a89e7b0 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:53.568503 23591 raft_consensus.cc:729] T cee02fff2c9c48a19f43eb358a89e7b0 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Initialized, Role: FOLLOWER
I20260809 15:45:53.568908 23397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3924b648461e463b9d7925e5671d68ae" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:45:53.569286 23283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3924b648461e463b9d7925e5671d68ae" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
W20260809 15:45:53.571345 23472 leader_election.cc:341] T 3924b648461e463b9d7925e5671d68ae P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:53.572261 23591 consensus_queue.cc:260] T cee02fff2c9c48a19f43eb358a89e7b0 P c0f38c1908414566aa9115bb2cbdd85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:45:53.574815 23470 leader_election.cc:341] T 3924b648461e463b9d7925e5671d68ae P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:53.575241 23470 leader_election.cc:302] T 3924b648461e463b9d7925e5671d68ae P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:45:53.575980 23591 ts_tablet_manager.cc:1428] T cee02fff2c9c48a19f43eb358a89e7b0 P c0f38c1908414566aa9115bb2cbdd85d: Time spent starting tablet: real 0.012s user 0.007s sys 0.000s
I20260809 15:45:53.576292 23643 raft_consensus.cc:2726] T 3924b648461e463b9d7925e5671d68ae P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 15:45:53.576798 23591 tablet_bootstrap.cc:492] T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d: Bootstrap starting.
I20260809 15:45:53.575985 23594 tablet_bootstrap.cc:492] T 3924b648461e463b9d7925e5671d68ae P 67b3aeadfa7a432abe8af4d292e04ab9: No bootstrap required, opened a new log
I20260809 15:45:53.579636 23594 ts_tablet_manager.cc:1397] T 3924b648461e463b9d7925e5671d68ae P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent bootstrapping tablet: real 0.091s user 0.011s sys 0.006s
I20260809 15:45:53.569415 23933 leader_election.cc:288] T 3924b648461e463b9d7925e5671d68ae P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:45:53.582690 23594 raft_consensus.cc:348] T 3924b648461e463b9d7925e5671d68ae P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:53.583290 23594 raft_consensus.cc:374] T 3924b648461e463b9d7925e5671d68ae P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:53.583559 23594 raft_consensus.cc:729] T 3924b648461e463b9d7925e5671d68ae P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Initialized, Role: FOLLOWER
I20260809 15:45:53.584237 23594 consensus_queue.cc:260] T 3924b648461e463b9d7925e5671d68ae P 67b3aeadfa7a432abe8af4d292e04ab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:53.596513 23594 ts_tablet_manager.cc:1428] T 3924b648461e463b9d7925e5671d68ae P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent starting tablet: real 0.017s user 0.005s sys 0.000s
I20260809 15:45:53.597966 23594 tablet_bootstrap.cc:492] T 294d4bcc1e7a477cb56455581277a60f P 67b3aeadfa7a432abe8af4d292e04ab9: Bootstrap starting.
I20260809 15:45:53.588264 23933 raft_consensus.cc:482] T df20671a7b874dbb98c0ef28371c1308 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:45:53.605926 23933 raft_consensus.cc:504] T df20671a7b874dbb98c0ef28371c1308 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:53.607306 24129 raft_consensus.cc:482] T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:45:53.607831 24129 raft_consensus.cc:504] T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:53.608204 23933 leader_election.cc:288] T df20671a7b874dbb98c0ef28371c1308 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:45:53.610602 23283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df20671a7b874dbb98c0ef28371c1308" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:45:53.611312 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3e94b7e50754a81b3d72a07476cb7b9" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:45:53.611514 23283 raft_consensus.cc:2454] T df20671a7b874dbb98c0ef28371c1308 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 0.
W20260809 15:45:53.612766 23353 leader_election.cc:341] T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:53.613238 24129 leader_election.cc:288] T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:45:53.614099 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df20671a7b874dbb98c0ef28371c1308" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
W20260809 15:45:53.615254 23472 leader_election.cc:341] T df20671a7b874dbb98c0ef28371c1308 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:53.615859 23470 leader_election.cc:302] T df20671a7b874dbb98c0ef28371c1308 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters: c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:45:53.617266 23283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3e94b7e50754a81b3d72a07476cb7b9" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:45:53.617995 23283 raft_consensus.cc:2454] T d3e94b7e50754a81b3d72a07476cb7b9 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 0.
I20260809 15:45:53.618697 24160 raft_consensus.cc:2781] T df20671a7b874dbb98c0ef28371c1308 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 15:45:53.619204 24160 raft_consensus.cc:482] T df20671a7b874dbb98c0ef28371c1308 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:45:53.619645 24160 raft_consensus.cc:3037] T df20671a7b874dbb98c0ef28371c1308 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:53.621007 23351 leader_election.cc:302] T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c
I20260809 15:45:53.621981 24212 raft_consensus.cc:2781] T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 15:45:53.622455 24212 raft_consensus.cc:482] T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:45:53.626333 23117 catalog_manager.cc:5495] T 28e17dcb5efe45b8b6a8c42da03b4819 P c0f38c1908414566aa9115bb2cbdd85d reported cstate change: term changed from 0 to 1, leader changed from <none> to c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66). New cstate: current_term: 1 leader_uuid: "c0f38c1908414566aa9115bb2cbdd85d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } health_report { overall_health: UNKNOWN } } }
I20260809 15:45:53.631582 23594 tablet_bootstrap.cc:654] T 294d4bcc1e7a477cb56455581277a60f P 67b3aeadfa7a432abe8af4d292e04ab9: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:53.627624 24212 raft_consensus.cc:3037] T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:53.642614 24212 raft_consensus.cc:504] T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:53.644691 23283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3e94b7e50754a81b3d72a07476cb7b9" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:45:53.644940 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3e94b7e50754a81b3d72a07476cb7b9" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:45:53.645221 23283 raft_consensus.cc:3037] T d3e94b7e50754a81b3d72a07476cb7b9 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Advancing to term 1
W20260809 15:45:53.646075 23353 leader_election.cc:341] T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:53.648387 23591 tablet_bootstrap.cc:654] T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:53.650462 23593 tablet_bootstrap.cc:492] T 07fcabb569414bddbd39df6cedd24e64 P 65c36d164b0540f099d592f38307ee6c: No bootstrap required, opened a new log
I20260809 15:45:53.650869 23593 ts_tablet_manager.cc:1397] T 07fcabb569414bddbd39df6cedd24e64 P 65c36d164b0540f099d592f38307ee6c: Time spent bootstrapping tablet: real 0.104s user 0.005s sys 0.005s
I20260809 15:45:53.651193 24212 leader_election.cc:288] T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:45:53.645196 24160 raft_consensus.cc:504] T df20671a7b874dbb98c0ef28371c1308 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:53.653047 23593 raft_consensus.cc:348] T 07fcabb569414bddbd39df6cedd24e64 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:53.653716 23593 raft_consensus.cc:374] T 07fcabb569414bddbd39df6cedd24e64 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:53.653968 23593 raft_consensus.cc:729] T 07fcabb569414bddbd39df6cedd24e64 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Initialized, Role: FOLLOWER
I20260809 15:45:53.654410 23593 consensus_queue.cc:260] T 07fcabb569414bddbd39df6cedd24e64 P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:53.655330 23273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df20671a7b874dbb98c0ef28371c1308" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:45:53.655874 23273 raft_consensus.cc:3037] T df20671a7b874dbb98c0ef28371c1308 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:53.660137 23283 raft_consensus.cc:2454] T d3e94b7e50754a81b3d72a07476cb7b9 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 1.
I20260809 15:45:53.661769 23593 ts_tablet_manager.cc:1428] T 07fcabb569414bddbd39df6cedd24e64 P 65c36d164b0540f099d592f38307ee6c: Time spent starting tablet: real 0.011s user 0.005s sys 0.001s
I20260809 15:45:53.663030 23593 tablet_bootstrap.cc:492] T d3be1e356b254d22b995be32f1c12f6f P 65c36d164b0540f099d592f38307ee6c: Bootstrap starting.
I20260809 15:45:53.654726 24160 leader_election.cc:288] T df20671a7b874dbb98c0ef28371c1308 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 1 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:45:53.664834 23351 leader_election.cc:302] T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c
I20260809 15:45:53.665241 23591 tablet_bootstrap.cc:492] T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d: No bootstrap required, opened a new log
I20260809 15:45:53.665647 23591 ts_tablet_manager.cc:1397] T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d: Time spent bootstrapping tablet: real 0.089s user 0.010s sys 0.003s
I20260809 15:45:53.666321 23397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df20671a7b874dbb98c0ef28371c1308" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:45:53.668951 23591 raft_consensus.cc:348] T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:53.669603 23591 raft_consensus.cc:374] T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:53.669834 23591 raft_consensus.cc:729] T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Initialized, Role: FOLLOWER
I20260809 15:45:53.665565 23790 raft_consensus.cc:2781] T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:45:53.670539 23591 consensus_queue.cc:260] T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:53.671069 23790 raft_consensus.cc:686] T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d [term 1 LEADER]: Becoming Leader. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
W20260809 15:45:53.672040 23472 leader_election.cc:341] T df20671a7b874dbb98c0ef28371c1308 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:53.671936 23790 consensus_queue.cc:237] T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:53.673583 23594 tablet_bootstrap.cc:492] T 294d4bcc1e7a477cb56455581277a60f P 67b3aeadfa7a432abe8af4d292e04ab9: No bootstrap required, opened a new log
I20260809 15:45:53.673969 23594 ts_tablet_manager.cc:1397] T 294d4bcc1e7a477cb56455581277a60f P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent bootstrapping tablet: real 0.077s user 0.008s sys 0.001s
I20260809 15:45:53.672075 23273 raft_consensus.cc:2454] T df20671a7b874dbb98c0ef28371c1308 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 1.
I20260809 15:45:53.675560 23470 leader_election.cc:302] T df20671a7b874dbb98c0ef28371c1308 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters: c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:45:53.676183 23594 raft_consensus.cc:348] T 294d4bcc1e7a477cb56455581277a60f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:53.676527 23691 raft_consensus.cc:2781] T df20671a7b874dbb98c0ef28371c1308 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:45:53.676826 23594 raft_consensus.cc:374] T 294d4bcc1e7a477cb56455581277a60f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:53.677003 23691 raft_consensus.cc:686] T df20671a7b874dbb98c0ef28371c1308 P 65c36d164b0540f099d592f38307ee6c [term 1 LEADER]: Becoming Leader. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:45:53.677127 23594 raft_consensus.cc:729] T 294d4bcc1e7a477cb56455581277a60f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Initialized, Role: FOLLOWER
I20260809 15:45:53.677834 23594 consensus_queue.cc:260] T 294d4bcc1e7a477cb56455581277a60f P 67b3aeadfa7a432abe8af4d292e04ab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:53.677861 23691 consensus_queue.cc:237] T df20671a7b874dbb98c0ef28371c1308 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:53.679302 23594 ts_tablet_manager.cc:1428] T 294d4bcc1e7a477cb56455581277a60f P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent starting tablet: real 0.005s user 0.005s sys 0.002s
I20260809 15:45:53.680094 23594 tablet_bootstrap.cc:492] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9: Bootstrap starting.
I20260809 15:45:53.682389 23591 ts_tablet_manager.cc:1428] T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d: Time spent starting tablet: real 0.016s user 0.004s sys 0.002s
I20260809 15:45:53.683336 23593 tablet_bootstrap.cc:654] T d3be1e356b254d22b995be32f1c12f6f P 65c36d164b0540f099d592f38307ee6c: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:53.685051 23591 tablet_bootstrap.cc:492] T df20671a7b874dbb98c0ef28371c1308 P c0f38c1908414566aa9115bb2cbdd85d: Bootstrap starting.
I20260809 15:45:53.688227 23594 tablet_bootstrap.cc:654] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:53.687678 23115 catalog_manager.cc:5495] T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d reported cstate change: term changed from 0 to 1, leader changed from <none> to c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66). New cstate: current_term: 1 leader_uuid: "c0f38c1908414566aa9115bb2cbdd85d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } health_report { overall_health: UNKNOWN } } }
I20260809 15:45:53.699715 23591 tablet_bootstrap.cc:654] T df20671a7b874dbb98c0ef28371c1308 P c0f38c1908414566aa9115bb2cbdd85d: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:53.707974 23790 raft_consensus.cc:482] T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:45:53.708595 23790 raft_consensus.cc:504] T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:53.711382 23790 leader_election.cc:288] T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:45:53.718154 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3924b648461e463b9d7925e5671d68ae" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:45:53.719048 23527 raft_consensus.cc:2454] T 3924b648461e463b9d7925e5671d68ae P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 0.
I20260809 15:45:53.719334 23273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3924b648461e463b9d7925e5671d68ae" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:45:53.720090 23273 raft_consensus.cc:2454] T 3924b648461e463b9d7925e5671d68ae P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 0.
I20260809 15:45:53.720623 23353 leader_election.cc:302] T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:45:53.721724 24212 raft_consensus.cc:2781] T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 15:45:53.722196 24212 raft_consensus.cc:482] T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:45:53.722702 24212 raft_consensus.cc:3037] T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:53.726806 23591 tablet_bootstrap.cc:492] T df20671a7b874dbb98c0ef28371c1308 P c0f38c1908414566aa9115bb2cbdd85d: No bootstrap required, opened a new log
I20260809 15:45:53.727154 23591 ts_tablet_manager.cc:1397] T df20671a7b874dbb98c0ef28371c1308 P c0f38c1908414566aa9115bb2cbdd85d: Time spent bootstrapping tablet: real 0.042s user 0.008s sys 0.007s
I20260809 15:45:53.722198 23790 raft_consensus.cc:482] T 4b30f28311964a0b9149a50495e83d8f P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:45:53.728123 23594 tablet_bootstrap.cc:492] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9: No bootstrap required, opened a new log
I20260809 15:45:53.728655 23594 ts_tablet_manager.cc:1397] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent bootstrapping tablet: real 0.049s user 0.017s sys 0.011s
I20260809 15:45:53.728183 23790 raft_consensus.cc:504] T 4b30f28311964a0b9149a50495e83d8f P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:53.729790 23591 raft_consensus.cc:348] T df20671a7b874dbb98c0ef28371c1308 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:53.730473 23591 raft_consensus.cc:374] T df20671a7b874dbb98c0ef28371c1308 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:53.730720 23591 raft_consensus.cc:729] T df20671a7b874dbb98c0ef28371c1308 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Initialized, Role: FOLLOWER
I20260809 15:45:53.731307 23591 consensus_queue.cc:260] T df20671a7b874dbb98c0ef28371c1308 P c0f38c1908414566aa9115bb2cbdd85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:53.732831 23591 ts_tablet_manager.cc:1428] T df20671a7b874dbb98c0ef28371c1308 P c0f38c1908414566aa9115bb2cbdd85d: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260809 15:45:53.732676 23594 raft_consensus.cc:348] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:53.733355 23594 raft_consensus.cc:374] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:53.733009 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b30f28311964a0b9149a50495e83d8f" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:45:53.733567 23594 raft_consensus.cc:729] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Initialized, Role: FOLLOWER
I20260809 15:45:53.734615 23591 tablet_bootstrap.cc:492] T 907bacfa79d04a55a68c8dbe60056462 P c0f38c1908414566aa9115bb2cbdd85d: Bootstrap starting.
W20260809 15:45:53.734521 23353 leader_election.cc:341] T 4b30f28311964a0b9149a50495e83d8f P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:53.735798 23594 consensus_queue.cc:260] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:53.738555 23594 ts_tablet_manager.cc:1428] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent starting tablet: real 0.009s user 0.005s sys 0.001s
I20260809 15:45:53.743748 23273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b30f28311964a0b9149a50495e83d8f" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:45:53.747859 23591 tablet_bootstrap.cc:654] T 907bacfa79d04a55a68c8dbe60056462 P c0f38c1908414566aa9115bb2cbdd85d: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:53.744932 23273 raft_consensus.cc:2454] T 4b30f28311964a0b9149a50495e83d8f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 0.
I20260809 15:45:53.752080 23790 leader_election.cc:288] T 4b30f28311964a0b9149a50495e83d8f P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:45:53.753741 23351 leader_election.cc:302] T 4b30f28311964a0b9149a50495e83d8f P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c
I20260809 15:45:53.755721 23691 consensus_queue.cc:1035] T 294d4bcc1e7a477cb56455581277a60f P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260809 15:45:53.756281 24129 raft_consensus.cc:2781] T 4b30f28311964a0b9149a50495e83d8f P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 15:45:53.756703 24129 raft_consensus.cc:482] T 4b30f28311964a0b9149a50495e83d8f P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:45:53.757040 24129 raft_consensus.cc:3037] T 4b30f28311964a0b9149a50495e83d8f P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Advancing to term 1
W20260809 15:45:53.757714 24300 log.cc:924] Time spent T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d: Append to log took a long time: real 0.083s user 0.000s sys 0.000s
I20260809 15:45:53.759912 24212 raft_consensus.cc:504] T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:53.768141 23273 raft_consensus.cc:3037] T 294d4bcc1e7a477cb56455581277a60f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:53.768271 23591 tablet_bootstrap.cc:492] T 907bacfa79d04a55a68c8dbe60056462 P c0f38c1908414566aa9115bb2cbdd85d: No bootstrap required, opened a new log
I20260809 15:45:53.768814 23591 ts_tablet_manager.cc:1397] T 907bacfa79d04a55a68c8dbe60056462 P c0f38c1908414566aa9115bb2cbdd85d: Time spent bootstrapping tablet: real 0.034s user 0.012s sys 0.009s
I20260809 15:45:53.776976 24212 leader_election.cc:288] T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:45:53.778074 23529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3924b648461e463b9d7925e5671d68ae" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:45:53.779143 23529 raft_consensus.cc:3037] T 3924b648461e463b9d7925e5671d68ae P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:53.781183 23591 raft_consensus.cc:348] T 907bacfa79d04a55a68c8dbe60056462 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:53.781822 23591 raft_consensus.cc:374] T 907bacfa79d04a55a68c8dbe60056462 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:53.782084 23591 raft_consensus.cc:729] T 907bacfa79d04a55a68c8dbe60056462 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Initialized, Role: FOLLOWER
I20260809 15:45:53.782686 23591 consensus_queue.cc:260] T 907bacfa79d04a55a68c8dbe60056462 P c0f38c1908414566aa9115bb2cbdd85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:53.783790 23274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3924b648461e463b9d7925e5671d68ae" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:45:53.784439 23274 raft_consensus.cc:3037] T 3924b648461e463b9d7925e5671d68ae P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:53.784901 23591 ts_tablet_manager.cc:1428] T 907bacfa79d04a55a68c8dbe60056462 P c0f38c1908414566aa9115bb2cbdd85d: Time spent starting tablet: real 0.016s user 0.000s sys 0.007s
I20260809 15:45:53.785792 23591 tablet_bootstrap.cc:492] T d3be1e356b254d22b995be32f1c12f6f P c0f38c1908414566aa9115bb2cbdd85d: Bootstrap starting.
I20260809 15:45:53.777976 24160 consensus_queue.cc:1035] T 294d4bcc1e7a477cb56455581277a60f P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:45:53.790211 23593 tablet_bootstrap.cc:492] T d3be1e356b254d22b995be32f1c12f6f P 65c36d164b0540f099d592f38307ee6c: No bootstrap required, opened a new log
I20260809 15:45:53.790750 23593 ts_tablet_manager.cc:1397] T d3be1e356b254d22b995be32f1c12f6f P 65c36d164b0540f099d592f38307ee6c: Time spent bootstrapping tablet: real 0.128s user 0.032s sys 0.008s
I20260809 15:45:53.795130 23591 tablet_bootstrap.cc:654] T d3be1e356b254d22b995be32f1c12f6f P c0f38c1908414566aa9115bb2cbdd85d: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:53.797307 23529 raft_consensus.cc:2454] T 3924b648461e463b9d7925e5671d68ae P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 1.
I20260809 15:45:53.798120 23353 leader_election.cc:302] T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:45:53.798800 24130 raft_consensus.cc:482] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:45:53.801324 24130 raft_consensus.cc:504] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:53.804039 23116 catalog_manager.cc:5495] T df20671a7b874dbb98c0ef28371c1308 P 65c36d164b0540f099d592f38307ee6c reported cstate change: term changed from 0 to 1, leader changed from <none> to 65c36d164b0540f099d592f38307ee6c (127.20.202.67). New cstate: current_term: 1 leader_uuid: "65c36d164b0540f099d592f38307ee6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } health_report { overall_health: HEALTHY } } }
I20260809 15:45:53.806285 23593 raft_consensus.cc:348] T d3be1e356b254d22b995be32f1c12f6f P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:53.806833 23593 raft_consensus.cc:374] T d3be1e356b254d22b995be32f1c12f6f P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:53.807025 23593 raft_consensus.cc:729] T d3be1e356b254d22b995be32f1c12f6f P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Initialized, Role: FOLLOWER
I20260809 15:45:53.805519 24212 raft_consensus.cc:2781] T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:45:53.808630 24212 raft_consensus.cc:686] T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d [term 1 LEADER]: Becoming Leader. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:45:53.811414 23404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "907bacfa79d04a55a68c8dbe60056462" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:45:53.812034 24130 leader_election.cc:288] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:45:53.812258 23404 raft_consensus.cc:2454] T 907bacfa79d04a55a68c8dbe60056462 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 0.
I20260809 15:45:53.813138 24212 consensus_queue.cc:237] T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:53.811025 23593 consensus_queue.cc:260] T d3be1e356b254d22b995be32f1c12f6f P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:53.819036 23228 leader_election.cc:302] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:45:53.821094 23529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "907bacfa79d04a55a68c8dbe60056462" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:45:53.823228 24159 consensus_queue.cc:1035] T f9e7cadcd24d45fcbfa84209225e8f2d P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:45:53.822736 24130 raft_consensus.cc:2781] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 15:45:53.830288 24130 raft_consensus.cc:482] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:45:53.830605 24130 raft_consensus.cc:3037] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Advancing to term 1
W20260809 15:45:53.832484 23228 leader_election.cc:341] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:53.834884 23593 ts_tablet_manager.cc:1428] T d3be1e356b254d22b995be32f1c12f6f P 65c36d164b0540f099d592f38307ee6c: Time spent starting tablet: real 0.044s user 0.006s sys 0.000s
I20260809 15:45:53.835732 23593 tablet_bootstrap.cc:492] T 28e17dcb5efe45b8b6a8c42da03b4819 P 65c36d164b0540f099d592f38307ee6c: Bootstrap starting.
I20260809 15:45:53.836035 23591 tablet_bootstrap.cc:492] T d3be1e356b254d22b995be32f1c12f6f P c0f38c1908414566aa9115bb2cbdd85d: No bootstrap required, opened a new log
I20260809 15:45:53.836448 23591 ts_tablet_manager.cc:1397] T d3be1e356b254d22b995be32f1c12f6f P c0f38c1908414566aa9115bb2cbdd85d: Time spent bootstrapping tablet: real 0.051s user 0.009s sys 0.009s
I20260809 15:45:53.857421 24129 raft_consensus.cc:504] T 4b30f28311964a0b9149a50495e83d8f P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:53.862931 23790 raft_consensus.cc:482] T df20671a7b874dbb98c0ef28371c1308 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:45:53.870133 24129 leader_election.cc:288] T 4b30f28311964a0b9149a50495e83d8f P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:45:53.865610 23790 raft_consensus.cc:504] T df20671a7b874dbb98c0ef28371c1308 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:53.871848 23273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b30f28311964a0b9149a50495e83d8f" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:45:53.872452 23273 raft_consensus.cc:3037] T 4b30f28311964a0b9149a50495e83d8f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:53.873032 23529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b30f28311964a0b9149a50495e83d8f" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
W20260809 15:45:53.875228 23353 leader_election.cc:341] T 4b30f28311964a0b9149a50495e83d8f P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:53.873409 23591 raft_consensus.cc:348] T d3be1e356b254d22b995be32f1c12f6f P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:53.879570 23591 raft_consensus.cc:374] T d3be1e356b254d22b995be32f1c12f6f P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:53.879799 23591 raft_consensus.cc:729] T d3be1e356b254d22b995be32f1c12f6f P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Initialized, Role: FOLLOWER
I20260809 15:45:53.880282 23591 consensus_queue.cc:260] T d3be1e356b254d22b995be32f1c12f6f P c0f38c1908414566aa9115bb2cbdd85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:53.890453 23591 ts_tablet_manager.cc:1428] T d3be1e356b254d22b995be32f1c12f6f P c0f38c1908414566aa9115bb2cbdd85d: Time spent starting tablet: real 0.054s user 0.005s sys 0.001s
I20260809 15:45:53.891729 23790 leader_election.cc:288] T df20671a7b874dbb98c0ef28371c1308 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:45:53.892467 23283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df20671a7b874dbb98c0ef28371c1308" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:45:53.893126 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df20671a7b874dbb98c0ef28371c1308" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:45:53.898336 23593 tablet_bootstrap.cc:654] T 28e17dcb5efe45b8b6a8c42da03b4819 P 65c36d164b0540f099d592f38307ee6c: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:53.904165 23283 raft_consensus.cc:2379] T df20671a7b874dbb98c0ef28371c1308 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d in current term 1: Already voted for candidate 65c36d164b0540f099d592f38307ee6c in this term.
I20260809 15:45:53.937695 24130 raft_consensus.cc:504] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:53.940492 24130 leader_election.cc:288] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:45:53.944135 23274 raft_consensus.cc:2454] T 3924b648461e463b9d7925e5671d68ae P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 1.
I20260809 15:45:53.944372 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "907bacfa79d04a55a68c8dbe60056462" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:45:53.944811 23351 leader_election.cc:302] T df20671a7b874dbb98c0ef28371c1308 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:45:53.946063 23648 consensus_queue.cc:1035] T 28e17dcb5efe45b8b6a8c42da03b4819 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:45:53.946935 23648 consensus_queue.cc:1035] T 07fcabb569414bddbd39df6cedd24e64 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:45:53.947518 23273 raft_consensus.cc:2454] T 4b30f28311964a0b9149a50495e83d8f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 1.
I20260809 15:45:53.947948 23648 raft_consensus.cc:3037] T df20671a7b874dbb98c0ef28371c1308 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Advancing to term 1
W20260809 15:45:53.949173 23228 leader_election.cc:341] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553): Illegal state: must be running to vote when last-logged opid is not known
I20260809 15:45:53.952641 23115 catalog_manager.cc:5495] T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d reported cstate change: term changed from 0 to 1, leader changed from <none> to c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66). New cstate: current_term: 1 leader_uuid: "c0f38c1908414566aa9115bb2cbdd85d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } health_report { overall_health: UNKNOWN } } }
I20260809 15:45:53.954317 23394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "907bacfa79d04a55a68c8dbe60056462" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:45:53.954278 23648 raft_consensus.cc:2726] T df20671a7b874dbb98c0ef28371c1308 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 15:45:53.957194 23351 leader_election.cc:302] T 4b30f28311964a0b9149a50495e83d8f P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c
I20260809 15:45:53.957803 23648 raft_consensus.cc:2781] T 4b30f28311964a0b9149a50495e83d8f P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:45:53.955024 23394 raft_consensus.cc:3037] T 907bacfa79d04a55a68c8dbe60056462 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:53.964022 23648 raft_consensus.cc:686] T 4b30f28311964a0b9149a50495e83d8f P c0f38c1908414566aa9115bb2cbdd85d [term 1 LEADER]: Becoming Leader. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:45:53.965723 23648 consensus_queue.cc:237] T 4b30f28311964a0b9149a50495e83d8f P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:53.968674 23790 raft_consensus.cc:482] T d3be1e356b254d22b995be32f1c12f6f P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:45:53.969275 23790 raft_consensus.cc:504] T d3be1e356b254d22b995be32f1c12f6f P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:53.970346 23933 consensus_queue.cc:1035] T f9e7cadcd24d45fcbfa84209225e8f2d P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:45:53.982002 23529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3be1e356b254d22b995be32f1c12f6f" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:45:53.982641 23529 raft_consensus.cc:2454] T d3be1e356b254d22b995be32f1c12f6f P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 0.
I20260809 15:45:53.981956 23280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3be1e356b254d22b995be32f1c12f6f" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:45:53.983551 23353 leader_election.cc:302] T d3be1e356b254d22b995be32f1c12f6f P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:45:53.974161 23790 leader_election.cc:288] T d3be1e356b254d22b995be32f1c12f6f P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:45:53.995054 23691 raft_consensus.cc:482] T cee02fff2c9c48a19f43eb358a89e7b0 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:45:53.995594 23691 raft_consensus.cc:504] T cee02fff2c9c48a19f43eb358a89e7b0 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:53.997485 23691 leader_election.cc:288] T cee02fff2c9c48a19f43eb358a89e7b0 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:45:53.988947 24211 raft_consensus.cc:2781] T d3be1e356b254d22b995be32f1c12f6f P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 15:45:53.990903 23280 raft_consensus.cc:2454] T d3be1e356b254d22b995be32f1c12f6f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 0.
I20260809 15:45:54.002270 24211 raft_consensus.cc:482] T d3be1e356b254d22b995be32f1c12f6f P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:45:54.025940 23527 raft_consensus.cc:3037] T 07fcabb569414bddbd39df6cedd24e64 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:54.011216 23397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee02fff2c9c48a19f43eb358a89e7b0" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:45:54.029155 23273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee02fff2c9c48a19f43eb358a89e7b0" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:45:54.014451 24211 raft_consensus.cc:3037] T d3be1e356b254d22b995be32f1c12f6f P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:54.039428 23593 tablet_bootstrap.cc:492] T 28e17dcb5efe45b8b6a8c42da03b4819 P 65c36d164b0540f099d592f38307ee6c: No bootstrap required, opened a new log
I20260809 15:45:54.040162 23593 ts_tablet_manager.cc:1397] T 28e17dcb5efe45b8b6a8c42da03b4819 P 65c36d164b0540f099d592f38307ee6c: Time spent bootstrapping tablet: real 0.205s user 0.017s sys 0.012s
I20260809 15:45:54.041887 23397 raft_consensus.cc:2454] T cee02fff2c9c48a19f43eb358a89e7b0 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 0.
I20260809 15:45:54.043866 23593 raft_consensus.cc:348] T 28e17dcb5efe45b8b6a8c42da03b4819 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:54.044538 23593 raft_consensus.cc:374] T 28e17dcb5efe45b8b6a8c42da03b4819 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:54.044785 23593 raft_consensus.cc:729] T 28e17dcb5efe45b8b6a8c42da03b4819 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Initialized, Role: FOLLOWER
I20260809 15:45:54.045375 23593 consensus_queue.cc:260] T 28e17dcb5efe45b8b6a8c42da03b4819 P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:54.046571 23394 raft_consensus.cc:2454] T 907bacfa79d04a55a68c8dbe60056462 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 1.
I20260809 15:45:54.049961 23273 raft_consensus.cc:2454] T cee02fff2c9c48a19f43eb358a89e7b0 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 0.
I20260809 15:45:54.055709 23472 leader_election.cc:302] T cee02fff2c9c48a19f43eb358a89e7b0 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:45:54.056576 24318 raft_consensus.cc:2781] T cee02fff2c9c48a19f43eb358a89e7b0 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 15:45:54.056910 24318 raft_consensus.cc:482] T cee02fff2c9c48a19f43eb358a89e7b0 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:45:54.057250 24318 raft_consensus.cc:3037] T cee02fff2c9c48a19f43eb358a89e7b0 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Advancing to term 1
W20260809 15:45:54.068164 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 69026.7us
I20260809 15:45:54.081882 24317 raft_consensus.cc:482] T 07fcabb569414bddbd39df6cedd24e64 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:45:54.082326 24317 raft_consensus.cc:504] T 07fcabb569414bddbd39df6cedd24e64 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:54.085515 23790 consensus_queue.cc:1035] T 07fcabb569414bddbd39df6cedd24e64 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:45:54.095790 23593 ts_tablet_manager.cc:1428] T 28e17dcb5efe45b8b6a8c42da03b4819 P 65c36d164b0540f099d592f38307ee6c: Time spent starting tablet: real 0.055s user 0.006s sys 0.000s
I20260809 15:45:54.096735 23593 tablet_bootstrap.cc:492] T 4b30f28311964a0b9149a50495e83d8f P 65c36d164b0540f099d592f38307ee6c: Bootstrap starting.
I20260809 15:45:54.097667 23691 consensus_queue.cc:1035] T df20671a7b874dbb98c0ef28371c1308 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:45:54.100023 23228 leader_election.cc:302] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c
I20260809 15:45:54.103174 23115 catalog_manager.cc:5495] T 4b30f28311964a0b9149a50495e83d8f P c0f38c1908414566aa9115bb2cbdd85d reported cstate change: term changed from 0 to 1, leader changed from <none> to c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66). New cstate: current_term: 1 leader_uuid: "c0f38c1908414566aa9115bb2cbdd85d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } health_report { overall_health: UNKNOWN } } }
I20260809 15:45:54.103823 24130 raft_consensus.cc:2781] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:45:54.109012 24130 raft_consensus.cc:686] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 LEADER]: Becoming Leader. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Running, Role: LEADER
I20260809 15:45:54.109748 23526 raft_consensus.cc:3037] T 28e17dcb5efe45b8b6a8c42da03b4819 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:54.113931 24211 raft_consensus.cc:504] T d3be1e356b254d22b995be32f1c12f6f P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:54.110427 24130 consensus_queue.cc:237] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:45:54.143936 23353 consensus_peers.cc:487] T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d -> Peer 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553): Couldn't send request to peer 65c36d164b0540f099d592f38307ee6c. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260809 15:45:54.148773 24317 leader_election.cc:288] T 07fcabb569414bddbd39df6cedd24e64 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:45:54.152014 23397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07fcabb569414bddbd39df6cedd24e64" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:45:54.157161 24211 leader_election.cc:288] T d3be1e356b254d22b995be32f1c12f6f P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:45:54.158123 23529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3be1e356b254d22b995be32f1c12f6f" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:45:54.158632 23529 raft_consensus.cc:3037] T d3be1e356b254d22b995be32f1c12f6f P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:54.165477 23529 raft_consensus.cc:2454] T d3be1e356b254d22b995be32f1c12f6f P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 1.
I20260809 15:45:54.166603 23353 leader_election.cc:302] T d3be1e356b254d22b995be32f1c12f6f P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:45:54.167390 23648 raft_consensus.cc:2781] T d3be1e356b254d22b995be32f1c12f6f P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:45:54.167860 23648 raft_consensus.cc:686] T d3be1e356b254d22b995be32f1c12f6f P c0f38c1908414566aa9115bb2cbdd85d [term 1 LEADER]: Becoming Leader. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:45:54.168553 23648 consensus_queue.cc:237] T d3be1e356b254d22b995be32f1c12f6f P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:54.176697 24123 consensus_queue.cc:1035] T 28e17dcb5efe45b8b6a8c42da03b4819 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20260809 15:45:54.180097 24318 raft_consensus.cc:504] T cee02fff2c9c48a19f43eb358a89e7b0 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:54.190944 24318 leader_election.cc:288] T cee02fff2c9c48a19f43eb358a89e7b0 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 1 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:45:54.192358 23593 tablet_bootstrap.cc:654] T 4b30f28311964a0b9149a50495e83d8f P 65c36d164b0540f099d592f38307ee6c: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:54.194773 23397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee02fff2c9c48a19f43eb358a89e7b0" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:45:54.195725 23397 raft_consensus.cc:3037] T cee02fff2c9c48a19f43eb358a89e7b0 P c0f38c1908414566aa9115bb2cbdd85d [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:54.209141 23397 raft_consensus.cc:2454] T cee02fff2c9c48a19f43eb358a89e7b0 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 1.
I20260809 15:45:54.210498 23472 leader_election.cc:302] T cee02fff2c9c48a19f43eb358a89e7b0 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:45:54.211349 24317 raft_consensus.cc:2781] T cee02fff2c9c48a19f43eb358a89e7b0 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Leader election won for term 1
I20260809 15:45:54.212060 24317 raft_consensus.cc:686] T cee02fff2c9c48a19f43eb358a89e7b0 P 65c36d164b0540f099d592f38307ee6c [term 1 LEADER]: Becoming Leader. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:45:54.217811 24317 consensus_queue.cc:237] T cee02fff2c9c48a19f43eb358a89e7b0 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:54.217943 23270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3be1e356b254d22b995be32f1c12f6f" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:45:54.219606 23270 raft_consensus.cc:3037] T d3be1e356b254d22b995be32f1c12f6f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:54.301975 23115 catalog_manager.cc:5495] T d3be1e356b254d22b995be32f1c12f6f P c0f38c1908414566aa9115bb2cbdd85d reported cstate change: term changed from 0 to 1, leader changed from <none> to c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66). New cstate: current_term: 1 leader_uuid: "c0f38c1908414566aa9115bb2cbdd85d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } health_report { overall_health: UNKNOWN } } }
I20260809 15:45:54.314687 23117 catalog_manager.cc:5495] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65). New cstate: current_term: 1 leader_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } health_report { overall_health: UNKNOWN } } }
I20260809 15:45:54.325057 23270 raft_consensus.cc:2454] T d3be1e356b254d22b995be32f1c12f6f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 1.
I20260809 15:45:54.328539 24123 consensus_queue.cc:1035] T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260809 15:45:54.333192 23193 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260809 15:45:54.333865 23274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07fcabb569414bddbd39df6cedd24e64" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:45:54.346221 23274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee02fff2c9c48a19f43eb358a89e7b0" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:45:54.346829 23274 raft_consensus.cc:3037] T cee02fff2c9c48a19f43eb358a89e7b0 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 0 FOLLOWER]: Advancing to term 1
W20260809 15:45:54.360646 23353 consensus_peers.cc:487] T 4b30f28311964a0b9149a50495e83d8f P c0f38c1908414566aa9115bb2cbdd85d -> Peer 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553): Couldn't send request to peer 65c36d164b0540f099d592f38307ee6c. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260809 15:45:54.397751 23274 raft_consensus.cc:2454] T cee02fff2c9c48a19f43eb358a89e7b0 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 1.
I20260809 15:45:54.398146 23593 tablet_bootstrap.cc:492] T 4b30f28311964a0b9149a50495e83d8f P 65c36d164b0540f099d592f38307ee6c: No bootstrap required, opened a new log
I20260809 15:45:54.399215 23593 ts_tablet_manager.cc:1397] T 4b30f28311964a0b9149a50495e83d8f P 65c36d164b0540f099d592f38307ee6c: Time spent bootstrapping tablet: real 0.303s user 0.038s sys 0.041s
I20260809 15:45:54.402299 23593 raft_consensus.cc:348] T 4b30f28311964a0b9149a50495e83d8f P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:54.403017 23593 raft_consensus.cc:374] T 4b30f28311964a0b9149a50495e83d8f P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:54.403321 23593 raft_consensus.cc:729] T 4b30f28311964a0b9149a50495e83d8f P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Initialized, Role: FOLLOWER
I20260809 15:45:54.403959 23593 consensus_queue.cc:260] T 4b30f28311964a0b9149a50495e83d8f P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:54.406183 23593 ts_tablet_manager.cc:1428] T 4b30f28311964a0b9149a50495e83d8f P 65c36d164b0540f099d592f38307ee6c: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260809 15:45:54.407164 24138 consensus_queue.cc:1035] T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260809 15:45:54.409034 23470 leader_election.cc:302] T 07fcabb569414bddbd39df6cedd24e64 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:45:54.410884 24160 raft_consensus.cc:2726] T 07fcabb569414bddbd39df6cedd24e64 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 15:45:54.411201 24159 consensus_queue.cc:1035] T df20671a7b874dbb98c0ef28371c1308 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260809 15:45:54.422324 23593 tablet_bootstrap.cc:492] T 907bacfa79d04a55a68c8dbe60056462 P 65c36d164b0540f099d592f38307ee6c: Bootstrap starting.
I20260809 15:45:54.436823 23648 consensus_queue.cc:1035] T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:45:54.438999 23117 catalog_manager.cc:5495] T cee02fff2c9c48a19f43eb358a89e7b0 P 65c36d164b0540f099d592f38307ee6c reported cstate change: term changed from 0 to 1, leader changed from <none> to 65c36d164b0540f099d592f38307ee6c (127.20.202.67). New cstate: current_term: 1 leader_uuid: "65c36d164b0540f099d592f38307ee6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } health_report { overall_health: HEALTHY } } }
I20260809 15:45:54.449438 23593 tablet_bootstrap.cc:654] T 907bacfa79d04a55a68c8dbe60056462 P 65c36d164b0540f099d592f38307ee6c: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:54.469410 23593 tablet_bootstrap.cc:492] T 907bacfa79d04a55a68c8dbe60056462 P 65c36d164b0540f099d592f38307ee6c: No bootstrap required, opened a new log
I20260809 15:45:54.469805 23593 ts_tablet_manager.cc:1397] T 907bacfa79d04a55a68c8dbe60056462 P 65c36d164b0540f099d592f38307ee6c: Time spent bootstrapping tablet: real 0.049s user 0.007s sys 0.006s
I20260809 15:45:54.474193 23593 raft_consensus.cc:348] T 907bacfa79d04a55a68c8dbe60056462 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:54.475051 23593 raft_consensus.cc:374] T 907bacfa79d04a55a68c8dbe60056462 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:54.475335 23593 raft_consensus.cc:729] T 907bacfa79d04a55a68c8dbe60056462 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Initialized, Role: FOLLOWER
I20260809 15:45:54.475976 23593 consensus_queue.cc:260] T 907bacfa79d04a55a68c8dbe60056462 P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:54.480224 23593 ts_tablet_manager.cc:1428] T 907bacfa79d04a55a68c8dbe60056462 P 65c36d164b0540f099d592f38307ee6c: Time spent starting tablet: real 0.010s user 0.005s sys 0.005s
I20260809 15:45:54.480989 23593 tablet_bootstrap.cc:492] T d3e94b7e50754a81b3d72a07476cb7b9 P 65c36d164b0540f099d592f38307ee6c: Bootstrap starting.
I20260809 15:45:54.489010 23593 tablet_bootstrap.cc:654] T d3e94b7e50754a81b3d72a07476cb7b9 P 65c36d164b0540f099d592f38307ee6c: Neither blocks nor log segments found. Creating new log.
I20260809 15:45:54.494967 23593 tablet_bootstrap.cc:492] T d3e94b7e50754a81b3d72a07476cb7b9 P 65c36d164b0540f099d592f38307ee6c: No bootstrap required, opened a new log
I20260809 15:45:54.495280 23593 ts_tablet_manager.cc:1397] T d3e94b7e50754a81b3d72a07476cb7b9 P 65c36d164b0540f099d592f38307ee6c: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.008s
I20260809 15:45:54.497548 23593 raft_consensus.cc:348] T d3e94b7e50754a81b3d72a07476cb7b9 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:54.498013 23593 raft_consensus.cc:374] T d3e94b7e50754a81b3d72a07476cb7b9 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 15:45:54.498227 23593 raft_consensus.cc:729] T d3e94b7e50754a81b3d72a07476cb7b9 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Initialized, Role: FOLLOWER
I20260809 15:45:54.498649 23593 consensus_queue.cc:260] T d3e94b7e50754a81b3d72a07476cb7b9 P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:45:54.500448 23593 ts_tablet_manager.cc:1428] T d3e94b7e50754a81b3d72a07476cb7b9 P 65c36d164b0540f099d592f38307ee6c: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260809 15:45:54.560523 24336 consensus_queue.cc:1035] T 4b30f28311964a0b9149a50495e83d8f P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:45:54.625841 23527 raft_consensus.cc:3037] T d3e94b7e50754a81b3d72a07476cb7b9 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:54.713914 23527 raft_consensus.cc:3037] T 907bacfa79d04a55a68c8dbe60056462 P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:54.718700 24346 consensus_queue.cc:1035] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:45:54.726672 24346 consensus_queue.cc:1035] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:45:54.747135 24123 consensus_queue.cc:1035] T d3be1e356b254d22b995be32f1c12f6f P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:45:54.756358 23527 raft_consensus.cc:3037] T 4b30f28311964a0b9149a50495e83d8f P 65c36d164b0540f099d592f38307ee6c [term 0 FOLLOWER]: Advancing to term 1
I20260809 15:45:54.756503 23648 consensus_queue.cc:1035] T d3be1e356b254d22b995be32f1c12f6f P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260809 15:45:54.758194 24159 consensus_queue.cc:1035] T cee02fff2c9c48a19f43eb358a89e7b0 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:45:54.765499 24159 consensus_queue.cc:1035] T cee02fff2c9c48a19f43eb358a89e7b0 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 15:45:56.254482 21289 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 15:45:56.281055 24390 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 15:45:56.281241 24393 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:45:56.282229 21289 server_base.cc:1034] running on GCE node
W20260809 15:45:56.282560 24391 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 15:45:56.285128 21289 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260809 15:45:56.285326 21289 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260809 15:45:56.285502 21289 hybrid_clock.cc:648] HybridClock initialized: now 1786290356285480 us; error 0 us; skew 500 ppm
I20260809 15:45:56.285993 21289 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 15:45:56.292450 21289 webserver.cc:460] Webserver started at http://127.20.202.68:36157/ using document root <none> and password file <none>
I20260809 15:45:56.292896 21289 fs_manager.cc:359] Metadata directory not provided
I20260809 15:45:56.293061 21289 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 15:45:56.293352 21289 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 15:45:56.294446 21289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskj2AJ_N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786290285951082-21289-0/minicluster-data/ts-3-root/instance:
uuid: "61e69695258f48baa984c9d395e8ce3f"
format_stamp: "Formatted at 2026-08-09 15:45:56 on dist-test-slave-gjrz"
I20260809 15:45:56.300935 21289 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260809 15:45:56.304122 24401 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 15:45:56.304903 21289 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 15:45:56.305174 21289 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskj2AJ_N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786290285951082-21289-0/minicluster-data/ts-3-root
uuid: "61e69695258f48baa984c9d395e8ce3f"
format_stamp: "Formatted at 2026-08-09 15:45:56 on dist-test-slave-gjrz"
I20260809 15:45:56.305414 21289 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskj2AJ_N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786290285951082-21289-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskj2AJ_N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786290285951082-21289-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskj2AJ_N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786290285951082-21289-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 15:45:56.327339 21289 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 15:45:56.328617 21289 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 15:45:56.330473 21289 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 15:45:56.332743 21289 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 15:45:56.332928 21289 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 15:45:56.333144 21289 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 15:45:56.333292 21289 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 15:45:56.383471 21289 rpc_server.cc:307] RPC server started. Bound to: 127.20.202.68:34689
I20260809 15:45:56.383673 24513 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.202.68:34689 every 8 connection(s)
I20260809 15:45:56.395210 21289 auto_leader_rebalancer-test.cc:413] add a tserver: 61e69695258f48baa984c9d395e8ce3f
I20260809 15:45:56.413653 24516 heartbeater.cc:347] Connected to a master server at 127.20.202.126:43895
I20260809 15:45:56.414011 24516 heartbeater.cc:464] Registering TS with master...
I20260809 15:45:56.414616 24516 heartbeater.cc:511] Master 127.20.202.126:43895 requested a full tablet report, sending...
I20260809 15:45:56.416287 23117 ts_manager.cc:194] Registered new tserver with Master: 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:45:56.417843 23117 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:37326
I20260809 15:45:57.421203 24516 heartbeater.cc:503] Master 127.20.202.126:43895 was elected leader, sending a full tablet report...
I20260809 15:45:57.520382 23270 consensus_queue.cc:237] T 3296ef50be254fc29cfc77d22a8f7a23 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } attrs { replace: true } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } }
I20260809 15:45:57.526500 23527 raft_consensus.cc:1261] T 3296ef50be254fc29cfc77d22a8f7a23 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Refusing update from remote peer 67b3aeadfa7a432abe8af4d292e04ab9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 15:45:57.526650 23396 raft_consensus.cc:1261] T 3296ef50be254fc29cfc77d22a8f7a23 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Refusing update from remote peer 67b3aeadfa7a432abe8af4d292e04ab9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 15:45:57.528445 24528 consensus_queue.cc:1035] T 3296ef50be254fc29cfc77d22a8f7a23 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260809 15:45:57.530256 24346 consensus_queue.cc:1035] T 3296ef50be254fc29cfc77d22a8f7a23 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260809 15:45:57.544344 24534 raft_consensus.cc:2932] T 3296ef50be254fc29cfc77d22a8f7a23 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } attrs { replace: true } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } } }
I20260809 15:45:57.546137 23526 raft_consensus.cc:2932] T 3296ef50be254fc29cfc77d22a8f7a23 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } attrs { replace: true } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } } }
W20260809 15:45:57.549901 23223 consensus_peers.cc:487] T 3296ef50be254fc29cfc77d22a8f7a23 P 67b3aeadfa7a432abe8af4d292e04ab9 -> Peer 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689): Couldn't send request to peer 61e69695258f48baa984c9d395e8ce3f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3296ef50be254fc29cfc77d22a8f7a23. This is attempt 1: this message will repeat every 5th retry.
I20260809 15:45:57.550750 23396 raft_consensus.cc:2932] T 3296ef50be254fc29cfc77d22a8f7a23 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } attrs { replace: true } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } } }
I20260809 15:45:57.565794 23117 catalog_manager.cc:5495] T 3296ef50be254fc29cfc77d22a8f7a23 P 65c36d164b0540f099d592f38307ee6c reported cstate change: config changed from index -1 to 2, NON_VOTER 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) added. New cstate: current_term: 1 leader_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } attrs { replace: true } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } } }
I20260809 15:45:57.590468 23404 consensus_queue.cc:237] T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } }
I20260809 15:45:57.598215 23526 raft_consensus.cc:1261] T 3924b648461e463b9d7925e5671d68ae P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 15:45:57.598973 23270 raft_consensus.cc:1261] T 3924b648461e463b9d7925e5671d68ae P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 15:45:57.600898 24389 consensus_queue.cc:1035] T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260809 15:45:57.601891 24551 consensus_queue.cc:1035] T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260809 15:45:57.615936 24551 raft_consensus.cc:2932] T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } } }
I20260809 15:45:57.617631 23270 raft_consensus.cc:2932] T 3924b648461e463b9d7925e5671d68ae P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } } }
I20260809 15:45:57.620033 23527 raft_consensus.cc:2932] T 3924b648461e463b9d7925e5671d68ae P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } } }
I20260809 15:45:57.635946 23116 catalog_manager.cc:5495] T 3924b648461e463b9d7925e5671d68ae P 65c36d164b0540f099d592f38307ee6c reported cstate change: config changed from index -1 to 2, NON_VOTER 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) added. New cstate: current_term: 1 leader_uuid: "c0f38c1908414566aa9115bb2cbdd85d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } } }
W20260809 15:45:57.639550 23345 consensus_peers.cc:487] T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d -> Peer 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689): Couldn't send request to peer 61e69695258f48baa984c9d395e8ce3f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3924b648461e463b9d7925e5671d68ae. This is attempt 1: this message will repeat every 5th retry.
I20260809 15:45:57.647295 23527 consensus_queue.cc:237] T 294d4bcc1e7a477cb56455581277a60f P 65c36d164b0540f099d592f38307ee6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } attrs { replace: true } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } }
I20260809 15:45:57.653738 23404 raft_consensus.cc:1261] T 294d4bcc1e7a477cb56455581277a60f P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 15:45:57.655014 24529 consensus_queue.cc:1035] T 294d4bcc1e7a477cb56455581277a60f P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 15:45:57.655392 23270 raft_consensus.cc:1261] T 294d4bcc1e7a477cb56455581277a60f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 15:45:57.657202 24523 consensus_queue.cc:1035] T 294d4bcc1e7a477cb56455581277a60f P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 15:45:57.667462 24529 raft_consensus.cc:2932] T 294d4bcc1e7a477cb56455581277a60f P 65c36d164b0540f099d592f38307ee6c [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } attrs { replace: true } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } } }
I20260809 15:45:57.669514 23285 raft_consensus.cc:2932] T 294d4bcc1e7a477cb56455581277a60f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } attrs { replace: true } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } } }
I20260809 15:45:57.675371 23404 raft_consensus.cc:2932] T 294d4bcc1e7a477cb56455581277a60f P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } attrs { replace: true } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } } }
I20260809 15:45:57.684281 23394 consensus_queue.cc:237] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } attrs { replace: true } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } }
I20260809 15:45:57.689635 23117 catalog_manager.cc:5495] T 294d4bcc1e7a477cb56455581277a60f P 65c36d164b0540f099d592f38307ee6c reported cstate change: config changed from index -1 to 2, NON_VOTER 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) added. New cstate: current_term: 1 leader_uuid: "65c36d164b0540f099d592f38307ee6c" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20260809 15:45:57.695389 23468 consensus_peers.cc:487] T 294d4bcc1e7a477cb56455581277a60f P 65c36d164b0540f099d592f38307ee6c -> Peer 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689): Couldn't send request to peer 61e69695258f48baa984c9d395e8ce3f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 294d4bcc1e7a477cb56455581277a60f. This is attempt 1: this message will repeat every 5th retry.
W20260809 15:45:57.708132 23345 consensus_peers.cc:487] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P c0f38c1908414566aa9115bb2cbdd85d -> Peer 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689): Couldn't send request to peer 61e69695258f48baa984c9d395e8ce3f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1f9d6565aa4c45a3b3f01210d0eb71f7. This is attempt 1: this message will repeat every 5th retry.
I20260809 15:45:57.708297 23527 raft_consensus.cc:1261] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 15:45:57.709749 24531 consensus_queue.cc:1035] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 15:45:57.712142 23285 raft_consensus.cc:1261] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 15:45:57.713374 24389 consensus_queue.cc:1035] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 15:45:57.719779 24531 raft_consensus.cc:2932] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P c0f38c1908414566aa9115bb2cbdd85d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } attrs { replace: true } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } } }
I20260809 15:45:57.722218 23526 raft_consensus.cc:2932] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } attrs { replace: true } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } } }
I20260809 15:45:57.723783 23285 raft_consensus.cc:2932] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } attrs { replace: true } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } } }
I20260809 15:45:57.742141 23117 catalog_manager.cc:5495] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 65c36d164b0540f099d592f38307ee6c reported cstate change: config changed from index -1 to 2, NON_VOTER 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) added. New cstate: current_term: 1 leader_uuid: "c0f38c1908414566aa9115bb2cbdd85d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } attrs { replace: true } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } } }
I20260809 15:45:57.982061 24575 ts_tablet_manager.cc:927] T 3924b648461e463b9d7925e5671d68ae P 61e69695258f48baa984c9d395e8ce3f: Initiating tablet copy from peer c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:45:57.986047 24575 tablet_copy_client.cc:316] T 3924b648461e463b9d7925e5671d68ae P 61e69695258f48baa984c9d395e8ce3f: tablet copy: Beginning tablet copy session from remote peer at address 127.20.202.66:36263
I20260809 15:45:58.004539 23424 tablet_copy_service.cc:140] P c0f38c1908414566aa9115bb2cbdd85d: Received BeginTabletCopySession request for tablet 3924b648461e463b9d7925e5671d68ae from peer 61e69695258f48baa984c9d395e8ce3f ({username='slave'} at 127.0.0.1:52856)
I20260809 15:45:58.005123 23424 tablet_copy_service.cc:161] P c0f38c1908414566aa9115bb2cbdd85d: Beginning new tablet copy session on tablet 3924b648461e463b9d7925e5671d68ae from peer 61e69695258f48baa984c9d395e8ce3f at {username='slave'} at 127.0.0.1:52856: session id = 61e69695258f48baa984c9d395e8ce3f-3924b648461e463b9d7925e5671d68ae
I20260809 15:45:58.014823 23424 tablet_copy_source_session.cc:205] T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d: Tablet Copy: opened 0 blocks and 1 log segments
I20260809 15:45:58.018641 24575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3924b648461e463b9d7925e5671d68ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:58.032290 24575 tablet_copy_client.cc:791] T 3924b648461e463b9d7925e5671d68ae P 61e69695258f48baa984c9d395e8ce3f: tablet copy: Starting download of 0 data blocks...
I20260809 15:45:58.033030 24575 tablet_copy_client.cc:655] T 3924b648461e463b9d7925e5671d68ae P 61e69695258f48baa984c9d395e8ce3f: tablet copy: Starting download of 1 WAL segments...
I20260809 15:45:58.037504 24575 tablet_copy_client.cc:531] T 3924b648461e463b9d7925e5671d68ae P 61e69695258f48baa984c9d395e8ce3f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260809 15:45:58.043515 24575 tablet_bootstrap.cc:492] T 3924b648461e463b9d7925e5671d68ae P 61e69695258f48baa984c9d395e8ce3f: Bootstrap starting.
I20260809 15:45:58.062613 24575 tablet_bootstrap.cc:492] T 3924b648461e463b9d7925e5671d68ae P 61e69695258f48baa984c9d395e8ce3f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 15:45:58.063544 24575 tablet_bootstrap.cc:492] T 3924b648461e463b9d7925e5671d68ae P 61e69695258f48baa984c9d395e8ce3f: Bootstrap complete.
I20260809 15:45:58.064188 24575 ts_tablet_manager.cc:1397] T 3924b648461e463b9d7925e5671d68ae P 61e69695258f48baa984c9d395e8ce3f: Time spent bootstrapping tablet: real 0.021s user 0.019s sys 0.000s
I20260809 15:45:58.067636 24575 raft_consensus.cc:348] T 3924b648461e463b9d7925e5671d68ae P 61e69695258f48baa984c9d395e8ce3f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } }
I20260809 15:45:58.068547 24575 raft_consensus.cc:729] T 3924b648461e463b9d7925e5671d68ae P 61e69695258f48baa984c9d395e8ce3f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 61e69695258f48baa984c9d395e8ce3f, State: Initialized, Role: LEARNER
I20260809 15:45:58.069154 24575 consensus_queue.cc:260] T 3924b648461e463b9d7925e5671d68ae P 61e69695258f48baa984c9d395e8ce3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } }
I20260809 15:45:58.071470 24575 ts_tablet_manager.cc:1428] T 3924b648461e463b9d7925e5671d68ae P 61e69695258f48baa984c9d395e8ce3f: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260809 15:45:58.074046 23424 tablet_copy_service.cc:342] P c0f38c1908414566aa9115bb2cbdd85d: Request end of tablet copy session 61e69695258f48baa984c9d395e8ce3f-3924b648461e463b9d7925e5671d68ae received from {username='slave'} at 127.0.0.1:52856
I20260809 15:45:58.074429 23424 tablet_copy_service.cc:433] P c0f38c1908414566aa9115bb2cbdd85d: ending tablet copy session 61e69695258f48baa984c9d395e8ce3f-3924b648461e463b9d7925e5671d68ae on tablet 3924b648461e463b9d7925e5671d68ae with peer 61e69695258f48baa984c9d395e8ce3f
I20260809 15:45:58.086737 24575 ts_tablet_manager.cc:927] T 3296ef50be254fc29cfc77d22a8f7a23 P 61e69695258f48baa984c9d395e8ce3f: Initiating tablet copy from peer 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459)
I20260809 15:45:58.088326 24575 tablet_copy_client.cc:316] T 3296ef50be254fc29cfc77d22a8f7a23 P 61e69695258f48baa984c9d395e8ce3f: tablet copy: Beginning tablet copy session from remote peer at address 127.20.202.65:35459
I20260809 15:45:58.103535 23298 tablet_copy_service.cc:140] P 67b3aeadfa7a432abe8af4d292e04ab9: Received BeginTabletCopySession request for tablet 3296ef50be254fc29cfc77d22a8f7a23 from peer 61e69695258f48baa984c9d395e8ce3f ({username='slave'} at 127.0.0.1:55434)
I20260809 15:45:58.103946 23298 tablet_copy_service.cc:161] P 67b3aeadfa7a432abe8af4d292e04ab9: Beginning new tablet copy session on tablet 3296ef50be254fc29cfc77d22a8f7a23 from peer 61e69695258f48baa984c9d395e8ce3f at {username='slave'} at 127.0.0.1:55434: session id = 61e69695258f48baa984c9d395e8ce3f-3296ef50be254fc29cfc77d22a8f7a23
I20260809 15:45:58.109576 23298 tablet_copy_source_session.cc:205] T 3296ef50be254fc29cfc77d22a8f7a23 P 67b3aeadfa7a432abe8af4d292e04ab9: Tablet Copy: opened 0 blocks and 1 log segments
I20260809 15:45:58.113502 24575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3296ef50be254fc29cfc77d22a8f7a23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:58.126387 24575 tablet_copy_client.cc:791] T 3296ef50be254fc29cfc77d22a8f7a23 P 61e69695258f48baa984c9d395e8ce3f: tablet copy: Starting download of 0 data blocks...
I20260809 15:45:58.126865 24575 tablet_copy_client.cc:655] T 3296ef50be254fc29cfc77d22a8f7a23 P 61e69695258f48baa984c9d395e8ce3f: tablet copy: Starting download of 1 WAL segments...
I20260809 15:45:58.130388 24575 tablet_copy_client.cc:531] T 3296ef50be254fc29cfc77d22a8f7a23 P 61e69695258f48baa984c9d395e8ce3f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260809 15:45:58.136322 24575 tablet_bootstrap.cc:492] T 3296ef50be254fc29cfc77d22a8f7a23 P 61e69695258f48baa984c9d395e8ce3f: Bootstrap starting.
I20260809 15:45:58.155862 24575 tablet_bootstrap.cc:492] T 3296ef50be254fc29cfc77d22a8f7a23 P 61e69695258f48baa984c9d395e8ce3f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 15:45:58.156601 24575 tablet_bootstrap.cc:492] T 3296ef50be254fc29cfc77d22a8f7a23 P 61e69695258f48baa984c9d395e8ce3f: Bootstrap complete.
I20260809 15:45:58.157194 24575 ts_tablet_manager.cc:1397] T 3296ef50be254fc29cfc77d22a8f7a23 P 61e69695258f48baa984c9d395e8ce3f: Time spent bootstrapping tablet: real 0.021s user 0.018s sys 0.000s
I20260809 15:45:58.159464 24575 raft_consensus.cc:348] T 3296ef50be254fc29cfc77d22a8f7a23 P 61e69695258f48baa984c9d395e8ce3f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } attrs { replace: true } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } }
I20260809 15:45:58.160032 24575 raft_consensus.cc:729] T 3296ef50be254fc29cfc77d22a8f7a23 P 61e69695258f48baa984c9d395e8ce3f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 61e69695258f48baa984c9d395e8ce3f, State: Initialized, Role: LEARNER
I20260809 15:45:58.160547 24575 consensus_queue.cc:260] T 3296ef50be254fc29cfc77d22a8f7a23 P 61e69695258f48baa984c9d395e8ce3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } attrs { replace: true } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } }
I20260809 15:45:58.163017 24575 ts_tablet_manager.cc:1428] T 3296ef50be254fc29cfc77d22a8f7a23 P 61e69695258f48baa984c9d395e8ce3f: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260809 15:45:58.172025 23298 tablet_copy_service.cc:342] P 67b3aeadfa7a432abe8af4d292e04ab9: Request end of tablet copy session 61e69695258f48baa984c9d395e8ce3f-3296ef50be254fc29cfc77d22a8f7a23 received from {username='slave'} at 127.0.0.1:55434
I20260809 15:45:58.172394 23298 tablet_copy_service.cc:433] P 67b3aeadfa7a432abe8af4d292e04ab9: ending tablet copy session 61e69695258f48baa984c9d395e8ce3f-3296ef50be254fc29cfc77d22a8f7a23 on tablet 3296ef50be254fc29cfc77d22a8f7a23 with peer 61e69695258f48baa984c9d395e8ce3f
I20260809 15:45:58.177678 24583 ts_tablet_manager.cc:927] T 294d4bcc1e7a477cb56455581277a60f P 61e69695258f48baa984c9d395e8ce3f: Initiating tablet copy from peer 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:45:58.180282 24583 tablet_copy_client.cc:316] T 294d4bcc1e7a477cb56455581277a60f P 61e69695258f48baa984c9d395e8ce3f: tablet copy: Beginning tablet copy session from remote peer at address 127.20.202.67:40553
I20260809 15:45:58.196189 23546 tablet_copy_service.cc:140] P 65c36d164b0540f099d592f38307ee6c: Received BeginTabletCopySession request for tablet 294d4bcc1e7a477cb56455581277a60f from peer 61e69695258f48baa984c9d395e8ce3f ({username='slave'} at 127.0.0.1:44018)
I20260809 15:45:58.196655 23546 tablet_copy_service.cc:161] P 65c36d164b0540f099d592f38307ee6c: Beginning new tablet copy session on tablet 294d4bcc1e7a477cb56455581277a60f from peer 61e69695258f48baa984c9d395e8ce3f at {username='slave'} at 127.0.0.1:44018: session id = 61e69695258f48baa984c9d395e8ce3f-294d4bcc1e7a477cb56455581277a60f
I20260809 15:45:58.202250 23546 tablet_copy_source_session.cc:205] T 294d4bcc1e7a477cb56455581277a60f P 65c36d164b0540f099d592f38307ee6c: Tablet Copy: opened 0 blocks and 1 log segments
I20260809 15:45:58.204602 24583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 294d4bcc1e7a477cb56455581277a60f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:58.216440 24583 tablet_copy_client.cc:791] T 294d4bcc1e7a477cb56455581277a60f P 61e69695258f48baa984c9d395e8ce3f: tablet copy: Starting download of 0 data blocks...
I20260809 15:45:58.216917 24583 tablet_copy_client.cc:655] T 294d4bcc1e7a477cb56455581277a60f P 61e69695258f48baa984c9d395e8ce3f: tablet copy: Starting download of 1 WAL segments...
I20260809 15:45:58.219681 24583 tablet_copy_client.cc:531] T 294d4bcc1e7a477cb56455581277a60f P 61e69695258f48baa984c9d395e8ce3f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260809 15:45:58.225488 24583 tablet_bootstrap.cc:492] T 294d4bcc1e7a477cb56455581277a60f P 61e69695258f48baa984c9d395e8ce3f: Bootstrap starting.
I20260809 15:45:58.248394 24583 tablet_bootstrap.cc:492] T 294d4bcc1e7a477cb56455581277a60f P 61e69695258f48baa984c9d395e8ce3f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 15:45:58.249035 24583 tablet_bootstrap.cc:492] T 294d4bcc1e7a477cb56455581277a60f P 61e69695258f48baa984c9d395e8ce3f: Bootstrap complete.
I20260809 15:45:58.249814 24583 ts_tablet_manager.cc:1397] T 294d4bcc1e7a477cb56455581277a60f P 61e69695258f48baa984c9d395e8ce3f: Time spent bootstrapping tablet: real 0.024s user 0.015s sys 0.010s
I20260809 15:45:58.252205 24583 raft_consensus.cc:348] T 294d4bcc1e7a477cb56455581277a60f P 61e69695258f48baa984c9d395e8ce3f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } attrs { replace: true } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } }
I20260809 15:45:58.253059 24583 raft_consensus.cc:729] T 294d4bcc1e7a477cb56455581277a60f P 61e69695258f48baa984c9d395e8ce3f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 61e69695258f48baa984c9d395e8ce3f, State: Initialized, Role: LEARNER
I20260809 15:45:58.253696 24583 consensus_queue.cc:260] T 294d4bcc1e7a477cb56455581277a60f P 61e69695258f48baa984c9d395e8ce3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } attrs { replace: true } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } }
I20260809 15:45:58.255687 24583 ts_tablet_manager.cc:1428] T 294d4bcc1e7a477cb56455581277a60f P 61e69695258f48baa984c9d395e8ce3f: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260809 15:45:58.257834 23546 tablet_copy_service.cc:342] P 65c36d164b0540f099d592f38307ee6c: Request end of tablet copy session 61e69695258f48baa984c9d395e8ce3f-294d4bcc1e7a477cb56455581277a60f received from {username='slave'} at 127.0.0.1:44018
I20260809 15:45:58.258208 23546 tablet_copy_service.cc:433] P 65c36d164b0540f099d592f38307ee6c: ending tablet copy session 61e69695258f48baa984c9d395e8ce3f-294d4bcc1e7a477cb56455581277a60f on tablet 294d4bcc1e7a477cb56455581277a60f with peer 61e69695258f48baa984c9d395e8ce3f
I20260809 15:45:58.320820 24583 ts_tablet_manager.cc:927] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 61e69695258f48baa984c9d395e8ce3f: Initiating tablet copy from peer c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:45:58.322400 24583 tablet_copy_client.cc:316] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 61e69695258f48baa984c9d395e8ce3f: tablet copy: Beginning tablet copy session from remote peer at address 127.20.202.66:36263
I20260809 15:45:58.323669 23424 tablet_copy_service.cc:140] P c0f38c1908414566aa9115bb2cbdd85d: Received BeginTabletCopySession request for tablet 1f9d6565aa4c45a3b3f01210d0eb71f7 from peer 61e69695258f48baa984c9d395e8ce3f ({username='slave'} at 127.0.0.1:52856)
I20260809 15:45:58.324177 23424 tablet_copy_service.cc:161] P c0f38c1908414566aa9115bb2cbdd85d: Beginning new tablet copy session on tablet 1f9d6565aa4c45a3b3f01210d0eb71f7 from peer 61e69695258f48baa984c9d395e8ce3f at {username='slave'} at 127.0.0.1:52856: session id = 61e69695258f48baa984c9d395e8ce3f-1f9d6565aa4c45a3b3f01210d0eb71f7
I20260809 15:45:58.329741 23424 tablet_copy_source_session.cc:205] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P c0f38c1908414566aa9115bb2cbdd85d: Tablet Copy: opened 0 blocks and 1 log segments
I20260809 15:45:58.332407 24583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f9d6565aa4c45a3b3f01210d0eb71f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:45:58.344362 24583 tablet_copy_client.cc:791] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 61e69695258f48baa984c9d395e8ce3f: tablet copy: Starting download of 0 data blocks...
I20260809 15:45:58.344869 24583 tablet_copy_client.cc:655] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 61e69695258f48baa984c9d395e8ce3f: tablet copy: Starting download of 1 WAL segments...
I20260809 15:45:58.347748 24583 tablet_copy_client.cc:531] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 61e69695258f48baa984c9d395e8ce3f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260809 15:45:58.354113 24583 tablet_bootstrap.cc:492] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 61e69695258f48baa984c9d395e8ce3f: Bootstrap starting.
I20260809 15:45:58.380470 24583 tablet_bootstrap.cc:492] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 61e69695258f48baa984c9d395e8ce3f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 15:45:58.381134 24583 tablet_bootstrap.cc:492] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 61e69695258f48baa984c9d395e8ce3f: Bootstrap complete.
I20260809 15:45:58.381711 24583 ts_tablet_manager.cc:1397] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 61e69695258f48baa984c9d395e8ce3f: Time spent bootstrapping tablet: real 0.028s user 0.020s sys 0.001s
I20260809 15:45:58.384912 24583 raft_consensus.cc:348] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 61e69695258f48baa984c9d395e8ce3f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } attrs { replace: true } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } }
I20260809 15:45:58.385603 24583 raft_consensus.cc:729] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 61e69695258f48baa984c9d395e8ce3f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 61e69695258f48baa984c9d395e8ce3f, State: Initialized, Role: LEARNER
I20260809 15:45:58.386147 24583 consensus_queue.cc:260] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 61e69695258f48baa984c9d395e8ce3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } attrs { replace: true } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } }
I20260809 15:45:58.388185 24583 ts_tablet_manager.cc:1428] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 61e69695258f48baa984c9d395e8ce3f: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260809 15:45:58.389745 23424 tablet_copy_service.cc:342] P c0f38c1908414566aa9115bb2cbdd85d: Request end of tablet copy session 61e69695258f48baa984c9d395e8ce3f-1f9d6565aa4c45a3b3f01210d0eb71f7 received from {username='slave'} at 127.0.0.1:52856
I20260809 15:45:58.390110 23424 tablet_copy_service.cc:433] P c0f38c1908414566aa9115bb2cbdd85d: ending tablet copy session 61e69695258f48baa984c9d395e8ce3f-1f9d6565aa4c45a3b3f01210d0eb71f7 on tablet 1f9d6565aa4c45a3b3f01210d0eb71f7 with peer 61e69695258f48baa984c9d395e8ce3f
I20260809 15:45:58.409004 24475 raft_consensus.cc:1203] T 3924b648461e463b9d7925e5671d68ae P 61e69695258f48baa984c9d395e8ce3f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260809 15:45:58.563481 24475 raft_consensus.cc:1203] T 3296ef50be254fc29cfc77d22a8f7a23 P 61e69695258f48baa984c9d395e8ce3f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260809 15:45:58.584853 24475 raft_consensus.cc:1203] T 294d4bcc1e7a477cb56455581277a60f P 61e69695258f48baa984c9d395e8ce3f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260809 15:45:58.865177 24475 raft_consensus.cc:1203] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 61e69695258f48baa984c9d395e8ce3f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260809 15:45:58.962476 24598 raft_consensus.cc:1050] T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d: attempting to promote NON_VOTER 61e69695258f48baa984c9d395e8ce3f to VOTER
I20260809 15:45:58.965030 24598 consensus_queue.cc:237] T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:45:58.973511 24475 raft_consensus.cc:1261] T 3924b648461e463b9d7925e5671d68ae P 61e69695258f48baa984c9d395e8ce3f [term 1 LEARNER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 15:45:58.978122 23526 raft_consensus.cc:1261] T 3924b648461e463b9d7925e5671d68ae P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 15:45:58.978158 24531 consensus_queue.cc:1035] T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260809 15:45:58.979043 23285 raft_consensus.cc:1261] T 3924b648461e463b9d7925e5671d68ae P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 15:45:58.980930 24595 consensus_queue.cc:1035] T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 15:45:58.983803 24389 consensus_queue.cc:1035] T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 15:45:59.001368 24595 raft_consensus.cc:2932] T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:45:59.002643 23527 raft_consensus.cc:2932] T 3924b648461e463b9d7925e5671d68ae P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:45:59.018033 23117 catalog_manager.cc:5495] T 3924b648461e463b9d7925e5671d68ae P 65c36d164b0540f099d592f38307ee6c reported cstate change: config changed from index 2 to 3, 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c0f38c1908414566aa9115bb2cbdd85d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:45:59.018893 23270 raft_consensus.cc:2932] T 3924b648461e463b9d7925e5671d68ae P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:45:59.041286 24603 raft_consensus.cc:1050] T 294d4bcc1e7a477cb56455581277a60f P 65c36d164b0540f099d592f38307ee6c: attempting to promote NON_VOTER 61e69695258f48baa984c9d395e8ce3f to VOTER
I20260809 15:45:59.043385 24603 consensus_queue.cc:237] T 294d4bcc1e7a477cb56455581277a60f P 65c36d164b0540f099d592f38307ee6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } attrs { replace: true } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:45:59.044443 24475 raft_consensus.cc:2932] T 3924b648461e463b9d7925e5671d68ae P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:45:59.050649 24472 raft_consensus.cc:1261] T 294d4bcc1e7a477cb56455581277a60f P 61e69695258f48baa984c9d395e8ce3f [term 1 LEARNER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 15:45:59.051086 23404 raft_consensus.cc:1261] T 294d4bcc1e7a477cb56455581277a60f P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 15:45:59.052350 24603 consensus_queue.cc:1035] T 294d4bcc1e7a477cb56455581277a60f P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 15:45:59.053272 24529 consensus_queue.cc:1035] T 294d4bcc1e7a477cb56455581277a60f P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 15:45:59.054823 23285 raft_consensus.cc:1261] T 294d4bcc1e7a477cb56455581277a60f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 15:45:59.056087 24529 consensus_queue.cc:1035] T 294d4bcc1e7a477cb56455581277a60f P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 15:45:59.085566 24529 raft_consensus.cc:2932] T 294d4bcc1e7a477cb56455581277a60f P 65c36d164b0540f099d592f38307ee6c [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } attrs { replace: true } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:45:59.088521 23404 raft_consensus.cc:2932] T 294d4bcc1e7a477cb56455581277a60f P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } attrs { replace: true } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:45:59.092682 23394 consensus_queue.cc:237] T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:45:59.098975 23529 raft_consensus.cc:1261] T 3924b648461e463b9d7925e5671d68ae P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260809 15:45:59.100267 24595 consensus_queue.cc:1035] T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260809 15:45:59.101744 24475 raft_consensus.cc:1261] T 3924b648461e463b9d7925e5671d68ae P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260809 15:45:59.104818 24472 raft_consensus.cc:2932] T 294d4bcc1e7a477cb56455581277a60f P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } attrs { replace: true } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:45:59.108856 24531 consensus_queue.cc:1035] T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260809 15:45:59.116546 23277 raft_consensus.cc:2932] T 294d4bcc1e7a477cb56455581277a60f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } attrs { replace: true } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:45:59.118387 24389 raft_consensus.cc:2932] T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:45:59.120497 23118 catalog_manager.cc:5495] T 294d4bcc1e7a477cb56455581277a60f P 65c36d164b0540f099d592f38307ee6c reported cstate change: config changed from index 2 to 3, 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "65c36d164b0540f099d592f38307ee6c" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 15:45:59.121518 23527 raft_consensus.cc:2932] T 3924b648461e463b9d7925e5671d68ae P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:45:59.127074 23095 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 3924b648461e463b9d7925e5671d68ae with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260809 15:45:59.137856 24472 raft_consensus.cc:2932] T 3924b648461e463b9d7925e5671d68ae P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:45:59.150923 23115 catalog_manager.cc:5495] T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d reported cstate change: config changed from index 3 to 4, VOTER 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65) evicted. New cstate: current_term: 1 leader_uuid: "c0f38c1908414566aa9115bb2cbdd85d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 15:45:59.156222 24346 raft_consensus.cc:1050] T 3296ef50be254fc29cfc77d22a8f7a23 P 67b3aeadfa7a432abe8af4d292e04ab9: attempting to promote NON_VOTER 61e69695258f48baa984c9d395e8ce3f to VOTER
I20260809 15:45:59.158216 24346 consensus_queue.cc:237] T 3296ef50be254fc29cfc77d22a8f7a23 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } attrs { replace: true } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:45:59.169718 23260 tablet_service.cc:1514] Processing DeleteTablet for tablet 3924b648461e463b9d7925e5671d68ae with delete_type TABLET_DATA_TOMBSTONED (TS 67b3aeadfa7a432abe8af4d292e04ab9 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:53552
I20260809 15:45:59.172006 24629 tablet_replica.cc:330] T 3924b648461e463b9d7925e5671d68ae P 67b3aeadfa7a432abe8af4d292e04ab9: stopping tablet replica
I20260809 15:45:59.173221 24629 raft_consensus.cc:2229] T 3924b648461e463b9d7925e5671d68ae P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 15:45:59.173954 24629 raft_consensus.cc:2258] T 3924b648461e463b9d7925e5671d68ae P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 15:45:59.177346 23526 consensus_queue.cc:237] T 294d4bcc1e7a477cb56455581277a60f P 65c36d164b0540f099d592f38307ee6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:45:59.178282 23527 raft_consensus.cc:1261] T 3296ef50be254fc29cfc77d22a8f7a23 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Refusing update from remote peer 67b3aeadfa7a432abe8af4d292e04ab9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 15:45:59.179602 24528 consensus_queue.cc:1035] T 3296ef50be254fc29cfc77d22a8f7a23 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 15:45:59.180052 24629 ts_tablet_manager.cc:1905] T 3924b648461e463b9d7925e5671d68ae P 67b3aeadfa7a432abe8af4d292e04ab9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260809 15:45:59.182919 24472 raft_consensus.cc:1261] T 3296ef50be254fc29cfc77d22a8f7a23 P 61e69695258f48baa984c9d395e8ce3f [term 1 LEARNER]: Refusing update from remote peer 67b3aeadfa7a432abe8af4d292e04ab9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 15:45:59.184099 23404 raft_consensus.cc:1261] T 3296ef50be254fc29cfc77d22a8f7a23 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Refusing update from remote peer 67b3aeadfa7a432abe8af4d292e04ab9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 15:45:59.200902 23277 raft_consensus.cc:1261] T 294d4bcc1e7a477cb56455581277a60f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260809 15:45:59.202970 24625 consensus_queue.cc:1035] T 294d4bcc1e7a477cb56455581277a60f P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260809 15:45:59.203902 24528 consensus_queue.cc:1035] T 3296ef50be254fc29cfc77d22a8f7a23 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 15:45:59.204882 24629 ts_tablet_manager.cc:1918] T 3924b648461e463b9d7925e5671d68ae P 67b3aeadfa7a432abe8af4d292e04ab9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260809 15:45:59.205333 24629 log.cc:1196] T 3924b648461e463b9d7925e5671d68ae P 67b3aeadfa7a432abe8af4d292e04ab9: Deleting WAL directory at /tmp/dist-test-taskj2AJ_N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786290285951082-21289-0/minicluster-data/ts-0-root/wals/3924b648461e463b9d7925e5671d68ae
I20260809 15:45:59.206540 24528 consensus_queue.cc:1035] T 3296ef50be254fc29cfc77d22a8f7a23 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 15:45:59.210052 24603 raft_consensus.cc:2932] T 294d4bcc1e7a477cb56455581277a60f P 65c36d164b0540f099d592f38307ee6c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:45:59.212034 24472 raft_consensus.cc:1261] T 294d4bcc1e7a477cb56455581277a60f P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260809 15:45:59.210474 23094 catalog_manager.cc:4841] TS 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459): tablet 3924b648461e463b9d7925e5671d68ae (table test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]) successfully deleted
I20260809 15:45:59.213155 23277 raft_consensus.cc:2932] T 294d4bcc1e7a477cb56455581277a60f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:45:59.214955 24625 consensus_queue.cc:1035] T 294d4bcc1e7a477cb56455581277a60f P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260809 15:45:59.218202 24472 raft_consensus.cc:2932] T 294d4bcc1e7a477cb56455581277a60f P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:45:59.225952 24600 raft_consensus.cc:2932] T 3296ef50be254fc29cfc77d22a8f7a23 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } attrs { replace: true } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:45:59.230587 23118 catalog_manager.cc:5495] T 294d4bcc1e7a477cb56455581277a60f P 61e69695258f48baa984c9d395e8ce3f reported cstate change: config changed from index 3 to 4, VOTER c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66) evicted. New cstate: current_term: 1 leader_uuid: "65c36d164b0540f099d592f38307ee6c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:45:59.233042 23521 raft_consensus.cc:2932] T 3296ef50be254fc29cfc77d22a8f7a23 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } attrs { replace: true } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:45:59.242443 23389 tablet_service.cc:1514] Processing DeleteTablet for tablet 294d4bcc1e7a477cb56455581277a60f with delete_type TABLET_DATA_TOMBSTONED (TS c0f38c1908414566aa9115bb2cbdd85d not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:55312
I20260809 15:45:59.250813 23095 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 294d4bcc1e7a477cb56455581277a60f with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260809 15:45:59.255352 23118 catalog_manager.cc:5495] T 3296ef50be254fc29cfc77d22a8f7a23 P 65c36d164b0540f099d592f38307ee6c reported cstate change: config changed from index 2 to 3, 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } attrs { replace: true } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:45:59.271481 23396 raft_consensus.cc:2932] T 3296ef50be254fc29cfc77d22a8f7a23 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } attrs { replace: true } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:45:59.283327 24472 raft_consensus.cc:2932] T 3296ef50be254fc29cfc77d22a8f7a23 P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } attrs { replace: true } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:45:59.287508 24635 tablet_replica.cc:330] T 294d4bcc1e7a477cb56455581277a60f P c0f38c1908414566aa9115bb2cbdd85d: stopping tablet replica
I20260809 15:45:59.288535 24635 raft_consensus.cc:2229] T 294d4bcc1e7a477cb56455581277a60f P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 15:45:59.289070 24635 raft_consensus.cc:2258] T 294d4bcc1e7a477cb56455581277a60f P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 15:45:59.311710 24635 ts_tablet_manager.cc:1905] T 294d4bcc1e7a477cb56455581277a60f P c0f38c1908414566aa9115bb2cbdd85d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260809 15:45:59.312069 23285 consensus_queue.cc:237] T 3296ef50be254fc29cfc77d22a8f7a23 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:45:59.317718 23396 raft_consensus.cc:1261] T 3296ef50be254fc29cfc77d22a8f7a23 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Refusing update from remote peer 67b3aeadfa7a432abe8af4d292e04ab9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260809 15:45:59.318817 24472 raft_consensus.cc:1261] T 3296ef50be254fc29cfc77d22a8f7a23 P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Refusing update from remote peer 67b3aeadfa7a432abe8af4d292e04ab9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260809 15:45:59.319236 24639 consensus_queue.cc:1035] T 3296ef50be254fc29cfc77d22a8f7a23 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260809 15:45:59.320492 24633 consensus_queue.cc:1035] T 3296ef50be254fc29cfc77d22a8f7a23 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260809 15:45:59.327206 24633 raft_consensus.cc:2932] T 3296ef50be254fc29cfc77d22a8f7a23 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 65c36d164b0540f099d592f38307ee6c (127.20.202.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:45:59.332353 23404 raft_consensus.cc:2932] T 3296ef50be254fc29cfc77d22a8f7a23 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 65c36d164b0540f099d592f38307ee6c (127.20.202.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:45:59.334406 24472 raft_consensus.cc:2932] T 3296ef50be254fc29cfc77d22a8f7a23 P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 65c36d164b0540f099d592f38307ee6c (127.20.202.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:45:59.337760 23094 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 3296ef50be254fc29cfc77d22a8f7a23 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260809 15:45:59.339359 24635 ts_tablet_manager.cc:1918] T 294d4bcc1e7a477cb56455581277a60f P c0f38c1908414566aa9115bb2cbdd85d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260809 15:45:59.339799 24635 log.cc:1196] T 294d4bcc1e7a477cb56455581277a60f P c0f38c1908414566aa9115bb2cbdd85d: Deleting WAL directory at /tmp/dist-test-taskj2AJ_N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786290285951082-21289-0/minicluster-data/ts-1-root/wals/294d4bcc1e7a477cb56455581277a60f
I20260809 15:45:59.342375 23095 catalog_manager.cc:4841] TS c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263): tablet 294d4bcc1e7a477cb56455581277a60f (table test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]) successfully deleted
I20260809 15:45:59.351684 23118 catalog_manager.cc:5495] T 3296ef50be254fc29cfc77d22a8f7a23 P 67b3aeadfa7a432abe8af4d292e04ab9 reported cstate change: config changed from index 3 to 4, VOTER 65c36d164b0540f099d592f38307ee6c (127.20.202.67) evicted. New cstate: current_term: 1 leader_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 15:45:59.362955 23500 tablet_service.cc:1514] Processing DeleteTablet for tablet 3296ef50be254fc29cfc77d22a8f7a23 with delete_type TABLET_DATA_TOMBSTONED (TS 65c36d164b0540f099d592f38307ee6c not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:56760
I20260809 15:45:59.366460 24640 tablet_replica.cc:330] T 3296ef50be254fc29cfc77d22a8f7a23 P 65c36d164b0540f099d592f38307ee6c: stopping tablet replica
I20260809 15:45:59.367323 24640 raft_consensus.cc:2229] T 3296ef50be254fc29cfc77d22a8f7a23 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 15:45:59.367851 24640 raft_consensus.cc:2258] T 3296ef50be254fc29cfc77d22a8f7a23 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 15:45:59.370144 24640 ts_tablet_manager.cc:1905] T 3296ef50be254fc29cfc77d22a8f7a23 P 65c36d164b0540f099d592f38307ee6c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260809 15:45:59.399971 24640 ts_tablet_manager.cc:1918] T 3296ef50be254fc29cfc77d22a8f7a23 P 65c36d164b0540f099d592f38307ee6c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260809 15:45:59.400336 24640 log.cc:1196] T 3296ef50be254fc29cfc77d22a8f7a23 P 65c36d164b0540f099d592f38307ee6c: Deleting WAL directory at /tmp/dist-test-taskj2AJ_N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786290285951082-21289-0/minicluster-data/ts-2-root/wals/3296ef50be254fc29cfc77d22a8f7a23
I20260809 15:45:59.401732 23095 catalog_manager.cc:4841] TS 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553): tablet 3296ef50be254fc29cfc77d22a8f7a23 (table test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]) successfully deleted
I20260809 15:45:59.461349 24389 raft_consensus.cc:1050] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P c0f38c1908414566aa9115bb2cbdd85d: attempting to promote NON_VOTER 61e69695258f48baa984c9d395e8ce3f to VOTER
I20260809 15:45:59.462987 24389 consensus_queue.cc:237] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } attrs { replace: true } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:45:59.467393 24472 raft_consensus.cc:1261] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 61e69695258f48baa984c9d395e8ce3f [term 1 LEARNER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 15:45:59.468037 23529 raft_consensus.cc:1261] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 15:45:59.468523 24389 consensus_queue.cc:1035] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 15:45:59.469125 23277 raft_consensus.cc:1261] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 15:45:59.470410 24595 consensus_queue.cc:1035] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 15:45:59.472697 24531 consensus_queue.cc:1035] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 15:45:59.490264 24389 raft_consensus.cc:2932] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P c0f38c1908414566aa9115bb2cbdd85d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } attrs { replace: true } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:45:59.495579 23277 raft_consensus.cc:2932] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } attrs { replace: true } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:45:59.501530 23529 raft_consensus.cc:2932] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } attrs { replace: true } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:45:59.503896 24472 raft_consensus.cc:2932] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } attrs { replace: true } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:45:59.512801 23115 catalog_manager.cc:5495] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 67b3aeadfa7a432abe8af4d292e04ab9 reported cstate change: config changed from index 2 to 3, 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c0f38c1908414566aa9115bb2cbdd85d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } attrs { replace: true } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:45:59.533725 23397 consensus_queue.cc:237] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:45:59.538307 23277 raft_consensus.cc:1261] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260809 15:45:59.539805 24472 raft_consensus.cc:1261] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260809 15:45:59.540884 24606 consensus_queue.cc:1035] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260809 15:45:59.542649 24389 consensus_queue.cc:1035] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260809 15:45:59.550156 24389 raft_consensus.cc:2932] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P c0f38c1908414566aa9115bb2cbdd85d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 65c36d164b0540f099d592f38307ee6c (127.20.202.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:45:59.551841 23274 raft_consensus.cc:2932] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 65c36d164b0540f099d592f38307ee6c (127.20.202.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:45:59.553864 24472 raft_consensus.cc:2932] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 65c36d164b0540f099d592f38307ee6c (127.20.202.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:45:59.561419 23095 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 1f9d6565aa4c45a3b3f01210d0eb71f7 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260809 15:45:59.564393 23116 catalog_manager.cc:5495] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 61e69695258f48baa984c9d395e8ce3f reported cstate change: config changed from index 3 to 4, VOTER 65c36d164b0540f099d592f38307ee6c (127.20.202.67) evicted. New cstate: current_term: 1 leader_uuid: "c0f38c1908414566aa9115bb2cbdd85d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:45:59.581175 23500 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f9d6565aa4c45a3b3f01210d0eb71f7 with delete_type TABLET_DATA_TOMBSTONED (TS 65c36d164b0540f099d592f38307ee6c not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:56760
I20260809 15:45:59.582010 24640 tablet_replica.cc:330] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 65c36d164b0540f099d592f38307ee6c: stopping tablet replica
I20260809 15:45:59.586709 24640 raft_consensus.cc:2229] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 15:45:59.587327 24640 raft_consensus.cc:2258] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 15:45:59.590426 24640 ts_tablet_manager.cc:1905] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 65c36d164b0540f099d592f38307ee6c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260809 15:45:59.601092 24640 ts_tablet_manager.cc:1918] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 65c36d164b0540f099d592f38307ee6c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260809 15:45:59.601356 24640 log.cc:1196] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 65c36d164b0540f099d592f38307ee6c: Deleting WAL directory at /tmp/dist-test-taskj2AJ_N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786290285951082-21289-0/minicluster-data/ts-2-root/wals/1f9d6565aa4c45a3b3f01210d0eb71f7
I20260809 15:45:59.602375 23095 catalog_manager.cc:4841] TS 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553): tablet 1f9d6565aa4c45a3b3f01210d0eb71f7 (table test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]) successfully deleted
I20260809 15:46:00.627712 23274 consensus_queue.cc:237] T 0c48a3049897452383257aa757834fb5 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } attrs { replace: true } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } }
W20260809 15:46:00.634426 23223 consensus_peers.cc:487] T 0c48a3049897452383257aa757834fb5 P 67b3aeadfa7a432abe8af4d292e04ab9 -> Peer 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689): Couldn't send request to peer 61e69695258f48baa984c9d395e8ce3f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0c48a3049897452383257aa757834fb5. This is attempt 1: this message will repeat every 5th retry.
I20260809 15:46:00.634824 23521 raft_consensus.cc:1261] T 0c48a3049897452383257aa757834fb5 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Refusing update from remote peer 67b3aeadfa7a432abe8af4d292e04ab9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 15:46:00.636411 23396 raft_consensus.cc:1261] T 0c48a3049897452383257aa757834fb5 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Refusing update from remote peer 67b3aeadfa7a432abe8af4d292e04ab9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 15:46:00.636482 24652 consensus_queue.cc:1035] T 0c48a3049897452383257aa757834fb5 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260809 15:46:00.637567 24633 consensus_queue.cc:1035] T 0c48a3049897452383257aa757834fb5 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 15:46:00.645258 24652 raft_consensus.cc:2932] T 0c48a3049897452383257aa757834fb5 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } attrs { replace: true } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } } }
I20260809 15:46:00.646634 23529 raft_consensus.cc:2932] T 0c48a3049897452383257aa757834fb5 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } attrs { replace: true } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } } }
I20260809 15:46:00.648639 23396 raft_consensus.cc:2932] T 0c48a3049897452383257aa757834fb5 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } attrs { replace: true } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } } }
I20260809 15:46:00.659099 23397 consensus_queue.cc:237] T 4bcac4b24e7a4448abdc934bba5a624e P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } }
I20260809 15:46:00.660377 23118 catalog_manager.cc:5495] T 0c48a3049897452383257aa757834fb5 P 65c36d164b0540f099d592f38307ee6c reported cstate change: config changed from index -1 to 2, NON_VOTER 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) added. New cstate: current_term: 1 leader_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } attrs { replace: true } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } } }
W20260809 15:46:00.667311 23345 consensus_peers.cc:487] T 4bcac4b24e7a4448abdc934bba5a624e P c0f38c1908414566aa9115bb2cbdd85d -> Peer 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689): Couldn't send request to peer 61e69695258f48baa984c9d395e8ce3f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4bcac4b24e7a4448abdc934bba5a624e. This is attempt 1: this message will repeat every 5th retry.
I20260809 15:46:00.667325 23274 raft_consensus.cc:1261] T 4bcac4b24e7a4448abdc934bba5a624e P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 15:46:00.667882 23521 raft_consensus.cc:1261] T 4bcac4b24e7a4448abdc934bba5a624e P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 15:46:00.669282 24606 consensus_queue.cc:1035] T 4bcac4b24e7a4448abdc934bba5a624e P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 15:46:00.670213 24531 consensus_queue.cc:1035] T 4bcac4b24e7a4448abdc934bba5a624e P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 15:46:00.692961 24531 raft_consensus.cc:2932] T 4bcac4b24e7a4448abdc934bba5a624e P c0f38c1908414566aa9115bb2cbdd85d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } } }
I20260809 15:46:00.694454 23285 raft_consensus.cc:2932] T 4bcac4b24e7a4448abdc934bba5a624e P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } } }
I20260809 15:46:00.709136 23396 consensus_queue.cc:237] T d3be1e356b254d22b995be32f1c12f6f P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } attrs { replace: true } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } }
I20260809 15:46:00.706658 23521 raft_consensus.cc:2932] T 4bcac4b24e7a4448abdc934bba5a624e P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } } }
I20260809 15:46:00.721777 23117 catalog_manager.cc:5495] T 4bcac4b24e7a4448abdc934bba5a624e P c0f38c1908414566aa9115bb2cbdd85d reported cstate change: config changed from index -1 to 2, NON_VOTER 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) added. New cstate: current_term: 1 leader_uuid: "c0f38c1908414566aa9115bb2cbdd85d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20260809 15:46:00.740757 23345 consensus_peers.cc:487] T d3be1e356b254d22b995be32f1c12f6f P c0f38c1908414566aa9115bb2cbdd85d -> Peer 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689): Couldn't send request to peer 61e69695258f48baa984c9d395e8ce3f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d3be1e356b254d22b995be32f1c12f6f. This is attempt 1: this message will repeat every 5th retry.
I20260809 15:46:00.742983 23521 raft_consensus.cc:1261] T d3be1e356b254d22b995be32f1c12f6f P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 15:46:00.744354 23270 raft_consensus.cc:1261] T d3be1e356b254d22b995be32f1c12f6f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 15:46:00.745040 24531 consensus_queue.cc:1035] T d3be1e356b254d22b995be32f1c12f6f P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 15:46:00.747049 24531 consensus_queue.cc:1035] T d3be1e356b254d22b995be32f1c12f6f P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 15:46:00.753629 24664 raft_consensus.cc:2932] T d3be1e356b254d22b995be32f1c12f6f P c0f38c1908414566aa9115bb2cbdd85d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } attrs { replace: true } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } } }
I20260809 15:46:00.754884 23529 raft_consensus.cc:2932] T d3be1e356b254d22b995be32f1c12f6f P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } attrs { replace: true } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } } }
I20260809 15:46:00.758227 23270 raft_consensus.cc:2932] T d3be1e356b254d22b995be32f1c12f6f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } attrs { replace: true } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } } }
I20260809 15:46:00.768467 23396 consensus_queue.cc:237] T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } }
I20260809 15:46:00.770687 23117 catalog_manager.cc:5495] T d3be1e356b254d22b995be32f1c12f6f P 65c36d164b0540f099d592f38307ee6c reported cstate change: config changed from index -1 to 2, NON_VOTER 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) added. New cstate: current_term: 1 leader_uuid: "c0f38c1908414566aa9115bb2cbdd85d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } attrs { replace: true } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } } }
I20260809 15:46:00.776821 23529 raft_consensus.cc:1261] T d3e94b7e50754a81b3d72a07476cb7b9 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260809 15:46:00.776885 23345 consensus_peers.cc:487] T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d -> Peer 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689): Couldn't send request to peer 61e69695258f48baa984c9d395e8ce3f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d3e94b7e50754a81b3d72a07476cb7b9. This is attempt 1: this message will repeat every 5th retry.
I20260809 15:46:00.778385 24531 consensus_queue.cc:1035] T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 15:46:00.781435 23270 raft_consensus.cc:1261] T d3e94b7e50754a81b3d72a07476cb7b9 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 15:46:00.782572 24664 consensus_queue.cc:1035] T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 15:46:00.789949 24606 raft_consensus.cc:2932] T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } } }
I20260809 15:46:00.790951 23529 raft_consensus.cc:2932] T d3e94b7e50754a81b3d72a07476cb7b9 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } } }
I20260809 15:46:00.799299 23270 raft_consensus.cc:2932] T d3e94b7e50754a81b3d72a07476cb7b9 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } } }
I20260809 15:46:00.808192 23118 catalog_manager.cc:5495] T d3e94b7e50754a81b3d72a07476cb7b9 P 65c36d164b0540f099d592f38307ee6c reported cstate change: config changed from index -1 to 2, NON_VOTER 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) added. New cstate: current_term: 1 leader_uuid: "c0f38c1908414566aa9115bb2cbdd85d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } } }
I20260809 15:46:01.142282 24687 ts_tablet_manager.cc:927] T 0c48a3049897452383257aa757834fb5 P 61e69695258f48baa984c9d395e8ce3f: Initiating tablet copy from peer 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459)
I20260809 15:46:01.146294 24687 tablet_copy_client.cc:316] T 0c48a3049897452383257aa757834fb5 P 61e69695258f48baa984c9d395e8ce3f: tablet copy: Beginning tablet copy session from remote peer at address 127.20.202.65:35459
I20260809 15:46:01.147828 23298 tablet_copy_service.cc:140] P 67b3aeadfa7a432abe8af4d292e04ab9: Received BeginTabletCopySession request for tablet 0c48a3049897452383257aa757834fb5 from peer 61e69695258f48baa984c9d395e8ce3f ({username='slave'} at 127.0.0.1:55434)
I20260809 15:46:01.148294 23298 tablet_copy_service.cc:161] P 67b3aeadfa7a432abe8af4d292e04ab9: Beginning new tablet copy session on tablet 0c48a3049897452383257aa757834fb5 from peer 61e69695258f48baa984c9d395e8ce3f at {username='slave'} at 127.0.0.1:55434: session id = 61e69695258f48baa984c9d395e8ce3f-0c48a3049897452383257aa757834fb5
I20260809 15:46:01.155375 23298 tablet_copy_source_session.cc:205] T 0c48a3049897452383257aa757834fb5 P 67b3aeadfa7a432abe8af4d292e04ab9: Tablet Copy: opened 0 blocks and 1 log segments
I20260809 15:46:01.163794 24687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c48a3049897452383257aa757834fb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:46:01.175967 24687 tablet_copy_client.cc:791] T 0c48a3049897452383257aa757834fb5 P 61e69695258f48baa984c9d395e8ce3f: tablet copy: Starting download of 0 data blocks...
I20260809 15:46:01.176414 24687 tablet_copy_client.cc:655] T 0c48a3049897452383257aa757834fb5 P 61e69695258f48baa984c9d395e8ce3f: tablet copy: Starting download of 1 WAL segments...
I20260809 15:46:01.179498 24687 tablet_copy_client.cc:531] T 0c48a3049897452383257aa757834fb5 P 61e69695258f48baa984c9d395e8ce3f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260809 15:46:01.186496 24687 tablet_bootstrap.cc:492] T 0c48a3049897452383257aa757834fb5 P 61e69695258f48baa984c9d395e8ce3f: Bootstrap starting.
I20260809 15:46:01.195201 24689 ts_tablet_manager.cc:927] T 4bcac4b24e7a4448abdc934bba5a624e P 61e69695258f48baa984c9d395e8ce3f: Initiating tablet copy from peer c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:01.197147 24689 tablet_copy_client.cc:316] T 4bcac4b24e7a4448abdc934bba5a624e P 61e69695258f48baa984c9d395e8ce3f: tablet copy: Beginning tablet copy session from remote peer at address 127.20.202.66:36263
I20260809 15:46:01.198628 23424 tablet_copy_service.cc:140] P c0f38c1908414566aa9115bb2cbdd85d: Received BeginTabletCopySession request for tablet 4bcac4b24e7a4448abdc934bba5a624e from peer 61e69695258f48baa984c9d395e8ce3f ({username='slave'} at 127.0.0.1:52856)
I20260809 15:46:01.199108 23424 tablet_copy_service.cc:161] P c0f38c1908414566aa9115bb2cbdd85d: Beginning new tablet copy session on tablet 4bcac4b24e7a4448abdc934bba5a624e from peer 61e69695258f48baa984c9d395e8ce3f at {username='slave'} at 127.0.0.1:52856: session id = 61e69695258f48baa984c9d395e8ce3f-4bcac4b24e7a4448abdc934bba5a624e
I20260809 15:46:01.204851 24687 tablet_bootstrap.cc:492] T 0c48a3049897452383257aa757834fb5 P 61e69695258f48baa984c9d395e8ce3f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 15:46:01.205652 24687 tablet_bootstrap.cc:492] T 0c48a3049897452383257aa757834fb5 P 61e69695258f48baa984c9d395e8ce3f: Bootstrap complete.
I20260809 15:46:01.206370 24687 ts_tablet_manager.cc:1397] T 0c48a3049897452383257aa757834fb5 P 61e69695258f48baa984c9d395e8ce3f: Time spent bootstrapping tablet: real 0.020s user 0.015s sys 0.004s
I20260809 15:46:01.209127 24687 raft_consensus.cc:348] T 0c48a3049897452383257aa757834fb5 P 61e69695258f48baa984c9d395e8ce3f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } attrs { replace: true } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } }
I20260809 15:46:01.209750 23424 tablet_copy_source_session.cc:205] T 4bcac4b24e7a4448abdc934bba5a624e P c0f38c1908414566aa9115bb2cbdd85d: Tablet Copy: opened 0 blocks and 1 log segments
I20260809 15:46:01.209921 24687 raft_consensus.cc:729] T 0c48a3049897452383257aa757834fb5 P 61e69695258f48baa984c9d395e8ce3f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 61e69695258f48baa984c9d395e8ce3f, State: Initialized, Role: LEARNER
I20260809 15:46:01.210546 24687 consensus_queue.cc:260] T 0c48a3049897452383257aa757834fb5 P 61e69695258f48baa984c9d395e8ce3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } attrs { replace: true } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } }
I20260809 15:46:01.214190 24689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bcac4b24e7a4448abdc934bba5a624e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:46:01.228477 24687 ts_tablet_manager.cc:1428] T 0c48a3049897452383257aa757834fb5 P 61e69695258f48baa984c9d395e8ce3f: Time spent starting tablet: real 0.022s user 0.013s sys 0.008s
I20260809 15:46:01.229887 23298 tablet_copy_service.cc:342] P 67b3aeadfa7a432abe8af4d292e04ab9: Request end of tablet copy session 61e69695258f48baa984c9d395e8ce3f-0c48a3049897452383257aa757834fb5 received from {username='slave'} at 127.0.0.1:55434
I20260809 15:46:01.230257 23298 tablet_copy_service.cc:433] P 67b3aeadfa7a432abe8af4d292e04ab9: ending tablet copy session 61e69695258f48baa984c9d395e8ce3f-0c48a3049897452383257aa757834fb5 on tablet 0c48a3049897452383257aa757834fb5 with peer 61e69695258f48baa984c9d395e8ce3f
I20260809 15:46:01.233021 24689 tablet_copy_client.cc:791] T 4bcac4b24e7a4448abdc934bba5a624e P 61e69695258f48baa984c9d395e8ce3f: tablet copy: Starting download of 0 data blocks...
I20260809 15:46:01.233510 24689 tablet_copy_client.cc:655] T 4bcac4b24e7a4448abdc934bba5a624e P 61e69695258f48baa984c9d395e8ce3f: tablet copy: Starting download of 1 WAL segments...
I20260809 15:46:01.238444 24689 tablet_copy_client.cc:531] T 4bcac4b24e7a4448abdc934bba5a624e P 61e69695258f48baa984c9d395e8ce3f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260809 15:46:01.245658 24689 tablet_bootstrap.cc:492] T 4bcac4b24e7a4448abdc934bba5a624e P 61e69695258f48baa984c9d395e8ce3f: Bootstrap starting.
I20260809 15:46:01.250095 24687 ts_tablet_manager.cc:927] T d3e94b7e50754a81b3d72a07476cb7b9 P 61e69695258f48baa984c9d395e8ce3f: Initiating tablet copy from peer c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:01.251713 24687 tablet_copy_client.cc:316] T d3e94b7e50754a81b3d72a07476cb7b9 P 61e69695258f48baa984c9d395e8ce3f: tablet copy: Beginning tablet copy session from remote peer at address 127.20.202.66:36263
I20260809 15:46:01.253013 23424 tablet_copy_service.cc:140] P c0f38c1908414566aa9115bb2cbdd85d: Received BeginTabletCopySession request for tablet d3e94b7e50754a81b3d72a07476cb7b9 from peer 61e69695258f48baa984c9d395e8ce3f ({username='slave'} at 127.0.0.1:52856)
I20260809 15:46:01.253508 23424 tablet_copy_service.cc:161] P c0f38c1908414566aa9115bb2cbdd85d: Beginning new tablet copy session on tablet d3e94b7e50754a81b3d72a07476cb7b9 from peer 61e69695258f48baa984c9d395e8ce3f at {username='slave'} at 127.0.0.1:52856: session id = 61e69695258f48baa984c9d395e8ce3f-d3e94b7e50754a81b3d72a07476cb7b9
I20260809 15:46:01.270960 23424 tablet_copy_source_session.cc:205] T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d: Tablet Copy: opened 0 blocks and 1 log segments
I20260809 15:46:01.273744 24687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3e94b7e50754a81b3d72a07476cb7b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:46:01.275172 24689 tablet_bootstrap.cc:492] T 4bcac4b24e7a4448abdc934bba5a624e P 61e69695258f48baa984c9d395e8ce3f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 15:46:01.275962 24689 tablet_bootstrap.cc:492] T 4bcac4b24e7a4448abdc934bba5a624e P 61e69695258f48baa984c9d395e8ce3f: Bootstrap complete.
I20260809 15:46:01.276571 24689 ts_tablet_manager.cc:1397] T 4bcac4b24e7a4448abdc934bba5a624e P 61e69695258f48baa984c9d395e8ce3f: Time spent bootstrapping tablet: real 0.031s user 0.012s sys 0.012s
I20260809 15:46:01.279119 24689 raft_consensus.cc:348] T 4bcac4b24e7a4448abdc934bba5a624e P 61e69695258f48baa984c9d395e8ce3f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } }
I20260809 15:46:01.279923 24689 raft_consensus.cc:729] T 4bcac4b24e7a4448abdc934bba5a624e P 61e69695258f48baa984c9d395e8ce3f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 61e69695258f48baa984c9d395e8ce3f, State: Initialized, Role: LEARNER
I20260809 15:46:01.280766 24701 ts_tablet_manager.cc:927] T d3be1e356b254d22b995be32f1c12f6f P 61e69695258f48baa984c9d395e8ce3f: Initiating tablet copy from peer c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:01.280522 24689 consensus_queue.cc:260] T 4bcac4b24e7a4448abdc934bba5a624e P 61e69695258f48baa984c9d395e8ce3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } }
I20260809 15:46:01.285660 24689 ts_tablet_manager.cc:1428] T 4bcac4b24e7a4448abdc934bba5a624e P 61e69695258f48baa984c9d395e8ce3f: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20260809 15:46:01.287246 23424 tablet_copy_service.cc:342] P c0f38c1908414566aa9115bb2cbdd85d: Request end of tablet copy session 61e69695258f48baa984c9d395e8ce3f-4bcac4b24e7a4448abdc934bba5a624e received from {username='slave'} at 127.0.0.1:52856
I20260809 15:46:01.287699 23424 tablet_copy_service.cc:433] P c0f38c1908414566aa9115bb2cbdd85d: ending tablet copy session 61e69695258f48baa984c9d395e8ce3f-4bcac4b24e7a4448abdc934bba5a624e on tablet 4bcac4b24e7a4448abdc934bba5a624e with peer 61e69695258f48baa984c9d395e8ce3f
I20260809 15:46:01.295183 24701 tablet_copy_client.cc:316] T d3be1e356b254d22b995be32f1c12f6f P 61e69695258f48baa984c9d395e8ce3f: tablet copy: Beginning tablet copy session from remote peer at address 127.20.202.66:36263
I20260809 15:46:01.296639 23424 tablet_copy_service.cc:140] P c0f38c1908414566aa9115bb2cbdd85d: Received BeginTabletCopySession request for tablet d3be1e356b254d22b995be32f1c12f6f from peer 61e69695258f48baa984c9d395e8ce3f ({username='slave'} at 127.0.0.1:52856)
I20260809 15:46:01.297150 23424 tablet_copy_service.cc:161] P c0f38c1908414566aa9115bb2cbdd85d: Beginning new tablet copy session on tablet d3be1e356b254d22b995be32f1c12f6f from peer 61e69695258f48baa984c9d395e8ce3f at {username='slave'} at 127.0.0.1:52856: session id = 61e69695258f48baa984c9d395e8ce3f-d3be1e356b254d22b995be32f1c12f6f
I20260809 15:46:01.300650 24687 tablet_copy_client.cc:791] T d3e94b7e50754a81b3d72a07476cb7b9 P 61e69695258f48baa984c9d395e8ce3f: tablet copy: Starting download of 0 data blocks...
I20260809 15:46:01.301091 24687 tablet_copy_client.cc:655] T d3e94b7e50754a81b3d72a07476cb7b9 P 61e69695258f48baa984c9d395e8ce3f: tablet copy: Starting download of 1 WAL segments...
I20260809 15:46:01.305078 24687 tablet_copy_client.cc:531] T d3e94b7e50754a81b3d72a07476cb7b9 P 61e69695258f48baa984c9d395e8ce3f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260809 15:46:01.308775 23424 tablet_copy_source_session.cc:205] T d3be1e356b254d22b995be32f1c12f6f P c0f38c1908414566aa9115bb2cbdd85d: Tablet Copy: opened 0 blocks and 1 log segments
I20260809 15:46:01.312419 24701 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3be1e356b254d22b995be32f1c12f6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:46:01.317344 24687 tablet_bootstrap.cc:492] T d3e94b7e50754a81b3d72a07476cb7b9 P 61e69695258f48baa984c9d395e8ce3f: Bootstrap starting.
I20260809 15:46:01.328302 24701 tablet_copy_client.cc:791] T d3be1e356b254d22b995be32f1c12f6f P 61e69695258f48baa984c9d395e8ce3f: tablet copy: Starting download of 0 data blocks...
I20260809 15:46:01.329226 24701 tablet_copy_client.cc:655] T d3be1e356b254d22b995be32f1c12f6f P 61e69695258f48baa984c9d395e8ce3f: tablet copy: Starting download of 1 WAL segments...
I20260809 15:46:01.332894 24701 tablet_copy_client.cc:531] T d3be1e356b254d22b995be32f1c12f6f P 61e69695258f48baa984c9d395e8ce3f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260809 15:46:01.337453 24687 tablet_bootstrap.cc:492] T d3e94b7e50754a81b3d72a07476cb7b9 P 61e69695258f48baa984c9d395e8ce3f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 15:46:01.338218 24687 tablet_bootstrap.cc:492] T d3e94b7e50754a81b3d72a07476cb7b9 P 61e69695258f48baa984c9d395e8ce3f: Bootstrap complete.
I20260809 15:46:01.338801 24687 ts_tablet_manager.cc:1397] T d3e94b7e50754a81b3d72a07476cb7b9 P 61e69695258f48baa984c9d395e8ce3f: Time spent bootstrapping tablet: real 0.022s user 0.012s sys 0.006s
I20260809 15:46:01.341717 24687 raft_consensus.cc:348] T d3e94b7e50754a81b3d72a07476cb7b9 P 61e69695258f48baa984c9d395e8ce3f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } }
I20260809 15:46:01.342793 24687 raft_consensus.cc:729] T d3e94b7e50754a81b3d72a07476cb7b9 P 61e69695258f48baa984c9d395e8ce3f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 61e69695258f48baa984c9d395e8ce3f, State: Initialized, Role: LEARNER
I20260809 15:46:01.342960 24701 tablet_bootstrap.cc:492] T d3be1e356b254d22b995be32f1c12f6f P 61e69695258f48baa984c9d395e8ce3f: Bootstrap starting.
I20260809 15:46:01.343452 24687 consensus_queue.cc:260] T d3e94b7e50754a81b3d72a07476cb7b9 P 61e69695258f48baa984c9d395e8ce3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } }
I20260809 15:46:01.351600 24687 ts_tablet_manager.cc:1428] T d3e94b7e50754a81b3d72a07476cb7b9 P 61e69695258f48baa984c9d395e8ce3f: Time spent starting tablet: real 0.013s user 0.009s sys 0.001s
I20260809 15:46:01.355937 23424 tablet_copy_service.cc:342] P c0f38c1908414566aa9115bb2cbdd85d: Request end of tablet copy session 61e69695258f48baa984c9d395e8ce3f-d3e94b7e50754a81b3d72a07476cb7b9 received from {username='slave'} at 127.0.0.1:52856
I20260809 15:46:01.356308 23424 tablet_copy_service.cc:433] P c0f38c1908414566aa9115bb2cbdd85d: ending tablet copy session 61e69695258f48baa984c9d395e8ce3f-d3e94b7e50754a81b3d72a07476cb7b9 on tablet d3e94b7e50754a81b3d72a07476cb7b9 with peer 61e69695258f48baa984c9d395e8ce3f
I20260809 15:46:01.363881 24701 tablet_bootstrap.cc:492] T d3be1e356b254d22b995be32f1c12f6f P 61e69695258f48baa984c9d395e8ce3f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 15:46:01.364595 24701 tablet_bootstrap.cc:492] T d3be1e356b254d22b995be32f1c12f6f P 61e69695258f48baa984c9d395e8ce3f: Bootstrap complete.
I20260809 15:46:01.365170 24701 ts_tablet_manager.cc:1397] T d3be1e356b254d22b995be32f1c12f6f P 61e69695258f48baa984c9d395e8ce3f: Time spent bootstrapping tablet: real 0.022s user 0.021s sys 0.000s
I20260809 15:46:01.367188 24701 raft_consensus.cc:348] T d3be1e356b254d22b995be32f1c12f6f P 61e69695258f48baa984c9d395e8ce3f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } attrs { replace: true } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } }
I20260809 15:46:01.367822 24701 raft_consensus.cc:729] T d3be1e356b254d22b995be32f1c12f6f P 61e69695258f48baa984c9d395e8ce3f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 61e69695258f48baa984c9d395e8ce3f, State: Initialized, Role: LEARNER
I20260809 15:46:01.368287 24701 consensus_queue.cc:260] T d3be1e356b254d22b995be32f1c12f6f P 61e69695258f48baa984c9d395e8ce3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } attrs { replace: true } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } }
I20260809 15:46:01.370208 24701 ts_tablet_manager.cc:1428] T d3be1e356b254d22b995be32f1c12f6f P 61e69695258f48baa984c9d395e8ce3f: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260809 15:46:01.371824 23424 tablet_copy_service.cc:342] P c0f38c1908414566aa9115bb2cbdd85d: Request end of tablet copy session 61e69695258f48baa984c9d395e8ce3f-d3be1e356b254d22b995be32f1c12f6f received from {username='slave'} at 127.0.0.1:52856
I20260809 15:46:01.372200 23424 tablet_copy_service.cc:433] P c0f38c1908414566aa9115bb2cbdd85d: ending tablet copy session 61e69695258f48baa984c9d395e8ce3f-d3be1e356b254d22b995be32f1c12f6f on tablet d3be1e356b254d22b995be32f1c12f6f with peer 61e69695258f48baa984c9d395e8ce3f
I20260809 15:46:01.596422 24472 raft_consensus.cc:1203] T 0c48a3049897452383257aa757834fb5 P 61e69695258f48baa984c9d395e8ce3f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260809 15:46:01.778115 24472 raft_consensus.cc:1203] T 4bcac4b24e7a4448abdc934bba5a624e P 61e69695258f48baa984c9d395e8ce3f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260809 15:46:01.781375 24472 raft_consensus.cc:1203] T d3e94b7e50754a81b3d72a07476cb7b9 P 61e69695258f48baa984c9d395e8ce3f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260809 15:46:01.837736 24472 raft_consensus.cc:1203] T d3be1e356b254d22b995be32f1c12f6f P 61e69695258f48baa984c9d395e8ce3f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260809 15:46:02.168078 24633 raft_consensus.cc:1050] T 0c48a3049897452383257aa757834fb5 P 67b3aeadfa7a432abe8af4d292e04ab9: attempting to promote NON_VOTER 61e69695258f48baa984c9d395e8ce3f to VOTER
I20260809 15:46:02.170235 24633 consensus_queue.cc:237] T 0c48a3049897452383257aa757834fb5 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } attrs { replace: true } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:02.185672 24472 raft_consensus.cc:1261] T 0c48a3049897452383257aa757834fb5 P 61e69695258f48baa984c9d395e8ce3f [term 1 LEARNER]: Refusing update from remote peer 67b3aeadfa7a432abe8af4d292e04ab9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 15:46:02.186322 23397 raft_consensus.cc:1261] T 0c48a3049897452383257aa757834fb5 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Refusing update from remote peer 67b3aeadfa7a432abe8af4d292e04ab9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 15:46:02.188295 23521 raft_consensus.cc:1261] T 0c48a3049897452383257aa757834fb5 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Refusing update from remote peer 67b3aeadfa7a432abe8af4d292e04ab9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 15:46:02.188408 24633 consensus_queue.cc:1035] T 0c48a3049897452383257aa757834fb5 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260809 15:46:02.190313 24652 consensus_queue.cc:1035] T 0c48a3049897452383257aa757834fb5 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 15:46:02.191243 24639 consensus_queue.cc:1035] T 0c48a3049897452383257aa757834fb5 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 15:46:02.200137 24652 raft_consensus.cc:2932] T 0c48a3049897452383257aa757834fb5 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } attrs { replace: true } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:02.202001 24472 raft_consensus.cc:2932] T 0c48a3049897452383257aa757834fb5 P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } attrs { replace: true } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:02.214341 23521 raft_consensus.cc:2932] T 0c48a3049897452383257aa757834fb5 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } attrs { replace: true } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:02.215523 23404 raft_consensus.cc:2932] T 0c48a3049897452383257aa757834fb5 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } attrs { replace: true } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:02.215480 23118 catalog_manager.cc:5495] T 0c48a3049897452383257aa757834fb5 P 61e69695258f48baa984c9d395e8ce3f reported cstate change: config changed from index 2 to 3, 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } attrs { replace: true } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:02.251508 23274 consensus_queue.cc:237] T 0c48a3049897452383257aa757834fb5 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:02.256552 24472 raft_consensus.cc:1261] T 0c48a3049897452383257aa757834fb5 P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Refusing update from remote peer 67b3aeadfa7a432abe8af4d292e04ab9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260809 15:46:02.256714 23521 raft_consensus.cc:1261] T 0c48a3049897452383257aa757834fb5 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Refusing update from remote peer 67b3aeadfa7a432abe8af4d292e04ab9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260809 15:46:02.257748 24633 consensus_queue.cc:1035] T 0c48a3049897452383257aa757834fb5 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260809 15:46:02.258399 24652 consensus_queue.cc:1035] T 0c48a3049897452383257aa757834fb5 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260809 15:46:02.264518 24639 raft_consensus.cc:2932] T 0c48a3049897452383257aa757834fb5 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:02.269289 24472 raft_consensus.cc:2932] T 0c48a3049897452383257aa757834fb5 P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:02.274633 23094 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 0c48a3049897452383257aa757834fb5 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260809 15:46:02.285429 23527 raft_consensus.cc:2932] T 0c48a3049897452383257aa757834fb5 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:02.296787 23115 catalog_manager.cc:5495] T 0c48a3049897452383257aa757834fb5 P 61e69695258f48baa984c9d395e8ce3f reported cstate change: config changed from index 3 to 4, VOTER c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66) evicted. New cstate: current_term: 1 leader_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:02.306526 23389 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c48a3049897452383257aa757834fb5 with delete_type TABLET_DATA_TOMBSTONED (TS c0f38c1908414566aa9115bb2cbdd85d not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:55312
I20260809 15:46:02.309744 24726 tablet_replica.cc:330] T 0c48a3049897452383257aa757834fb5 P c0f38c1908414566aa9115bb2cbdd85d: stopping tablet replica
I20260809 15:46:02.310672 24726 raft_consensus.cc:2229] T 0c48a3049897452383257aa757834fb5 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 15:46:02.311373 24726 raft_consensus.cc:2258] T 0c48a3049897452383257aa757834fb5 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 15:46:02.315028 24726 ts_tablet_manager.cc:1905] T 0c48a3049897452383257aa757834fb5 P c0f38c1908414566aa9115bb2cbdd85d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260809 15:46:02.316337 24705 raft_consensus.cc:1050] T 4bcac4b24e7a4448abdc934bba5a624e P c0f38c1908414566aa9115bb2cbdd85d: attempting to promote NON_VOTER 61e69695258f48baa984c9d395e8ce3f to VOTER
I20260809 15:46:02.320770 24705 consensus_queue.cc:237] T 4bcac4b24e7a4448abdc934bba5a624e P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:02.338368 23274 raft_consensus.cc:1261] T 4bcac4b24e7a4448abdc934bba5a624e P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 15:46:02.344230 24686 consensus_queue.cc:1035] T 4bcac4b24e7a4448abdc934bba5a624e P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 15:46:02.348764 24726 ts_tablet_manager.cc:1918] T 0c48a3049897452383257aa757834fb5 P c0f38c1908414566aa9115bb2cbdd85d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260809 15:46:02.350253 24726 log.cc:1196] T 0c48a3049897452383257aa757834fb5 P c0f38c1908414566aa9115bb2cbdd85d: Deleting WAL directory at /tmp/dist-test-taskj2AJ_N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786290285951082-21289-0/minicluster-data/ts-1-root/wals/0c48a3049897452383257aa757834fb5
I20260809 15:46:02.352247 23529 raft_consensus.cc:1261] T 4bcac4b24e7a4448abdc934bba5a624e P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 15:46:02.352926 23095 catalog_manager.cc:4841] TS c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263): tablet 0c48a3049897452383257aa757834fb5 (table test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]) successfully deleted
I20260809 15:46:02.355432 24664 consensus_queue.cc:1035] T 4bcac4b24e7a4448abdc934bba5a624e P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 15:46:02.360060 24472 raft_consensus.cc:1261] T 4bcac4b24e7a4448abdc934bba5a624e P 61e69695258f48baa984c9d395e8ce3f [term 1 LEARNER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 15:46:02.371622 24686 consensus_queue.cc:1035] T 4bcac4b24e7a4448abdc934bba5a624e P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 15:46:02.380703 24707 raft_consensus.cc:2932] T 4bcac4b24e7a4448abdc934bba5a624e P c0f38c1908414566aa9115bb2cbdd85d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:02.384533 23529 raft_consensus.cc:2932] T 4bcac4b24e7a4448abdc934bba5a624e P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:02.391954 24472 raft_consensus.cc:2932] T 4bcac4b24e7a4448abdc934bba5a624e P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:02.394970 23270 raft_consensus.cc:2932] T 4bcac4b24e7a4448abdc934bba5a624e P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:02.399565 24706 raft_consensus.cc:1050] T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d: attempting to promote NON_VOTER 61e69695258f48baa984c9d395e8ce3f to VOTER
I20260809 15:46:02.401770 24706 consensus_queue.cc:237] T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:02.408699 23117 catalog_manager.cc:5495] T 4bcac4b24e7a4448abdc934bba5a624e P c0f38c1908414566aa9115bb2cbdd85d reported cstate change: config changed from index 2 to 3, 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c0f38c1908414566aa9115bb2cbdd85d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 15:46:02.419430 24664 raft_consensus.cc:1050] T d3be1e356b254d22b995be32f1c12f6f P c0f38c1908414566aa9115bb2cbdd85d: attempting to promote NON_VOTER 61e69695258f48baa984c9d395e8ce3f to VOTER
I20260809 15:46:02.421610 24664 consensus_queue.cc:237] T d3be1e356b254d22b995be32f1c12f6f P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } attrs { replace: true } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:02.424721 23404 consensus_queue.cc:237] T 4bcac4b24e7a4448abdc934bba5a624e P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:02.437311 23529 raft_consensus.cc:1261] T 4bcac4b24e7a4448abdc934bba5a624e P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260809 15:46:02.439092 24686 consensus_queue.cc:1035] T 4bcac4b24e7a4448abdc934bba5a624e P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260809 15:46:02.445001 23521 raft_consensus.cc:1261] T d3be1e356b254d22b995be32f1c12f6f P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 15:46:02.444958 24707 raft_consensus.cc:2932] T 4bcac4b24e7a4448abdc934bba5a624e P c0f38c1908414566aa9115bb2cbdd85d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:02.446048 24472 raft_consensus.cc:1261] T 4bcac4b24e7a4448abdc934bba5a624e P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260809 15:46:02.447429 24664 consensus_queue.cc:1035] T d3be1e356b254d22b995be32f1c12f6f P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 15:46:02.447013 23521 raft_consensus.cc:2932] T 4bcac4b24e7a4448abdc934bba5a624e P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:02.454800 24686 consensus_queue.cc:1035] T 4bcac4b24e7a4448abdc934bba5a624e P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260809 15:46:02.455463 24475 raft_consensus.cc:1261] T d3e94b7e50754a81b3d72a07476cb7b9 P 61e69695258f48baa984c9d395e8ce3f [term 1 LEARNER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 15:46:02.459179 23270 raft_consensus.cc:1261] T d3be1e356b254d22b995be32f1c12f6f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 15:46:02.462708 23095 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 4bcac4b24e7a4448abdc934bba5a624e with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260809 15:46:02.464041 23274 raft_consensus.cc:1261] T d3e94b7e50754a81b3d72a07476cb7b9 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 15:46:02.465176 24686 consensus_queue.cc:1035] T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 15:46:02.466910 23118 catalog_manager.cc:5495] T 4bcac4b24e7a4448abdc934bba5a624e P 65c36d164b0540f099d592f38307ee6c reported cstate change: config changed from index 3 to 4, VOTER 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65) evicted. New cstate: current_term: 1 leader_uuid: "c0f38c1908414566aa9115bb2cbdd85d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:02.466797 24475 raft_consensus.cc:2932] T 4bcac4b24e7a4448abdc934bba5a624e P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:02.460826 24706 consensus_queue.cc:1035] T d3be1e356b254d22b995be32f1c12f6f P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 15:46:02.481525 23260 tablet_service.cc:1514] Processing DeleteTablet for tablet 4bcac4b24e7a4448abdc934bba5a624e with delete_type TABLET_DATA_TOMBSTONED (TS 67b3aeadfa7a432abe8af4d292e04ab9 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:53552
I20260809 15:46:02.488143 23529 raft_consensus.cc:1261] T d3e94b7e50754a81b3d72a07476cb7b9 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 15:46:02.491835 24706 consensus_queue.cc:1035] T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 15:46:02.503619 24707 consensus_queue.cc:1035] T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 15:46:02.504563 24472 raft_consensus.cc:1261] T d3be1e356b254d22b995be32f1c12f6f P 61e69695258f48baa984c9d395e8ce3f [term 1 LEARNER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 15:46:02.511247 24707 raft_consensus.cc:2932] T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:02.512172 23529 raft_consensus.cc:2932] T d3e94b7e50754a81b3d72a07476cb7b9 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:02.523785 24749 tablet_replica.cc:330] T 4bcac4b24e7a4448abdc934bba5a624e P 67b3aeadfa7a432abe8af4d292e04ab9: stopping tablet replica
I20260809 15:46:02.523587 23274 raft_consensus.cc:2932] T d3e94b7e50754a81b3d72a07476cb7b9 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:02.532452 24749 raft_consensus.cc:2229] T 4bcac4b24e7a4448abdc934bba5a624e P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 15:46:02.533062 24749 raft_consensus.cc:2258] T 4bcac4b24e7a4448abdc934bba5a624e P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 15:46:02.548270 24707 consensus_queue.cc:1035] T d3be1e356b254d22b995be32f1c12f6f P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 15:46:02.557174 23118 catalog_manager.cc:5495] T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d reported cstate change: config changed from index 2 to 3, 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c0f38c1908414566aa9115bb2cbdd85d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260809 15:46:02.558691 24749 ts_tablet_manager.cc:1905] T 4bcac4b24e7a4448abdc934bba5a624e P 67b3aeadfa7a432abe8af4d292e04ab9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260809 15:46:02.575209 24686 raft_consensus.cc:2932] T d3be1e356b254d22b995be32f1c12f6f P c0f38c1908414566aa9115bb2cbdd85d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } attrs { replace: true } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:02.576630 24472 raft_consensus.cc:2932] T d3e94b7e50754a81b3d72a07476cb7b9 P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:02.577723 23270 raft_consensus.cc:2932] T d3be1e356b254d22b995be32f1c12f6f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } attrs { replace: true } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:02.578728 24749 ts_tablet_manager.cc:1918] T 4bcac4b24e7a4448abdc934bba5a624e P 67b3aeadfa7a432abe8af4d292e04ab9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260809 15:46:02.579103 24749 log.cc:1196] T 4bcac4b24e7a4448abdc934bba5a624e P 67b3aeadfa7a432abe8af4d292e04ab9: Deleting WAL directory at /tmp/dist-test-taskj2AJ_N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786290285951082-21289-0/minicluster-data/ts-0-root/wals/4bcac4b24e7a4448abdc934bba5a624e
I20260809 15:46:02.580983 23094 catalog_manager.cc:4841] TS 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459): tablet 4bcac4b24e7a4448abdc934bba5a624e (table test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]) successfully deleted
I20260809 15:46:02.597289 23529 raft_consensus.cc:2932] T d3be1e356b254d22b995be32f1c12f6f P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } attrs { replace: true } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:02.601238 24475 raft_consensus.cc:2932] T d3be1e356b254d22b995be32f1c12f6f P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } attrs { replace: true } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:02.606578 23116 catalog_manager.cc:5495] T d3be1e356b254d22b995be32f1c12f6f P 67b3aeadfa7a432abe8af4d292e04ab9 reported cstate change: config changed from index 2 to 3, 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c0f38c1908414566aa9115bb2cbdd85d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } attrs { replace: true } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:02.624323 23396 consensus_queue.cc:237] T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:02.632206 23529 raft_consensus.cc:1261] T d3e94b7e50754a81b3d72a07476cb7b9 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260809 15:46:02.631875 24475 raft_consensus.cc:1261] T d3e94b7e50754a81b3d72a07476cb7b9 P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260809 15:46:02.633503 24706 consensus_queue.cc:1035] T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260809 15:46:02.634456 24707 consensus_queue.cc:1035] T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260809 15:46:02.640676 23397 consensus_queue.cc:237] T d3be1e356b254d22b995be32f1c12f6f P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:02.645476 24472 raft_consensus.cc:1261] T d3be1e356b254d22b995be32f1c12f6f P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260809 15:46:02.645934 23527 raft_consensus.cc:2932] T d3e94b7e50754a81b3d72a07476cb7b9 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:02.647917 23274 raft_consensus.cc:1261] T d3be1e356b254d22b995be32f1c12f6f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260809 15:46:02.649348 24739 consensus_queue.cc:1035] T d3be1e356b254d22b995be32f1c12f6f P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260809 15:46:02.650930 24739 consensus_queue.cc:1035] T d3be1e356b254d22b995be32f1c12f6f P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260809 15:46:02.653555 24475 raft_consensus.cc:2932] T d3e94b7e50754a81b3d72a07476cb7b9 P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:02.642525 24706 raft_consensus.cc:2932] T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:02.658425 24472 raft_consensus.cc:2932] T d3be1e356b254d22b995be32f1c12f6f P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 65c36d164b0540f099d592f38307ee6c (127.20.202.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:02.660531 23274 raft_consensus.cc:2932] T d3be1e356b254d22b995be32f1c12f6f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 65c36d164b0540f099d592f38307ee6c (127.20.202.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:02.656957 24686 raft_consensus.cc:2932] T d3be1e356b254d22b995be32f1c12f6f P c0f38c1908414566aa9115bb2cbdd85d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 65c36d164b0540f099d592f38307ee6c (127.20.202.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:02.668035 23115 catalog_manager.cc:5495] T d3e94b7e50754a81b3d72a07476cb7b9 P 61e69695258f48baa984c9d395e8ce3f reported cstate change: config changed from index 3 to 4, VOTER 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65) evicted. New cstate: current_term: 1 leader_uuid: "c0f38c1908414566aa9115bb2cbdd85d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:02.677627 23260 tablet_service.cc:1514] Processing DeleteTablet for tablet d3e94b7e50754a81b3d72a07476cb7b9 with delete_type TABLET_DATA_TOMBSTONED (TS 67b3aeadfa7a432abe8af4d292e04ab9 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:53552
I20260809 15:46:02.678437 24749 tablet_replica.cc:330] T d3e94b7e50754a81b3d72a07476cb7b9 P 67b3aeadfa7a432abe8af4d292e04ab9: stopping tablet replica
I20260809 15:46:02.679469 24749 raft_consensus.cc:2229] T d3e94b7e50754a81b3d72a07476cb7b9 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 15:46:02.680073 24749 raft_consensus.cc:2258] T d3e94b7e50754a81b3d72a07476cb7b9 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 15:46:02.681397 23115 catalog_manager.cc:5495] T d3be1e356b254d22b995be32f1c12f6f P 61e69695258f48baa984c9d395e8ce3f reported cstate change: config changed from index 3 to 4, VOTER 65c36d164b0540f099d592f38307ee6c (127.20.202.67) evicted. New cstate: current_term: 1 leader_uuid: "c0f38c1908414566aa9115bb2cbdd85d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:02.685297 24749 ts_tablet_manager.cc:1905] T d3e94b7e50754a81b3d72a07476cb7b9 P 67b3aeadfa7a432abe8af4d292e04ab9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260809 15:46:02.690606 23500 tablet_service.cc:1514] Processing DeleteTablet for tablet d3be1e356b254d22b995be32f1c12f6f with delete_type TABLET_DATA_TOMBSTONED (TS 65c36d164b0540f099d592f38307ee6c not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:56760
I20260809 15:46:02.692162 24757 tablet_replica.cc:330] T d3be1e356b254d22b995be32f1c12f6f P 65c36d164b0540f099d592f38307ee6c: stopping tablet replica
I20260809 15:46:02.693097 24757 raft_consensus.cc:2229] T d3be1e356b254d22b995be32f1c12f6f P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 15:46:02.693660 24757 raft_consensus.cc:2258] T d3be1e356b254d22b995be32f1c12f6f P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 15:46:02.696300 24757 ts_tablet_manager.cc:1905] T d3be1e356b254d22b995be32f1c12f6f P 65c36d164b0540f099d592f38307ee6c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260809 15:46:02.701452 24749 ts_tablet_manager.cc:1918] T d3e94b7e50754a81b3d72a07476cb7b9 P 67b3aeadfa7a432abe8af4d292e04ab9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260809 15:46:02.701836 24749 log.cc:1196] T d3e94b7e50754a81b3d72a07476cb7b9 P 67b3aeadfa7a432abe8af4d292e04ab9: Deleting WAL directory at /tmp/dist-test-taskj2AJ_N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786290285951082-21289-0/minicluster-data/ts-0-root/wals/d3e94b7e50754a81b3d72a07476cb7b9
I20260809 15:46:02.703325 23094 catalog_manager.cc:4841] TS 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459): tablet d3e94b7e50754a81b3d72a07476cb7b9 (table test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]) successfully deleted
I20260809 15:46:02.707657 23095 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet d3be1e356b254d22b995be32f1c12f6f with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260809 15:46:02.710057 24757 ts_tablet_manager.cc:1918] T d3be1e356b254d22b995be32f1c12f6f P 65c36d164b0540f099d592f38307ee6c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260809 15:46:02.710433 24757 log.cc:1196] T d3be1e356b254d22b995be32f1c12f6f P 65c36d164b0540f099d592f38307ee6c: Deleting WAL directory at /tmp/dist-test-taskj2AJ_N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786290285951082-21289-0/minicluster-data/ts-2-root/wals/d3be1e356b254d22b995be32f1c12f6f
I20260809 15:46:02.711824 23095 catalog_manager.cc:4841] TS 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553): tablet d3be1e356b254d22b995be32f1c12f6f (table test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]) successfully deleted
I20260809 15:46:02.713307 23095 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet d3e94b7e50754a81b3d72a07476cb7b9 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260809 15:46:03.776510 23527 consensus_queue.cc:237] T cee02fff2c9c48a19f43eb358a89e7b0 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } attrs { replace: true } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } }
I20260809 15:46:03.782384 23396 raft_consensus.cc:1261] T cee02fff2c9c48a19f43eb358a89e7b0 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 15:46:03.782658 23274 raft_consensus.cc:1261] T cee02fff2c9c48a19f43eb358a89e7b0 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 15:46:03.783589 24675 consensus_queue.cc:1035] T cee02fff2c9c48a19f43eb358a89e7b0 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260809 15:46:03.784369 23468 consensus_peers.cc:487] T cee02fff2c9c48a19f43eb358a89e7b0 P 65c36d164b0540f099d592f38307ee6c -> Peer 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689): Couldn't send request to peer 61e69695258f48baa984c9d395e8ce3f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cee02fff2c9c48a19f43eb358a89e7b0. This is attempt 1: this message will repeat every 5th retry.
I20260809 15:46:03.784454 24768 consensus_queue.cc:1035] T cee02fff2c9c48a19f43eb358a89e7b0 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 15:46:03.791828 24768 raft_consensus.cc:2932] T cee02fff2c9c48a19f43eb358a89e7b0 P 65c36d164b0540f099d592f38307ee6c [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } attrs { replace: true } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } } }
I20260809 15:46:03.793216 23396 raft_consensus.cc:2932] T cee02fff2c9c48a19f43eb358a89e7b0 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } attrs { replace: true } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } } }
I20260809 15:46:03.794531 23274 raft_consensus.cc:2932] T cee02fff2c9c48a19f43eb358a89e7b0 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } attrs { replace: true } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } } }
I20260809 15:46:03.813284 23397 consensus_queue.cc:237] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } }
I20260809 15:46:03.811388 23115 catalog_manager.cc:5495] T cee02fff2c9c48a19f43eb358a89e7b0 P 67b3aeadfa7a432abe8af4d292e04ab9 reported cstate change: config changed from index -1 to 2, NON_VOTER 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) added. New cstate: current_term: 1 leader_uuid: "65c36d164b0540f099d592f38307ee6c" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } attrs { replace: true } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } } }
W20260809 15:46:03.828294 23345 consensus_peers.cc:487] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P c0f38c1908414566aa9115bb2cbdd85d -> Peer 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689): Couldn't send request to peer 61e69695258f48baa984c9d395e8ce3f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b3aa6a9a52804df0ba9b0ca3c82f38c3. This is attempt 1: this message will repeat every 5th retry.
I20260809 15:46:03.829571 23270 raft_consensus.cc:1261] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 15:46:03.829996 23529 raft_consensus.cc:1261] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 15:46:03.830885 24763 consensus_queue.cc:1035] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 15:46:03.832430 24763 consensus_queue.cc:1035] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 15:46:03.838092 24686 raft_consensus.cc:2932] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P c0f38c1908414566aa9115bb2cbdd85d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } } }
I20260809 15:46:03.844022 23270 raft_consensus.cc:2932] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } } }
I20260809 15:46:03.844353 23527 raft_consensus.cc:2932] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } } }
I20260809 15:46:03.863649 23277 consensus_queue.cc:237] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } attrs { replace: true } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } }
I20260809 15:46:03.867879 23116 catalog_manager.cc:5495] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P c0f38c1908414566aa9115bb2cbdd85d reported cstate change: config changed from index -1 to 2, NON_VOTER 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) added. New cstate: current_term: 1 leader_uuid: "c0f38c1908414566aa9115bb2cbdd85d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20260809 15:46:03.873548 23223 consensus_peers.cc:487] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 67b3aeadfa7a432abe8af4d292e04ab9 -> Peer 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689): Couldn't send request to peer 61e69695258f48baa984c9d395e8ce3f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5712f6b6fffc4cc5a8ddfbe63e4097df. This is attempt 1: this message will repeat every 5th retry.
I20260809 15:46:03.875144 23396 raft_consensus.cc:1261] T 5712f6b6fffc4cc5a8ddfbe63e4097df P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Refusing update from remote peer 67b3aeadfa7a432abe8af4d292e04ab9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 15:46:03.875622 23527 raft_consensus.cc:1261] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Refusing update from remote peer 67b3aeadfa7a432abe8af4d292e04ab9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 15:46:03.876559 24652 consensus_queue.cc:1035] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 15:46:03.881148 24762 consensus_queue.cc:1035] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 15:46:03.890940 24717 raft_consensus.cc:2932] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } attrs { replace: true } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } } }
I20260809 15:46:03.898018 23397 raft_consensus.cc:2932] T 5712f6b6fffc4cc5a8ddfbe63e4097df P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } attrs { replace: true } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } } }
I20260809 15:46:03.901486 23527 raft_consensus.cc:2932] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } attrs { replace: true } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } } }
I20260809 15:46:03.910993 23270 consensus_queue.cc:237] T 0a51c0baaa6e432997c9d3552af4f254 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } attrs { replace: true } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } }
I20260809 15:46:03.914682 23116 catalog_manager.cc:5495] T 5712f6b6fffc4cc5a8ddfbe63e4097df P c0f38c1908414566aa9115bb2cbdd85d reported cstate change: config changed from index -1 to 2, NON_VOTER 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) added. New cstate: current_term: 1 leader_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } attrs { replace: true } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } } }
I20260809 15:46:03.922003 23396 raft_consensus.cc:1261] T 0a51c0baaa6e432997c9d3552af4f254 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Refusing update from remote peer 67b3aeadfa7a432abe8af4d292e04ab9: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 15:46:03.924423 24717 consensus_queue.cc:1035] T 0a51c0baaa6e432997c9d3552af4f254 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
W20260809 15:46:03.928767 23223 consensus_peers.cc:487] T 0a51c0baaa6e432997c9d3552af4f254 P 67b3aeadfa7a432abe8af4d292e04ab9 -> Peer 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689): Couldn't send request to peer 61e69695258f48baa984c9d395e8ce3f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0a51c0baaa6e432997c9d3552af4f254. This is attempt 1: this message will repeat every 5th retry.
I20260809 15:46:03.929280 23527 raft_consensus.cc:1261] T 0a51c0baaa6e432997c9d3552af4f254 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Refusing update from remote peer 67b3aeadfa7a432abe8af4d292e04ab9: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 15:46:03.930478 24652 consensus_queue.cc:1035] T 0a51c0baaa6e432997c9d3552af4f254 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 15:46:03.941820 24652 raft_consensus.cc:2932] T 0a51c0baaa6e432997c9d3552af4f254 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 LEADER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } attrs { replace: true } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } } }
I20260809 15:46:03.943231 23397 raft_consensus.cc:2932] T 0a51c0baaa6e432997c9d3552af4f254 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } attrs { replace: true } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } } }
I20260809 15:46:03.944140 23527 raft_consensus.cc:2932] T 0a51c0baaa6e432997c9d3552af4f254 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } attrs { replace: true } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } } }
I20260809 15:46:03.959004 23117 catalog_manager.cc:5495] T 0a51c0baaa6e432997c9d3552af4f254 P c0f38c1908414566aa9115bb2cbdd85d reported cstate change: config changed from index -1 to 2, NON_VOTER 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) added. New cstate: current_term: 2 leader_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } attrs { replace: true } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } } }
I20260809 15:46:04.285244 24803 ts_tablet_manager.cc:927] T cee02fff2c9c48a19f43eb358a89e7b0 P 61e69695258f48baa984c9d395e8ce3f: Initiating tablet copy from peer 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:04.287191 24803 tablet_copy_client.cc:316] T cee02fff2c9c48a19f43eb358a89e7b0 P 61e69695258f48baa984c9d395e8ce3f: tablet copy: Beginning tablet copy session from remote peer at address 127.20.202.67:40553
I20260809 15:46:04.288599 23546 tablet_copy_service.cc:140] P 65c36d164b0540f099d592f38307ee6c: Received BeginTabletCopySession request for tablet cee02fff2c9c48a19f43eb358a89e7b0 from peer 61e69695258f48baa984c9d395e8ce3f ({username='slave'} at 127.0.0.1:44018)
I20260809 15:46:04.289104 23546 tablet_copy_service.cc:161] P 65c36d164b0540f099d592f38307ee6c: Beginning new tablet copy session on tablet cee02fff2c9c48a19f43eb358a89e7b0 from peer 61e69695258f48baa984c9d395e8ce3f at {username='slave'} at 127.0.0.1:44018: session id = 61e69695258f48baa984c9d395e8ce3f-cee02fff2c9c48a19f43eb358a89e7b0
I20260809 15:46:04.295655 23546 tablet_copy_source_session.cc:205] T cee02fff2c9c48a19f43eb358a89e7b0 P 65c36d164b0540f099d592f38307ee6c: Tablet Copy: opened 0 blocks and 1 log segments
I20260809 15:46:04.298220 24803 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cee02fff2c9c48a19f43eb358a89e7b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:46:04.311671 24803 tablet_copy_client.cc:791] T cee02fff2c9c48a19f43eb358a89e7b0 P 61e69695258f48baa984c9d395e8ce3f: tablet copy: Starting download of 0 data blocks...
I20260809 15:46:04.312139 24803 tablet_copy_client.cc:655] T cee02fff2c9c48a19f43eb358a89e7b0 P 61e69695258f48baa984c9d395e8ce3f: tablet copy: Starting download of 1 WAL segments...
I20260809 15:46:04.315357 24803 tablet_copy_client.cc:531] T cee02fff2c9c48a19f43eb358a89e7b0 P 61e69695258f48baa984c9d395e8ce3f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260809 15:46:04.321664 24803 tablet_bootstrap.cc:492] T cee02fff2c9c48a19f43eb358a89e7b0 P 61e69695258f48baa984c9d395e8ce3f: Bootstrap starting.
I20260809 15:46:04.338259 24803 tablet_bootstrap.cc:492] T cee02fff2c9c48a19f43eb358a89e7b0 P 61e69695258f48baa984c9d395e8ce3f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 15:46:04.338982 24803 tablet_bootstrap.cc:492] T cee02fff2c9c48a19f43eb358a89e7b0 P 61e69695258f48baa984c9d395e8ce3f: Bootstrap complete.
I20260809 15:46:04.339564 24803 ts_tablet_manager.cc:1397] T cee02fff2c9c48a19f43eb358a89e7b0 P 61e69695258f48baa984c9d395e8ce3f: Time spent bootstrapping tablet: real 0.018s user 0.015s sys 0.005s
I20260809 15:46:04.341584 24803 raft_consensus.cc:348] T cee02fff2c9c48a19f43eb358a89e7b0 P 61e69695258f48baa984c9d395e8ce3f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } attrs { replace: true } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } }
I20260809 15:46:04.342291 24803 raft_consensus.cc:729] T cee02fff2c9c48a19f43eb358a89e7b0 P 61e69695258f48baa984c9d395e8ce3f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 61e69695258f48baa984c9d395e8ce3f, State: Initialized, Role: LEARNER
I20260809 15:46:04.342761 24803 consensus_queue.cc:260] T cee02fff2c9c48a19f43eb358a89e7b0 P 61e69695258f48baa984c9d395e8ce3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } attrs { replace: true } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } }
I20260809 15:46:04.345423 24803 ts_tablet_manager.cc:1428] T cee02fff2c9c48a19f43eb358a89e7b0 P 61e69695258f48baa984c9d395e8ce3f: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260809 15:46:04.347200 23546 tablet_copy_service.cc:342] P 65c36d164b0540f099d592f38307ee6c: Request end of tablet copy session 61e69695258f48baa984c9d395e8ce3f-cee02fff2c9c48a19f43eb358a89e7b0 received from {username='slave'} at 127.0.0.1:44018
I20260809 15:46:04.347591 23546 tablet_copy_service.cc:433] P 65c36d164b0540f099d592f38307ee6c: ending tablet copy session 61e69695258f48baa984c9d395e8ce3f-cee02fff2c9c48a19f43eb358a89e7b0 on tablet cee02fff2c9c48a19f43eb358a89e7b0 with peer 61e69695258f48baa984c9d395e8ce3f
I20260809 15:46:04.360460 24803 ts_tablet_manager.cc:927] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 61e69695258f48baa984c9d395e8ce3f: Initiating tablet copy from peer c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:04.361979 24803 tablet_copy_client.cc:316] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 61e69695258f48baa984c9d395e8ce3f: tablet copy: Beginning tablet copy session from remote peer at address 127.20.202.66:36263
I20260809 15:46:04.365017 23424 tablet_copy_service.cc:140] P c0f38c1908414566aa9115bb2cbdd85d: Received BeginTabletCopySession request for tablet b3aa6a9a52804df0ba9b0ca3c82f38c3 from peer 61e69695258f48baa984c9d395e8ce3f ({username='slave'} at 127.0.0.1:52856)
I20260809 15:46:04.365526 23424 tablet_copy_service.cc:161] P c0f38c1908414566aa9115bb2cbdd85d: Beginning new tablet copy session on tablet b3aa6a9a52804df0ba9b0ca3c82f38c3 from peer 61e69695258f48baa984c9d395e8ce3f at {username='slave'} at 127.0.0.1:52856: session id = 61e69695258f48baa984c9d395e8ce3f-b3aa6a9a52804df0ba9b0ca3c82f38c3
I20260809 15:46:04.373063 23424 tablet_copy_source_session.cc:205] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P c0f38c1908414566aa9115bb2cbdd85d: Tablet Copy: opened 0 blocks and 1 log segments
I20260809 15:46:04.375958 24803 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3aa6a9a52804df0ba9b0ca3c82f38c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:46:04.391737 24803 tablet_copy_client.cc:791] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 61e69695258f48baa984c9d395e8ce3f: tablet copy: Starting download of 0 data blocks...
I20260809 15:46:04.392143 24803 tablet_copy_client.cc:655] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 61e69695258f48baa984c9d395e8ce3f: tablet copy: Starting download of 1 WAL segments...
I20260809 15:46:04.394973 24803 tablet_copy_client.cc:531] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 61e69695258f48baa984c9d395e8ce3f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260809 15:46:04.401345 24803 tablet_bootstrap.cc:492] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 61e69695258f48baa984c9d395e8ce3f: Bootstrap starting.
I20260809 15:46:04.427285 24803 tablet_bootstrap.cc:492] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 61e69695258f48baa984c9d395e8ce3f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 15:46:04.427953 24803 tablet_bootstrap.cc:492] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 61e69695258f48baa984c9d395e8ce3f: Bootstrap complete.
I20260809 15:46:04.428449 24803 ts_tablet_manager.cc:1397] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 61e69695258f48baa984c9d395e8ce3f: Time spent bootstrapping tablet: real 0.027s user 0.024s sys 0.004s
I20260809 15:46:04.430593 24803 raft_consensus.cc:348] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 61e69695258f48baa984c9d395e8ce3f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } }
I20260809 15:46:04.431326 24803 raft_consensus.cc:729] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 61e69695258f48baa984c9d395e8ce3f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 61e69695258f48baa984c9d395e8ce3f, State: Initialized, Role: LEARNER
I20260809 15:46:04.431978 24803 consensus_queue.cc:260] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 61e69695258f48baa984c9d395e8ce3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } }
I20260809 15:46:04.435878 24803 ts_tablet_manager.cc:1428] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 61e69695258f48baa984c9d395e8ce3f: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260809 15:46:04.439729 23424 tablet_copy_service.cc:342] P c0f38c1908414566aa9115bb2cbdd85d: Request end of tablet copy session 61e69695258f48baa984c9d395e8ce3f-b3aa6a9a52804df0ba9b0ca3c82f38c3 received from {username='slave'} at 127.0.0.1:52856
I20260809 15:46:04.440083 23424 tablet_copy_service.cc:433] P c0f38c1908414566aa9115bb2cbdd85d: ending tablet copy session 61e69695258f48baa984c9d395e8ce3f-b3aa6a9a52804df0ba9b0ca3c82f38c3 on tablet b3aa6a9a52804df0ba9b0ca3c82f38c3 with peer 61e69695258f48baa984c9d395e8ce3f
I20260809 15:46:04.458627 24803 ts_tablet_manager.cc:927] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 61e69695258f48baa984c9d395e8ce3f: Initiating tablet copy from peer 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459)
I20260809 15:46:04.459913 24803 tablet_copy_client.cc:316] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 61e69695258f48baa984c9d395e8ce3f: tablet copy: Beginning tablet copy session from remote peer at address 127.20.202.65:35459
I20260809 15:46:04.461122 23298 tablet_copy_service.cc:140] P 67b3aeadfa7a432abe8af4d292e04ab9: Received BeginTabletCopySession request for tablet 5712f6b6fffc4cc5a8ddfbe63e4097df from peer 61e69695258f48baa984c9d395e8ce3f ({username='slave'} at 127.0.0.1:55434)
I20260809 15:46:04.461501 23298 tablet_copy_service.cc:161] P 67b3aeadfa7a432abe8af4d292e04ab9: Beginning new tablet copy session on tablet 5712f6b6fffc4cc5a8ddfbe63e4097df from peer 61e69695258f48baa984c9d395e8ce3f at {username='slave'} at 127.0.0.1:55434: session id = 61e69695258f48baa984c9d395e8ce3f-5712f6b6fffc4cc5a8ddfbe63e4097df
I20260809 15:46:04.468741 23298 tablet_copy_source_session.cc:205] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 67b3aeadfa7a432abe8af4d292e04ab9: Tablet Copy: opened 0 blocks and 1 log segments
I20260809 15:46:04.471318 24803 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5712f6b6fffc4cc5a8ddfbe63e4097df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:46:04.483374 24803 tablet_copy_client.cc:791] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 61e69695258f48baa984c9d395e8ce3f: tablet copy: Starting download of 0 data blocks...
I20260809 15:46:04.483830 24803 tablet_copy_client.cc:655] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 61e69695258f48baa984c9d395e8ce3f: tablet copy: Starting download of 1 WAL segments...
I20260809 15:46:04.489660 24803 tablet_copy_client.cc:531] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 61e69695258f48baa984c9d395e8ce3f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260809 15:46:04.497431 24803 tablet_bootstrap.cc:492] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 61e69695258f48baa984c9d395e8ce3f: Bootstrap starting.
I20260809 15:46:04.511194 24803 tablet_bootstrap.cc:492] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 61e69695258f48baa984c9d395e8ce3f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 15:46:04.511793 24803 tablet_bootstrap.cc:492] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 61e69695258f48baa984c9d395e8ce3f: Bootstrap complete.
I20260809 15:46:04.512235 24803 ts_tablet_manager.cc:1397] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 61e69695258f48baa984c9d395e8ce3f: Time spent bootstrapping tablet: real 0.015s user 0.014s sys 0.002s
I20260809 15:46:04.513929 24803 raft_consensus.cc:348] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 61e69695258f48baa984c9d395e8ce3f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } attrs { replace: true } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } }
I20260809 15:46:04.514436 24803 raft_consensus.cc:729] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 61e69695258f48baa984c9d395e8ce3f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 61e69695258f48baa984c9d395e8ce3f, State: Initialized, Role: LEARNER
I20260809 15:46:04.514861 24803 consensus_queue.cc:260] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 61e69695258f48baa984c9d395e8ce3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } attrs { replace: true } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } }
I20260809 15:46:04.516577 24803 ts_tablet_manager.cc:1428] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 61e69695258f48baa984c9d395e8ce3f: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260809 15:46:04.518138 23298 tablet_copy_service.cc:342] P 67b3aeadfa7a432abe8af4d292e04ab9: Request end of tablet copy session 61e69695258f48baa984c9d395e8ce3f-5712f6b6fffc4cc5a8ddfbe63e4097df received from {username='slave'} at 127.0.0.1:55434
I20260809 15:46:04.518427 23298 tablet_copy_service.cc:433] P 67b3aeadfa7a432abe8af4d292e04ab9: ending tablet copy session 61e69695258f48baa984c9d395e8ce3f-5712f6b6fffc4cc5a8ddfbe63e4097df on tablet 5712f6b6fffc4cc5a8ddfbe63e4097df with peer 61e69695258f48baa984c9d395e8ce3f
I20260809 15:46:04.531942 24803 ts_tablet_manager.cc:927] T 0a51c0baaa6e432997c9d3552af4f254 P 61e69695258f48baa984c9d395e8ce3f: Initiating tablet copy from peer 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459)
I20260809 15:46:04.533192 24803 tablet_copy_client.cc:316] T 0a51c0baaa6e432997c9d3552af4f254 P 61e69695258f48baa984c9d395e8ce3f: tablet copy: Beginning tablet copy session from remote peer at address 127.20.202.65:35459
I20260809 15:46:04.534353 23298 tablet_copy_service.cc:140] P 67b3aeadfa7a432abe8af4d292e04ab9: Received BeginTabletCopySession request for tablet 0a51c0baaa6e432997c9d3552af4f254 from peer 61e69695258f48baa984c9d395e8ce3f ({username='slave'} at 127.0.0.1:55434)
I20260809 15:46:04.534780 23298 tablet_copy_service.cc:161] P 67b3aeadfa7a432abe8af4d292e04ab9: Beginning new tablet copy session on tablet 0a51c0baaa6e432997c9d3552af4f254 from peer 61e69695258f48baa984c9d395e8ce3f at {username='slave'} at 127.0.0.1:55434: session id = 61e69695258f48baa984c9d395e8ce3f-0a51c0baaa6e432997c9d3552af4f254
I20260809 15:46:04.541267 23298 tablet_copy_source_session.cc:205] T 0a51c0baaa6e432997c9d3552af4f254 P 67b3aeadfa7a432abe8af4d292e04ab9: Tablet Copy: opened 0 blocks and 1 log segments
I20260809 15:46:04.543591 24803 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a51c0baaa6e432997c9d3552af4f254. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:46:04.553900 24803 tablet_copy_client.cc:791] T 0a51c0baaa6e432997c9d3552af4f254 P 61e69695258f48baa984c9d395e8ce3f: tablet copy: Starting download of 0 data blocks...
I20260809 15:46:04.554323 24803 tablet_copy_client.cc:655] T 0a51c0baaa6e432997c9d3552af4f254 P 61e69695258f48baa984c9d395e8ce3f: tablet copy: Starting download of 1 WAL segments...
I20260809 15:46:04.557300 24803 tablet_copy_client.cc:531] T 0a51c0baaa6e432997c9d3552af4f254 P 61e69695258f48baa984c9d395e8ce3f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260809 15:46:04.563668 24803 tablet_bootstrap.cc:492] T 0a51c0baaa6e432997c9d3552af4f254 P 61e69695258f48baa984c9d395e8ce3f: Bootstrap starting.
I20260809 15:46:04.577207 24803 tablet_bootstrap.cc:492] T 0a51c0baaa6e432997c9d3552af4f254 P 61e69695258f48baa984c9d395e8ce3f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 15:46:04.577785 24803 tablet_bootstrap.cc:492] T 0a51c0baaa6e432997c9d3552af4f254 P 61e69695258f48baa984c9d395e8ce3f: Bootstrap complete.
I20260809 15:46:04.578316 24803 ts_tablet_manager.cc:1397] T 0a51c0baaa6e432997c9d3552af4f254 P 61e69695258f48baa984c9d395e8ce3f: Time spent bootstrapping tablet: real 0.015s user 0.011s sys 0.000s
I20260809 15:46:04.580706 24803 raft_consensus.cc:348] T 0a51c0baaa6e432997c9d3552af4f254 P 61e69695258f48baa984c9d395e8ce3f [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } attrs { replace: true } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } }
I20260809 15:46:04.581290 24803 raft_consensus.cc:729] T 0a51c0baaa6e432997c9d3552af4f254 P 61e69695258f48baa984c9d395e8ce3f [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 61e69695258f48baa984c9d395e8ce3f, State: Initialized, Role: LEARNER
I20260809 15:46:04.581796 24803 consensus_queue.cc:260] T 0a51c0baaa6e432997c9d3552af4f254 P 61e69695258f48baa984c9d395e8ce3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } attrs { replace: true } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } }
I20260809 15:46:04.583642 24803 ts_tablet_manager.cc:1428] T 0a51c0baaa6e432997c9d3552af4f254 P 61e69695258f48baa984c9d395e8ce3f: Time spent starting tablet: real 0.005s user 0.004s sys 0.004s
I20260809 15:46:04.585091 23298 tablet_copy_service.cc:342] P 67b3aeadfa7a432abe8af4d292e04ab9: Request end of tablet copy session 61e69695258f48baa984c9d395e8ce3f-0a51c0baaa6e432997c9d3552af4f254 received from {username='slave'} at 127.0.0.1:55434
I20260809 15:46:04.585428 23298 tablet_copy_service.cc:433] P 67b3aeadfa7a432abe8af4d292e04ab9: ending tablet copy session 61e69695258f48baa984c9d395e8ce3f-0a51c0baaa6e432997c9d3552af4f254 on tablet 0a51c0baaa6e432997c9d3552af4f254 with peer 61e69695258f48baa984c9d395e8ce3f
I20260809 15:46:04.725940 24472 raft_consensus.cc:1203] T cee02fff2c9c48a19f43eb358a89e7b0 P 61e69695258f48baa984c9d395e8ce3f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260809 15:46:04.931334 24472 raft_consensus.cc:1203] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 61e69695258f48baa984c9d395e8ce3f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260809 15:46:05.056398 24472 raft_consensus.cc:1203] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 61e69695258f48baa984c9d395e8ce3f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260809 15:46:05.124723 24472 raft_consensus.cc:1203] T 0a51c0baaa6e432997c9d3552af4f254 P 61e69695258f48baa984c9d395e8ce3f [term 2 LEARNER]: Deduplicated request from leader. Original: 2.1->[2.2-2.2] Dedup: 2.2->[]
I20260809 15:46:05.305724 24827 raft_consensus.cc:1050] T cee02fff2c9c48a19f43eb358a89e7b0 P 65c36d164b0540f099d592f38307ee6c: attempting to promote NON_VOTER 61e69695258f48baa984c9d395e8ce3f to VOTER
I20260809 15:46:05.307832 24827 consensus_queue.cc:237] T cee02fff2c9c48a19f43eb358a89e7b0 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } attrs { replace: true } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:05.313309 23394 raft_consensus.cc:1261] T cee02fff2c9c48a19f43eb358a89e7b0 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 15:46:05.313942 24472 raft_consensus.cc:1261] T cee02fff2c9c48a19f43eb358a89e7b0 P 61e69695258f48baa984c9d395e8ce3f [term 1 LEARNER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 15:46:05.315066 24775 consensus_queue.cc:1035] T cee02fff2c9c48a19f43eb358a89e7b0 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 15:46:05.315994 24839 consensus_queue.cc:1035] T cee02fff2c9c48a19f43eb358a89e7b0 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20260809 15:46:05.321123 23274 raft_consensus.cc:1261] T cee02fff2c9c48a19f43eb358a89e7b0 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 15:46:05.323698 24775 consensus_queue.cc:1035] T cee02fff2c9c48a19f43eb358a89e7b0 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 15:46:05.346849 24775 raft_consensus.cc:2932] T cee02fff2c9c48a19f43eb358a89e7b0 P 65c36d164b0540f099d592f38307ee6c [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } attrs { replace: true } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:05.348428 23396 raft_consensus.cc:2932] T cee02fff2c9c48a19f43eb358a89e7b0 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } attrs { replace: true } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:05.344893 23274 raft_consensus.cc:2932] T cee02fff2c9c48a19f43eb358a89e7b0 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } attrs { replace: true } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:05.359215 24472 raft_consensus.cc:2932] T cee02fff2c9c48a19f43eb358a89e7b0 P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } attrs { replace: true } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:05.368489 23116 catalog_manager.cc:5495] T cee02fff2c9c48a19f43eb358a89e7b0 P 65c36d164b0540f099d592f38307ee6c reported cstate change: config changed from index 2 to 3, 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "65c36d164b0540f099d592f38307ee6c" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 15:46:05.413575 23527 consensus_queue.cc:237] T cee02fff2c9c48a19f43eb358a89e7b0 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:05.422799 23277 raft_consensus.cc:1261] T cee02fff2c9c48a19f43eb358a89e7b0 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260809 15:46:05.423245 24472 raft_consensus.cc:1261] T cee02fff2c9c48a19f43eb358a89e7b0 P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260809 15:46:05.424443 24827 consensus_queue.cc:1035] T cee02fff2c9c48a19f43eb358a89e7b0 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260809 15:46:05.425552 24839 consensus_queue.cc:1035] T cee02fff2c9c48a19f43eb358a89e7b0 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260809 15:46:05.432125 24827 raft_consensus.cc:2932] T cee02fff2c9c48a19f43eb358a89e7b0 P 65c36d164b0540f099d592f38307ee6c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:05.436882 23274 raft_consensus.cc:2932] T cee02fff2c9c48a19f43eb358a89e7b0 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:05.444676 24472 raft_consensus.cc:2932] T cee02fff2c9c48a19f43eb358a89e7b0 P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:05.598361 24837 raft_consensus.cc:1050] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 67b3aeadfa7a432abe8af4d292e04ab9: attempting to promote NON_VOTER 61e69695258f48baa984c9d395e8ce3f to VOTER
I20260809 15:46:05.601212 24837 consensus_queue.cc:237] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } attrs { replace: true } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:05.798885 23095 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet cee02fff2c9c48a19f43eb358a89e7b0 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260809 15:46:05.805543 24472 raft_consensus.cc:1261] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 61e69695258f48baa984c9d395e8ce3f [term 1 LEARNER]: Refusing update from remote peer 67b3aeadfa7a432abe8af4d292e04ab9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 15:46:05.806864 24652 consensus_queue.cc:1035] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 15:46:05.860703 23117 catalog_manager.cc:5495] T cee02fff2c9c48a19f43eb358a89e7b0 P 67b3aeadfa7a432abe8af4d292e04ab9 reported cstate change: config changed from index 3 to 4, VOTER c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66) evicted. New cstate: current_term: 1 leader_uuid: "65c36d164b0540f099d592f38307ee6c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:05.864405 23397 raft_consensus.cc:1261] T 5712f6b6fffc4cc5a8ddfbe63e4097df P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Refusing update from remote peer 67b3aeadfa7a432abe8af4d292e04ab9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 15:46:05.877736 23526 raft_consensus.cc:1261] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Refusing update from remote peer 67b3aeadfa7a432abe8af4d292e04ab9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 15:46:05.880590 24837 consensus_queue.cc:1035] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 15:46:05.888466 24717 consensus_queue.cc:1035] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20260809 15:46:06.179616 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 295693us
I20260809 15:46:06.182478 24831 raft_consensus.cc:1050] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P c0f38c1908414566aa9115bb2cbdd85d: attempting to promote NON_VOTER 61e69695258f48baa984c9d395e8ce3f to VOTER
I20260809 15:46:06.184799 24831 consensus_queue.cc:237] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
W20260809 15:46:06.200667 23470 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 269148us
I20260809 15:46:06.219010 24868 raft_consensus.cc:1050] T 0a51c0baaa6e432997c9d3552af4f254 P 67b3aeadfa7a432abe8af4d292e04ab9: attempting to promote NON_VOTER 61e69695258f48baa984c9d395e8ce3f to VOTER
I20260809 15:46:06.219990 23389 tablet_service.cc:1514] Processing DeleteTablet for tablet cee02fff2c9c48a19f43eb358a89e7b0 with delete_type TABLET_DATA_TOMBSTONED (TS c0f38c1908414566aa9115bb2cbdd85d not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:55312
W20260809 15:46:06.222595 23353 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 45413.3us
W20260809 15:46:06.238574 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 52545us
I20260809 15:46:06.240226 24868 consensus_queue.cc:237] T 0a51c0baaa6e432997c9d3552af4f254 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 0, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } attrs { replace: true } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:06.277750 24895 tablet_replica.cc:330] T cee02fff2c9c48a19f43eb358a89e7b0 P c0f38c1908414566aa9115bb2cbdd85d: stopping tablet replica
I20260809 15:46:06.278764 24895 raft_consensus.cc:2229] T cee02fff2c9c48a19f43eb358a89e7b0 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 15:46:06.278218 24873 raft_consensus.cc:2932] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } attrs { replace: true } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:06.301333 24895 raft_consensus.cc:2258] T cee02fff2c9c48a19f43eb358a89e7b0 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 15:46:06.304512 24895 ts_tablet_manager.cc:1905] T cee02fff2c9c48a19f43eb358a89e7b0 P c0f38c1908414566aa9115bb2cbdd85d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260809 15:46:06.288076 23522 raft_consensus.cc:2932] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } attrs { replace: true } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:06.313800 23527 raft_consensus.cc:1261] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 15:46:06.315462 24472 raft_consensus.cc:1261] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 61e69695258f48baa984c9d395e8ce3f [term 1 LEARNER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 15:46:06.316048 24882 consensus_queue.cc:1035] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 15:46:06.317726 24876 consensus_queue.cc:1035] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 15:46:06.335810 23274 raft_consensus.cc:1261] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 15:46:06.345144 23526 raft_consensus.cc:1261] T 0a51c0baaa6e432997c9d3552af4f254 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Refusing update from remote peer 67b3aeadfa7a432abe8af4d292e04ab9: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260809 15:46:06.359956 23404 raft_consensus.cc:1261] T 0a51c0baaa6e432997c9d3552af4f254 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Refusing update from remote peer 67b3aeadfa7a432abe8af4d292e04ab9: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260809 15:46:06.358371 24475 raft_consensus.cc:1261] T 0a51c0baaa6e432997c9d3552af4f254 P 61e69695258f48baa984c9d395e8ce3f [term 2 LEARNER]: Refusing update from remote peer 67b3aeadfa7a432abe8af4d292e04ab9: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260809 15:46:06.362175 24837 consensus_queue.cc:1035] T 0a51c0baaa6e432997c9d3552af4f254 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 15:46:06.363838 24652 consensus_queue.cc:1035] T 0a51c0baaa6e432997c9d3552af4f254 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 15:46:06.365384 24869 consensus_queue.cc:1035] T 0a51c0baaa6e432997c9d3552af4f254 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260809 15:46:06.373234 23397 raft_consensus.cc:2932] T 5712f6b6fffc4cc5a8ddfbe63e4097df P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } attrs { replace: true } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:06.392015 23117 catalog_manager.cc:5495] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 67b3aeadfa7a432abe8af4d292e04ab9 reported cstate change: config changed from index 2 to 3, 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 15:46:06.369829 24475 raft_consensus.cc:2932] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } attrs { replace: true } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:06.498181 24895 ts_tablet_manager.cc:1918] T cee02fff2c9c48a19f43eb358a89e7b0 P c0f38c1908414566aa9115bb2cbdd85d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260809 15:46:06.512599 24895 log.cc:1196] T cee02fff2c9c48a19f43eb358a89e7b0 P c0f38c1908414566aa9115bb2cbdd85d: Deleting WAL directory at /tmp/dist-test-taskj2AJ_N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786290285951082-21289-0/minicluster-data/ts-1-root/wals/cee02fff2c9c48a19f43eb358a89e7b0
I20260809 15:46:06.514245 24868 raft_consensus.cc:2932] T 0a51c0baaa6e432997c9d3552af4f254 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } attrs { replace: true } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:06.521559 23095 catalog_manager.cc:4841] TS c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263): tablet cee02fff2c9c48a19f43eb358a89e7b0 (table test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]) successfully deleted
I20260809 15:46:06.526455 23404 raft_consensus.cc:2932] T 0a51c0baaa6e432997c9d3552af4f254 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } attrs { replace: true } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
W20260809 15:46:06.530287 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 129026us
I20260809 15:46:06.532461 24472 raft_consensus.cc:2932] T 0a51c0baaa6e432997c9d3552af4f254 P 61e69695258f48baa984c9d395e8ce3f [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } attrs { replace: true } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:06.530921 24471 raft_consensus.cc:2932] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:06.524026 24812 raft_consensus.cc:2932] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P c0f38c1908414566aa9115bb2cbdd85d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:06.568884 24879 consensus_queue.cc:1035] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.016s
I20260809 15:46:06.550156 23519 raft_consensus.cc:2932] T 0a51c0baaa6e432997c9d3552af4f254 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } attrs { replace: true } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:06.610709 23527 raft_consensus.cc:2932] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:06.626977 23273 raft_consensus.cc:2932] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:06.648047 23274 consensus_queue.cc:237] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:06.648231 23116 catalog_manager.cc:5495] T 0a51c0baaa6e432997c9d3552af4f254 P 61e69695258f48baa984c9d395e8ce3f reported cstate change: config changed from index 2 to 3, 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } attrs { replace: true } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:06.661525 23526 raft_consensus.cc:1261] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Refusing update from remote peer 67b3aeadfa7a432abe8af4d292e04ab9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260809 15:46:06.661942 24475 raft_consensus.cc:1261] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Refusing update from remote peer 67b3aeadfa7a432abe8af4d292e04ab9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260809 15:46:06.663134 24717 consensus_queue.cc:1035] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260809 15:46:06.666988 24868 consensus_queue.cc:1035] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260809 15:46:06.671504 23118 catalog_manager.cc:5495] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P c0f38c1908414566aa9115bb2cbdd85d reported cstate change: config changed from index 2 to 3, 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c0f38c1908414566aa9115bb2cbdd85d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 15:46:06.685568 24873 raft_consensus.cc:2932] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:06.686830 24475 raft_consensus.cc:2932] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:06.711613 23094 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 5712f6b6fffc4cc5a8ddfbe63e4097df with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260809 15:46:06.711524 23404 consensus_queue.cc:237] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:06.712924 23519 raft_consensus.cc:2932] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:06.729158 23522 raft_consensus.cc:1261] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260809 15:46:06.736217 24475 raft_consensus.cc:1261] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260809 15:46:06.738626 24879 consensus_queue.cc:1035] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260809 15:46:06.737918 23273 consensus_queue.cc:237] T 0a51c0baaa6e432997c9d3552af4f254 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:06.740494 23116 catalog_manager.cc:5495] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 61e69695258f48baa984c9d395e8ce3f reported cstate change: config changed from index 3 to 4, VOTER c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66) evicted. New cstate: current_term: 1 leader_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:06.747115 24881 consensus_queue.cc:1035] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260809 15:46:06.763238 24876 raft_consensus.cc:2932] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P c0f38c1908414566aa9115bb2cbdd85d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:06.778086 23389 tablet_service.cc:1514] Processing DeleteTablet for tablet 5712f6b6fffc4cc5a8ddfbe63e4097df with delete_type TABLET_DATA_TOMBSTONED (TS c0f38c1908414566aa9115bb2cbdd85d not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:55312
I20260809 15:46:06.779413 24895 tablet_replica.cc:330] T 5712f6b6fffc4cc5a8ddfbe63e4097df P c0f38c1908414566aa9115bb2cbdd85d: stopping tablet replica
I20260809 15:46:06.781172 24895 raft_consensus.cc:2229] T 5712f6b6fffc4cc5a8ddfbe63e4097df P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 15:46:06.782150 24895 raft_consensus.cc:2258] T 5712f6b6fffc4cc5a8ddfbe63e4097df P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 15:46:06.783744 23526 raft_consensus.cc:2932] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:06.818497 24895 ts_tablet_manager.cc:1905] T 5712f6b6fffc4cc5a8ddfbe63e4097df P c0f38c1908414566aa9115bb2cbdd85d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260809 15:46:06.820052 23095 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet b3aa6a9a52804df0ba9b0ca3c82f38c3 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260809 15:46:06.822268 23394 raft_consensus.cc:1261] T 0a51c0baaa6e432997c9d3552af4f254 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Refusing update from remote peer 67b3aeadfa7a432abe8af4d292e04ab9: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260809 15:46:06.824936 24475 raft_consensus.cc:1261] T 0a51c0baaa6e432997c9d3552af4f254 P 61e69695258f48baa984c9d395e8ce3f [term 2 FOLLOWER]: Refusing update from remote peer 67b3aeadfa7a432abe8af4d292e04ab9: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260809 15:46:06.828166 24873 consensus_queue.cc:1035] T 0a51c0baaa6e432997c9d3552af4f254 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260809 15:46:06.829469 24652 consensus_queue.cc:1035] T 0a51c0baaa6e432997c9d3552af4f254 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260809 15:46:06.832530 24471 raft_consensus.cc:2932] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:06.897229 23116 catalog_manager.cc:5495] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 61e69695258f48baa984c9d395e8ce3f reported cstate change: config changed from index 3 to 4, VOTER 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65) evicted. New cstate: current_term: 1 leader_uuid: "c0f38c1908414566aa9115bb2cbdd85d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:06.906798 24895 ts_tablet_manager.cc:1918] T 5712f6b6fffc4cc5a8ddfbe63e4097df P c0f38c1908414566aa9115bb2cbdd85d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260809 15:46:06.907881 24895 log.cc:1196] T 5712f6b6fffc4cc5a8ddfbe63e4097df P c0f38c1908414566aa9115bb2cbdd85d: Deleting WAL directory at /tmp/dist-test-taskj2AJ_N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786290285951082-21289-0/minicluster-data/ts-1-root/wals/5712f6b6fffc4cc5a8ddfbe63e4097df
W20260809 15:46:06.911954 23562 debug-util.cc:398] Leaking SignalData structure 0x7b08001e8c20 after lost signal to thread 21293
I20260809 15:46:06.913297 23095 catalog_manager.cc:4841] TS c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263): tablet 5712f6b6fffc4cc5a8ddfbe63e4097df (table test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]) successfully deleted
W20260809 15:46:06.913615 23562 debug-util.cc:398] Leaking SignalData structure 0x7b08001d97e0 after lost signal to thread 23176
I20260809 15:46:06.920266 23397 raft_consensus.cc:2932] T 0a51c0baaa6e432997c9d3552af4f254 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 65c36d164b0540f099d592f38307ee6c (127.20.202.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
W20260809 15:46:06.923137 23562 debug-util.cc:398] Leaking SignalData structure 0x7b0800263fa0 after lost signal to thread 23320
W20260809 15:46:06.925045 23562 debug-util.cc:398] Leaking SignalData structure 0x7b08002f7160 after lost signal to thread 23443
W20260809 15:46:06.926510 23562 debug-util.cc:398] Leaking SignalData structure 0x7b08002f50a0 after lost signal to thread 23567
W20260809 15:46:06.928417 23562 debug-util.cc:398] Leaking SignalData structure 0x7b08003178a0 after lost signal to thread 24513
I20260809 15:46:06.950704 24471 raft_consensus.cc:2932] T 0a51c0baaa6e432997c9d3552af4f254 P 61e69695258f48baa984c9d395e8ce3f [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 65c36d164b0540f099d592f38307ee6c (127.20.202.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:06.959815 24868 raft_consensus.cc:2932] T 0a51c0baaa6e432997c9d3552af4f254 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 65c36d164b0540f099d592f38307ee6c (127.20.202.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:06.964911 23260 tablet_service.cc:1514] Processing DeleteTablet for tablet b3aa6a9a52804df0ba9b0ca3c82f38c3 with delete_type TABLET_DATA_TOMBSTONED (TS 67b3aeadfa7a432abe8af4d292e04ab9 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:53552
I20260809 15:46:06.983750 23094 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 0a51c0baaa6e432997c9d3552af4f254 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260809 15:46:07.003248 23115 catalog_manager.cc:5495] T 0a51c0baaa6e432997c9d3552af4f254 P 67b3aeadfa7a432abe8af4d292e04ab9 reported cstate change: config changed from index 3 to 4, VOTER 65c36d164b0540f099d592f38307ee6c (127.20.202.67) evicted. New cstate: current_term: 2 leader_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 15:46:07.024995 24923 tablet_replica.cc:330] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 67b3aeadfa7a432abe8af4d292e04ab9: stopping tablet replica
I20260809 15:46:07.027397 24923 raft_consensus.cc:2229] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 15:46:07.028306 24923 raft_consensus.cc:2258] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 15:46:07.044485 23500 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a51c0baaa6e432997c9d3552af4f254 with delete_type TABLET_DATA_TOMBSTONED (TS 65c36d164b0540f099d592f38307ee6c not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:56760
I20260809 15:46:07.046868 24925 tablet_replica.cc:330] T 0a51c0baaa6e432997c9d3552af4f254 P 65c36d164b0540f099d592f38307ee6c: stopping tablet replica
I20260809 15:46:07.052080 24923 ts_tablet_manager.cc:1905] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 67b3aeadfa7a432abe8af4d292e04ab9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260809 15:46:07.075764 24925 raft_consensus.cc:2229] T 0a51c0baaa6e432997c9d3552af4f254 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Raft consensus shutting down.
I20260809 15:46:07.076884 24925 raft_consensus.cc:2258] T 0a51c0baaa6e432997c9d3552af4f254 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Raft consensus is shut down!
I20260809 15:46:07.084121 24925 ts_tablet_manager.cc:1905] T 0a51c0baaa6e432997c9d3552af4f254 P 65c36d164b0540f099d592f38307ee6c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260809 15:46:07.103124 24923 ts_tablet_manager.cc:1918] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 67b3aeadfa7a432abe8af4d292e04ab9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260809 15:46:07.103565 24923 log.cc:1196] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 67b3aeadfa7a432abe8af4d292e04ab9: Deleting WAL directory at /tmp/dist-test-taskj2AJ_N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786290285951082-21289-0/minicluster-data/ts-0-root/wals/b3aa6a9a52804df0ba9b0ca3c82f38c3
I20260809 15:46:07.105320 23094 catalog_manager.cc:4841] TS 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459): tablet b3aa6a9a52804df0ba9b0ca3c82f38c3 (table test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]) successfully deleted
I20260809 15:46:07.122763 24925 ts_tablet_manager.cc:1918] T 0a51c0baaa6e432997c9d3552af4f254 P 65c36d164b0540f099d592f38307ee6c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.3
I20260809 15:46:07.123322 24925 log.cc:1196] T 0a51c0baaa6e432997c9d3552af4f254 P 65c36d164b0540f099d592f38307ee6c: Deleting WAL directory at /tmp/dist-test-taskj2AJ_N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786290285951082-21289-0/minicluster-data/ts-2-root/wals/0a51c0baaa6e432997c9d3552af4f254
I20260809 15:46:07.126322 23095 catalog_manager.cc:4841] TS 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553): tablet 0a51c0baaa6e432997c9d3552af4f254 (table test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]) successfully deleted
I20260809 15:46:08.108891 23522 consensus_queue.cc:237] T e14d118b2eaa42aa8a7cd4025aba7128 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } attrs { replace: true } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } }
I20260809 15:46:08.119550 23394 raft_consensus.cc:1261] T e14d118b2eaa42aa8a7cd4025aba7128 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260809 15:46:08.119688 23468 consensus_peers.cc:487] T e14d118b2eaa42aa8a7cd4025aba7128 P 65c36d164b0540f099d592f38307ee6c -> Peer 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689): Couldn't send request to peer 61e69695258f48baa984c9d395e8ce3f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e14d118b2eaa42aa8a7cd4025aba7128. This is attempt 1: this message will repeat every 5th retry.
I20260809 15:46:08.119866 23274 raft_consensus.cc:1261] T e14d118b2eaa42aa8a7cd4025aba7128 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 15:46:08.122117 24839 consensus_queue.cc:1035] T e14d118b2eaa42aa8a7cd4025aba7128 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260809 15:46:08.122963 24840 consensus_queue.cc:1035] T e14d118b2eaa42aa8a7cd4025aba7128 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260809 15:46:08.279506 23472 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 156450us
I20260809 15:46:08.284894 24927 raft_consensus.cc:2932] T e14d118b2eaa42aa8a7cd4025aba7128 P 65c36d164b0540f099d592f38307ee6c [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } attrs { replace: true } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } } }
I20260809 15:46:08.289785 23282 raft_consensus.cc:2932] T e14d118b2eaa42aa8a7cd4025aba7128 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } attrs { replace: true } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } } }
I20260809 15:46:08.306638 23404 raft_consensus.cc:2932] T e14d118b2eaa42aa8a7cd4025aba7128 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } attrs { replace: true } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } } }
I20260809 15:46:08.318791 23117 catalog_manager.cc:5495] T e14d118b2eaa42aa8a7cd4025aba7128 P 67b3aeadfa7a432abe8af4d292e04ab9 reported cstate change: config changed from index -1 to 2, NON_VOTER 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) added. New cstate: current_term: 1 leader_uuid: "65c36d164b0540f099d592f38307ee6c" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } attrs { replace: true } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } } }
I20260809 15:46:08.345183 23273 consensus_queue.cc:237] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } attrs { replace: true } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } }
I20260809 15:46:08.356989 23397 raft_consensus.cc:1261] T 8e4ffcf83c234ae7910ddef3ec0b4bda P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Refusing update from remote peer 67b3aeadfa7a432abe8af4d292e04ab9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 15:46:08.357362 23522 raft_consensus.cc:1261] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Refusing update from remote peer 67b3aeadfa7a432abe8af4d292e04ab9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260809 15:46:08.358397 23223 consensus_peers.cc:487] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 67b3aeadfa7a432abe8af4d292e04ab9 -> Peer 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689): Couldn't send request to peer 61e69695258f48baa984c9d395e8ce3f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8e4ffcf83c234ae7910ddef3ec0b4bda. This is attempt 1: this message will repeat every 5th retry.
I20260809 15:46:08.358659 24868 consensus_queue.cc:1035] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 15:46:08.360445 24868 consensus_queue.cc:1035] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 15:46:08.381225 24868 raft_consensus.cc:2932] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } attrs { replace: true } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } } }
I20260809 15:46:08.381497 23522 raft_consensus.cc:2932] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } attrs { replace: true } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } } }
I20260809 15:46:08.385245 23404 raft_consensus.cc:2932] T 8e4ffcf83c234ae7910ddef3ec0b4bda P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } attrs { replace: true } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } } }
I20260809 15:46:08.397805 23394 consensus_queue.cc:237] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } }
I20260809 15:46:08.407403 23115 catalog_manager.cc:5495] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 65c36d164b0540f099d592f38307ee6c reported cstate change: config changed from index -1 to 2, NON_VOTER 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) added. New cstate: current_term: 1 leader_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } attrs { replace: true } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } } }
W20260809 15:46:08.425806 23345 consensus_peers.cc:487] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P c0f38c1908414566aa9115bb2cbdd85d -> Peer 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689): Couldn't send request to peer 61e69695258f48baa984c9d395e8ce3f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5ad8a3b25d7b49e5a2142d5709edd0c7. This is attempt 1: this message will repeat every 5th retry.
I20260809 15:46:08.427191 23273 raft_consensus.cc:1261] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 15:46:08.428647 24876 consensus_queue.cc:1035] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260809 15:46:08.429541 23522 raft_consensus.cc:1261] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 15:46:08.430846 24876 consensus_queue.cc:1035] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 15:46:08.449561 24932 raft_consensus.cc:2932] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P c0f38c1908414566aa9115bb2cbdd85d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } } }
I20260809 15:46:08.452997 23522 raft_consensus.cc:2932] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } } }
I20260809 15:46:08.465862 23273 raft_consensus.cc:2932] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } } }
I20260809 15:46:08.468166 23394 consensus_queue.cc:237] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } }
I20260809 15:46:08.468598 23116 catalog_manager.cc:5495] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P c0f38c1908414566aa9115bb2cbdd85d reported cstate change: config changed from index -1 to 2, NON_VOTER 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) added. New cstate: current_term: 1 leader_uuid: "c0f38c1908414566aa9115bb2cbdd85d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260809 15:46:08.483588 23526 raft_consensus.cc:1261] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 15:46:08.484090 23273 raft_consensus.cc:1261] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 15:46:08.484874 24932 consensus_queue.cc:1035] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 15:46:08.486485 24876 consensus_queue.cc:1035] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260809 15:46:08.496552 23345 consensus_peers.cc:487] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d -> Peer 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689): Couldn't send request to peer 61e69695258f48baa984c9d395e8ce3f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2a823d35f4f348d1b52ba9e6d80b00d2. This is attempt 1: this message will repeat every 5th retry.
I20260809 15:46:08.504853 24834 raft_consensus.cc:2932] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } } }
I20260809 15:46:08.506858 23522 raft_consensus.cc:2932] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } } }
I20260809 15:46:08.517058 23273 raft_consensus.cc:2932] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } } }
I20260809 15:46:08.524236 23116 catalog_manager.cc:5495] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 65c36d164b0540f099d592f38307ee6c reported cstate change: config changed from index -1 to 2, NON_VOTER 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) added. New cstate: current_term: 1 leader_uuid: "c0f38c1908414566aa9115bb2cbdd85d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } } }
I20260809 15:46:08.537954 24956 ts_tablet_manager.cc:927] T e14d118b2eaa42aa8a7cd4025aba7128 P 61e69695258f48baa984c9d395e8ce3f: Initiating tablet copy from peer 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:08.580943 24956 tablet_copy_client.cc:316] T e14d118b2eaa42aa8a7cd4025aba7128 P 61e69695258f48baa984c9d395e8ce3f: tablet copy: Beginning tablet copy session from remote peer at address 127.20.202.67:40553
I20260809 15:46:08.591321 23546 tablet_copy_service.cc:140] P 65c36d164b0540f099d592f38307ee6c: Received BeginTabletCopySession request for tablet e14d118b2eaa42aa8a7cd4025aba7128 from peer 61e69695258f48baa984c9d395e8ce3f ({username='slave'} at 127.0.0.1:44018)
I20260809 15:46:08.591809 23546 tablet_copy_service.cc:161] P 65c36d164b0540f099d592f38307ee6c: Beginning new tablet copy session on tablet e14d118b2eaa42aa8a7cd4025aba7128 from peer 61e69695258f48baa984c9d395e8ce3f at {username='slave'} at 127.0.0.1:44018: session id = 61e69695258f48baa984c9d395e8ce3f-e14d118b2eaa42aa8a7cd4025aba7128
I20260809 15:46:08.598474 23546 tablet_copy_source_session.cc:205] T e14d118b2eaa42aa8a7cd4025aba7128 P 65c36d164b0540f099d592f38307ee6c: Tablet Copy: opened 0 blocks and 1 log segments
I20260809 15:46:08.601255 24956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e14d118b2eaa42aa8a7cd4025aba7128. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:46:08.612375 24956 tablet_copy_client.cc:791] T e14d118b2eaa42aa8a7cd4025aba7128 P 61e69695258f48baa984c9d395e8ce3f: tablet copy: Starting download of 0 data blocks...
I20260809 15:46:08.612910 24956 tablet_copy_client.cc:655] T e14d118b2eaa42aa8a7cd4025aba7128 P 61e69695258f48baa984c9d395e8ce3f: tablet copy: Starting download of 1 WAL segments...
I20260809 15:46:08.615927 24956 tablet_copy_client.cc:531] T e14d118b2eaa42aa8a7cd4025aba7128 P 61e69695258f48baa984c9d395e8ce3f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260809 15:46:08.623065 24956 tablet_bootstrap.cc:492] T e14d118b2eaa42aa8a7cd4025aba7128 P 61e69695258f48baa984c9d395e8ce3f: Bootstrap starting.
I20260809 15:46:08.642354 24956 tablet_bootstrap.cc:492] T e14d118b2eaa42aa8a7cd4025aba7128 P 61e69695258f48baa984c9d395e8ce3f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 15:46:08.643124 24956 tablet_bootstrap.cc:492] T e14d118b2eaa42aa8a7cd4025aba7128 P 61e69695258f48baa984c9d395e8ce3f: Bootstrap complete.
I20260809 15:46:08.643743 24956 ts_tablet_manager.cc:1397] T e14d118b2eaa42aa8a7cd4025aba7128 P 61e69695258f48baa984c9d395e8ce3f: Time spent bootstrapping tablet: real 0.021s user 0.022s sys 0.000s
I20260809 15:46:08.646390 24956 raft_consensus.cc:348] T e14d118b2eaa42aa8a7cd4025aba7128 P 61e69695258f48baa984c9d395e8ce3f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } attrs { replace: true } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } }
I20260809 15:46:08.647204 24956 raft_consensus.cc:729] T e14d118b2eaa42aa8a7cd4025aba7128 P 61e69695258f48baa984c9d395e8ce3f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 61e69695258f48baa984c9d395e8ce3f, State: Initialized, Role: LEARNER
I20260809 15:46:08.647835 24956 consensus_queue.cc:260] T e14d118b2eaa42aa8a7cd4025aba7128 P 61e69695258f48baa984c9d395e8ce3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } attrs { replace: true } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } }
I20260809 15:46:08.652814 24956 ts_tablet_manager.cc:1428] T e14d118b2eaa42aa8a7cd4025aba7128 P 61e69695258f48baa984c9d395e8ce3f: Time spent starting tablet: real 0.009s user 0.008s sys 0.000s
I20260809 15:46:08.659698 23546 tablet_copy_service.cc:342] P 65c36d164b0540f099d592f38307ee6c: Request end of tablet copy session 61e69695258f48baa984c9d395e8ce3f-e14d118b2eaa42aa8a7cd4025aba7128 received from {username='slave'} at 127.0.0.1:44018
I20260809 15:46:08.660091 23546 tablet_copy_service.cc:433] P 65c36d164b0540f099d592f38307ee6c: ending tablet copy session 61e69695258f48baa984c9d395e8ce3f-e14d118b2eaa42aa8a7cd4025aba7128 on tablet e14d118b2eaa42aa8a7cd4025aba7128 with peer 61e69695258f48baa984c9d395e8ce3f
I20260809 15:46:08.846926 24956 ts_tablet_manager.cc:927] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 61e69695258f48baa984c9d395e8ce3f: Initiating tablet copy from peer c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:08.860272 24956 tablet_copy_client.cc:316] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 61e69695258f48baa984c9d395e8ce3f: tablet copy: Beginning tablet copy session from remote peer at address 127.20.202.66:36263
I20260809 15:46:08.864984 23424 tablet_copy_service.cc:140] P c0f38c1908414566aa9115bb2cbdd85d: Received BeginTabletCopySession request for tablet 5ad8a3b25d7b49e5a2142d5709edd0c7 from peer 61e69695258f48baa984c9d395e8ce3f ({username='slave'} at 127.0.0.1:52856)
I20260809 15:46:08.865505 23424 tablet_copy_service.cc:161] P c0f38c1908414566aa9115bb2cbdd85d: Beginning new tablet copy session on tablet 5ad8a3b25d7b49e5a2142d5709edd0c7 from peer 61e69695258f48baa984c9d395e8ce3f at {username='slave'} at 127.0.0.1:52856: session id = 61e69695258f48baa984c9d395e8ce3f-5ad8a3b25d7b49e5a2142d5709edd0c7
I20260809 15:46:08.873337 23424 tablet_copy_source_session.cc:205] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P c0f38c1908414566aa9115bb2cbdd85d: Tablet Copy: opened 0 blocks and 1 log segments
I20260809 15:46:08.877079 24956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ad8a3b25d7b49e5a2142d5709edd0c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:46:08.892374 24956 tablet_copy_client.cc:791] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 61e69695258f48baa984c9d395e8ce3f: tablet copy: Starting download of 0 data blocks...
I20260809 15:46:08.892871 24956 tablet_copy_client.cc:655] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 61e69695258f48baa984c9d395e8ce3f: tablet copy: Starting download of 1 WAL segments...
I20260809 15:46:08.898162 24956 tablet_copy_client.cc:531] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 61e69695258f48baa984c9d395e8ce3f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260809 15:46:08.905882 24956 tablet_bootstrap.cc:492] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 61e69695258f48baa984c9d395e8ce3f: Bootstrap starting.
I20260809 15:46:08.931340 24956 tablet_bootstrap.cc:492] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 61e69695258f48baa984c9d395e8ce3f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 15:46:08.932148 24956 tablet_bootstrap.cc:492] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 61e69695258f48baa984c9d395e8ce3f: Bootstrap complete.
I20260809 15:46:08.932828 24956 ts_tablet_manager.cc:1397] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 61e69695258f48baa984c9d395e8ce3f: Time spent bootstrapping tablet: real 0.027s user 0.018s sys 0.008s
I20260809 15:46:08.935524 24956 raft_consensus.cc:348] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 61e69695258f48baa984c9d395e8ce3f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } }
I20260809 15:46:08.936324 24956 raft_consensus.cc:729] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 61e69695258f48baa984c9d395e8ce3f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 61e69695258f48baa984c9d395e8ce3f, State: Initialized, Role: LEARNER
I20260809 15:46:08.936914 24956 consensus_queue.cc:260] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 61e69695258f48baa984c9d395e8ce3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } }
I20260809 15:46:08.937816 24471 raft_consensus.cc:1203] T e14d118b2eaa42aa8a7cd4025aba7128 P 61e69695258f48baa984c9d395e8ce3f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260809 15:46:08.939878 24956 ts_tablet_manager.cc:1428] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 61e69695258f48baa984c9d395e8ce3f: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260809 15:46:08.945209 23424 tablet_copy_service.cc:342] P c0f38c1908414566aa9115bb2cbdd85d: Request end of tablet copy session 61e69695258f48baa984c9d395e8ce3f-5ad8a3b25d7b49e5a2142d5709edd0c7 received from {username='slave'} at 127.0.0.1:52856
I20260809 15:46:08.970495 23424 tablet_copy_service.cc:433] P c0f38c1908414566aa9115bb2cbdd85d: ending tablet copy session 61e69695258f48baa984c9d395e8ce3f-5ad8a3b25d7b49e5a2142d5709edd0c7 on tablet 5ad8a3b25d7b49e5a2142d5709edd0c7 with peer 61e69695258f48baa984c9d395e8ce3f
I20260809 15:46:08.973491 24966 ts_tablet_manager.cc:927] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 61e69695258f48baa984c9d395e8ce3f: Initiating tablet copy from peer 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459)
I20260809 15:46:08.974922 24966 tablet_copy_client.cc:316] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 61e69695258f48baa984c9d395e8ce3f: tablet copy: Beginning tablet copy session from remote peer at address 127.20.202.65:35459
I20260809 15:46:08.977068 23298 tablet_copy_service.cc:140] P 67b3aeadfa7a432abe8af4d292e04ab9: Received BeginTabletCopySession request for tablet 8e4ffcf83c234ae7910ddef3ec0b4bda from peer 61e69695258f48baa984c9d395e8ce3f ({username='slave'} at 127.0.0.1:55434)
I20260809 15:46:08.977550 23298 tablet_copy_service.cc:161] P 67b3aeadfa7a432abe8af4d292e04ab9: Beginning new tablet copy session on tablet 8e4ffcf83c234ae7910ddef3ec0b4bda from peer 61e69695258f48baa984c9d395e8ce3f at {username='slave'} at 127.0.0.1:55434: session id = 61e69695258f48baa984c9d395e8ce3f-8e4ffcf83c234ae7910ddef3ec0b4bda
I20260809 15:46:08.986800 23298 tablet_copy_source_session.cc:205] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 67b3aeadfa7a432abe8af4d292e04ab9: Tablet Copy: opened 0 blocks and 1 log segments
I20260809 15:46:08.989703 24966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e4ffcf83c234ae7910ddef3ec0b4bda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:46:09.004487 24966 tablet_copy_client.cc:791] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 61e69695258f48baa984c9d395e8ce3f: tablet copy: Starting download of 0 data blocks...
I20260809 15:46:09.004967 24966 tablet_copy_client.cc:655] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 61e69695258f48baa984c9d395e8ce3f: tablet copy: Starting download of 1 WAL segments...
I20260809 15:46:09.010397 24966 tablet_copy_client.cc:531] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 61e69695258f48baa984c9d395e8ce3f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260809 15:46:09.019464 24966 tablet_bootstrap.cc:492] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 61e69695258f48baa984c9d395e8ce3f: Bootstrap starting.
I20260809 15:46:09.042238 24966 tablet_bootstrap.cc:492] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 61e69695258f48baa984c9d395e8ce3f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 15:46:09.042980 24966 tablet_bootstrap.cc:492] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 61e69695258f48baa984c9d395e8ce3f: Bootstrap complete.
I20260809 15:46:09.043577 24966 ts_tablet_manager.cc:1397] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 61e69695258f48baa984c9d395e8ce3f: Time spent bootstrapping tablet: real 0.024s user 0.014s sys 0.004s
I20260809 15:46:09.045697 24966 raft_consensus.cc:348] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 61e69695258f48baa984c9d395e8ce3f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } attrs { replace: true } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } }
I20260809 15:46:09.046535 24966 raft_consensus.cc:729] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 61e69695258f48baa984c9d395e8ce3f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 61e69695258f48baa984c9d395e8ce3f, State: Initialized, Role: LEARNER
I20260809 15:46:09.047132 24966 consensus_queue.cc:260] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 61e69695258f48baa984c9d395e8ce3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } attrs { replace: true } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } }
I20260809 15:46:09.055541 24966 ts_tablet_manager.cc:1428] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 61e69695258f48baa984c9d395e8ce3f: Time spent starting tablet: real 0.012s user 0.005s sys 0.000s
I20260809 15:46:09.064379 23298 tablet_copy_service.cc:342] P 67b3aeadfa7a432abe8af4d292e04ab9: Request end of tablet copy session 61e69695258f48baa984c9d395e8ce3f-8e4ffcf83c234ae7910ddef3ec0b4bda received from {username='slave'} at 127.0.0.1:55434
I20260809 15:46:09.064739 23298 tablet_copy_service.cc:433] P 67b3aeadfa7a432abe8af4d292e04ab9: ending tablet copy session 61e69695258f48baa984c9d395e8ce3f-8e4ffcf83c234ae7910ddef3ec0b4bda on tablet 8e4ffcf83c234ae7910ddef3ec0b4bda with peer 61e69695258f48baa984c9d395e8ce3f
I20260809 15:46:09.082396 24966 ts_tablet_manager.cc:927] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 61e69695258f48baa984c9d395e8ce3f: Initiating tablet copy from peer c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:09.084648 24966 tablet_copy_client.cc:316] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 61e69695258f48baa984c9d395e8ce3f: tablet copy: Beginning tablet copy session from remote peer at address 127.20.202.66:36263
I20260809 15:46:09.086022 23424 tablet_copy_service.cc:140] P c0f38c1908414566aa9115bb2cbdd85d: Received BeginTabletCopySession request for tablet 2a823d35f4f348d1b52ba9e6d80b00d2 from peer 61e69695258f48baa984c9d395e8ce3f ({username='slave'} at 127.0.0.1:52856)
I20260809 15:46:09.086567 23424 tablet_copy_service.cc:161] P c0f38c1908414566aa9115bb2cbdd85d: Beginning new tablet copy session on tablet 2a823d35f4f348d1b52ba9e6d80b00d2 from peer 61e69695258f48baa984c9d395e8ce3f at {username='slave'} at 127.0.0.1:52856: session id = 61e69695258f48baa984c9d395e8ce3f-2a823d35f4f348d1b52ba9e6d80b00d2
I20260809 15:46:09.095017 23424 tablet_copy_source_session.cc:205] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d: Tablet Copy: opened 0 blocks and 1 log segments
I20260809 15:46:09.108459 24966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a823d35f4f348d1b52ba9e6d80b00d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:46:09.124045 24966 tablet_copy_client.cc:791] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 61e69695258f48baa984c9d395e8ce3f: tablet copy: Starting download of 0 data blocks...
I20260809 15:46:09.124514 24966 tablet_copy_client.cc:655] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 61e69695258f48baa984c9d395e8ce3f: tablet copy: Starting download of 1 WAL segments...
I20260809 15:46:09.128854 24966 tablet_copy_client.cc:531] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 61e69695258f48baa984c9d395e8ce3f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260809 15:46:09.135864 24966 tablet_bootstrap.cc:492] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 61e69695258f48baa984c9d395e8ce3f: Bootstrap starting.
I20260809 15:46:09.157769 24966 tablet_bootstrap.cc:492] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 61e69695258f48baa984c9d395e8ce3f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 15:46:09.158593 24966 tablet_bootstrap.cc:492] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 61e69695258f48baa984c9d395e8ce3f: Bootstrap complete.
I20260809 15:46:09.159169 24966 ts_tablet_manager.cc:1397] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 61e69695258f48baa984c9d395e8ce3f: Time spent bootstrapping tablet: real 0.023s user 0.014s sys 0.004s
I20260809 15:46:09.161746 24966 raft_consensus.cc:348] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 61e69695258f48baa984c9d395e8ce3f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } }
I20260809 15:46:09.162549 24966 raft_consensus.cc:729] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 61e69695258f48baa984c9d395e8ce3f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 61e69695258f48baa984c9d395e8ce3f, State: Initialized, Role: LEARNER
I20260809 15:46:09.164985 24966 consensus_queue.cc:260] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 61e69695258f48baa984c9d395e8ce3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } }
I20260809 15:46:09.167639 24966 ts_tablet_manager.cc:1428] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 61e69695258f48baa984c9d395e8ce3f: Time spent starting tablet: real 0.008s user 0.003s sys 0.003s
I20260809 15:46:09.170984 23424 tablet_copy_service.cc:342] P c0f38c1908414566aa9115bb2cbdd85d: Request end of tablet copy session 61e69695258f48baa984c9d395e8ce3f-2a823d35f4f348d1b52ba9e6d80b00d2 received from {username='slave'} at 127.0.0.1:52856
I20260809 15:46:09.171446 23424 tablet_copy_service.cc:433] P c0f38c1908414566aa9115bb2cbdd85d: ending tablet copy session 61e69695258f48baa984c9d395e8ce3f-2a823d35f4f348d1b52ba9e6d80b00d2 on tablet 2a823d35f4f348d1b52ba9e6d80b00d2 with peer 61e69695258f48baa984c9d395e8ce3f
I20260809 15:46:09.256585 24475 raft_consensus.cc:1203] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 61e69695258f48baa984c9d395e8ce3f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260809 15:46:09.338121 24840 raft_consensus.cc:1050] T e14d118b2eaa42aa8a7cd4025aba7128 P 65c36d164b0540f099d592f38307ee6c: attempting to promote NON_VOTER 61e69695258f48baa984c9d395e8ce3f to VOTER
I20260809 15:46:09.340440 24840 consensus_queue.cc:237] T e14d118b2eaa42aa8a7cd4025aba7128 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } attrs { replace: true } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:09.346846 24471 raft_consensus.cc:1261] T e14d118b2eaa42aa8a7cd4025aba7128 P 61e69695258f48baa984c9d395e8ce3f [term 1 LEARNER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 15:46:09.347949 23273 raft_consensus.cc:1261] T e14d118b2eaa42aa8a7cd4025aba7128 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 15:46:09.348101 24927 consensus_queue.cc:1035] T e14d118b2eaa42aa8a7cd4025aba7128 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 15:46:09.350689 23397 raft_consensus.cc:1261] T e14d118b2eaa42aa8a7cd4025aba7128 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 15:46:09.361780 24980 consensus_queue.cc:1035] T e14d118b2eaa42aa8a7cd4025aba7128 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 15:46:09.371115 24840 consensus_queue.cc:1035] T e14d118b2eaa42aa8a7cd4025aba7128 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260809 15:46:09.388571 24475 raft_consensus.cc:1203] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 61e69695258f48baa984c9d395e8ce3f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260809 15:46:09.407387 24927 raft_consensus.cc:2932] T e14d118b2eaa42aa8a7cd4025aba7128 P 65c36d164b0540f099d592f38307ee6c [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } attrs { replace: true } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:09.408883 24475 raft_consensus.cc:2932] T e14d118b2eaa42aa8a7cd4025aba7128 P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } attrs { replace: true } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:09.424256 23394 raft_consensus.cc:2932] T e14d118b2eaa42aa8a7cd4025aba7128 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } attrs { replace: true } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:09.425074 23273 raft_consensus.cc:2932] T e14d118b2eaa42aa8a7cd4025aba7128 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } attrs { replace: true } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:09.435879 23117 catalog_manager.cc:5495] T e14d118b2eaa42aa8a7cd4025aba7128 P 65c36d164b0540f099d592f38307ee6c reported cstate change: config changed from index 2 to 3, 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "65c36d164b0540f099d592f38307ee6c" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 15:46:09.478416 23527 consensus_queue.cc:237] T e14d118b2eaa42aa8a7cd4025aba7128 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:09.504585 23282 raft_consensus.cc:1261] T e14d118b2eaa42aa8a7cd4025aba7128 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260809 15:46:09.505919 24981 consensus_queue.cc:1035] T e14d118b2eaa42aa8a7cd4025aba7128 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260809 15:46:09.512625 24927 raft_consensus.cc:2932] T e14d118b2eaa42aa8a7cd4025aba7128 P 65c36d164b0540f099d592f38307ee6c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:09.515949 24475 raft_consensus.cc:1261] T e14d118b2eaa42aa8a7cd4025aba7128 P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260809 15:46:09.518573 24840 consensus_queue.cc:1035] T e14d118b2eaa42aa8a7cd4025aba7128 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260809 15:46:09.522713 24471 raft_consensus.cc:2932] T e14d118b2eaa42aa8a7cd4025aba7128 P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:09.523679 23282 raft_consensus.cc:2932] T e14d118b2eaa42aa8a7cd4025aba7128 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:09.527527 23095 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet e14d118b2eaa42aa8a7cd4025aba7128 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260809 15:46:09.541802 23118 catalog_manager.cc:5495] T e14d118b2eaa42aa8a7cd4025aba7128 P 65c36d164b0540f099d592f38307ee6c reported cstate change: config changed from index 3 to 4, VOTER c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66) evicted. New cstate: current_term: 1 leader_uuid: "65c36d164b0540f099d592f38307ee6c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260809 15:46:09.564232 23389 tablet_service.cc:1514] Processing DeleteTablet for tablet e14d118b2eaa42aa8a7cd4025aba7128 with delete_type TABLET_DATA_TOMBSTONED (TS c0f38c1908414566aa9115bb2cbdd85d not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:55312
I20260809 15:46:09.571065 24993 tablet_replica.cc:330] T e14d118b2eaa42aa8a7cd4025aba7128 P c0f38c1908414566aa9115bb2cbdd85d: stopping tablet replica
I20260809 15:46:09.572232 24993 raft_consensus.cc:2229] T e14d118b2eaa42aa8a7cd4025aba7128 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 15:46:09.573855 24993 raft_consensus.cc:2258] T e14d118b2eaa42aa8a7cd4025aba7128 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 15:46:09.575032 24475 raft_consensus.cc:1203] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 61e69695258f48baa984c9d395e8ce3f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260809 15:46:09.577194 24993 ts_tablet_manager.cc:1905] T e14d118b2eaa42aa8a7cd4025aba7128 P c0f38c1908414566aa9115bb2cbdd85d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260809 15:46:09.597150 24993 ts_tablet_manager.cc:1918] T e14d118b2eaa42aa8a7cd4025aba7128 P c0f38c1908414566aa9115bb2cbdd85d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260809 15:46:09.597584 24993 log.cc:1196] T e14d118b2eaa42aa8a7cd4025aba7128 P c0f38c1908414566aa9115bb2cbdd85d: Deleting WAL directory at /tmp/dist-test-taskj2AJ_N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786290285951082-21289-0/minicluster-data/ts-1-root/wals/e14d118b2eaa42aa8a7cd4025aba7128
I20260809 15:46:09.599311 23095 catalog_manager.cc:4841] TS c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263): tablet e14d118b2eaa42aa8a7cd4025aba7128 (table test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]) successfully deleted
I20260809 15:46:09.838016 24882 raft_consensus.cc:1050] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P c0f38c1908414566aa9115bb2cbdd85d: attempting to promote NON_VOTER 61e69695258f48baa984c9d395e8ce3f to VOTER
I20260809 15:46:09.840327 24882 consensus_queue.cc:237] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:09.852044 23527 raft_consensus.cc:1261] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 15:46:09.853487 24876 consensus_queue.cc:1035] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 15:46:09.852280 23273 raft_consensus.cc:1261] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 15:46:09.864794 24475 raft_consensus.cc:1261] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 61e69695258f48baa984c9d395e8ce3f [term 1 LEARNER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 15:46:09.868408 25001 consensus_queue.cc:1035] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 15:46:09.869590 24882 consensus_queue.cc:1035] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 15:46:09.885619 24876 raft_consensus.cc:2932] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P c0f38c1908414566aa9115bb2cbdd85d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:09.888155 24475 raft_consensus.cc:2932] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:09.897985 23273 raft_consensus.cc:2932] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:09.903254 23527 raft_consensus.cc:2932] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:09.905016 23116 catalog_manager.cc:5495] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 61e69695258f48baa984c9d395e8ce3f reported cstate change: config changed from index 2 to 3, 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c0f38c1908414566aa9115bb2cbdd85d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:09.926090 23404 consensus_queue.cc:237] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:09.938629 24475 raft_consensus.cc:1261] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260809 15:46:09.940284 25000 consensus_queue.cc:1035] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260809 15:46:09.948249 23522 raft_consensus.cc:1261] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260809 15:46:09.958261 24876 consensus_queue.cc:1035] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260809 15:46:09.959157 24945 raft_consensus.cc:1050] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 67b3aeadfa7a432abe8af4d292e04ab9: attempting to promote NON_VOTER 61e69695258f48baa984c9d395e8ce3f to VOTER
I20260809 15:46:09.965152 24945 consensus_queue.cc:237] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } attrs { replace: true } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:09.967412 24876 raft_consensus.cc:2932] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P c0f38c1908414566aa9115bb2cbdd85d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:09.972900 23527 raft_consensus.cc:2932] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:09.977357 24472 raft_consensus.cc:2932] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:09.988835 23095 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 5ad8a3b25d7b49e5a2142d5709edd0c7 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260809 15:46:10.000538 23116 catalog_manager.cc:5495] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 61e69695258f48baa984c9d395e8ce3f reported cstate change: config changed from index 3 to 4, VOTER 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65) evicted. New cstate: current_term: 1 leader_uuid: "c0f38c1908414566aa9115bb2cbdd85d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:10.031875 24882 raft_consensus.cc:1050] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d: attempting to promote NON_VOTER 61e69695258f48baa984c9d395e8ce3f to VOTER
I20260809 15:46:10.034004 24882 consensus_queue.cc:237] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:10.044512 23260 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ad8a3b25d7b49e5a2142d5709edd0c7 with delete_type TABLET_DATA_TOMBSTONED (TS 67b3aeadfa7a432abe8af4d292e04ab9 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:53552
I20260809 15:46:10.052790 23522 raft_consensus.cc:1261] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 15:46:10.054116 24472 raft_consensus.cc:1261] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 61e69695258f48baa984c9d395e8ce3f [term 1 LEARNER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 15:46:10.055043 24882 consensus_queue.cc:1035] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260809 15:46:10.056003 25001 consensus_queue.cc:1035] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260809 15:46:10.066053 25011 tablet_replica.cc:330] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 67b3aeadfa7a432abe8af4d292e04ab9: stopping tablet replica
I20260809 15:46:10.067057 25011 raft_consensus.cc:2229] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 15:46:10.067715 25011 raft_consensus.cc:2258] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 15:46:10.068109 23522 raft_consensus.cc:1261] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Refusing update from remote peer 67b3aeadfa7a432abe8af4d292e04ab9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 15:46:10.071875 24475 raft_consensus.cc:1261] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 61e69695258f48baa984c9d395e8ce3f [term 1 LEARNER]: Refusing update from remote peer 67b3aeadfa7a432abe8af4d292e04ab9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 15:46:10.074980 24930 consensus_queue.cc:1035] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 15:46:10.079746 23404 raft_consensus.cc:1261] T 8e4ffcf83c234ae7910ddef3ec0b4bda P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Refusing update from remote peer 67b3aeadfa7a432abe8af4d292e04ab9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 15:46:10.085516 25007 consensus_queue.cc:1035] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260809 15:46:10.090062 24945 consensus_queue.cc:1035] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260809 15:46:10.114446 24472 raft_consensus.cc:2932] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:10.105590 24882 raft_consensus.cc:2932] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:10.121507 25011 ts_tablet_manager.cc:1905] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 67b3aeadfa7a432abe8af4d292e04ab9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260809 15:46:10.120837 23522 raft_consensus.cc:2932] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:10.140904 23118 catalog_manager.cc:5495] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 65c36d164b0540f099d592f38307ee6c reported cstate change: config changed from index 2 to 3, 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c0f38c1908414566aa9115bb2cbdd85d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:10.160041 23273 raft_consensus.cc:2932] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:10.162133 24471 raft_consensus.cc:2932] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } attrs { replace: true } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:10.162789 23527 raft_consensus.cc:2932] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } attrs { replace: true } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:10.159919 24945 raft_consensus.cc:2932] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } attrs { replace: true } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:10.182245 23118 catalog_manager.cc:5495] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 65c36d164b0540f099d592f38307ee6c reported cstate change: config changed from index 2 to 3, 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } attrs { replace: true } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:10.202610 23394 consensus_queue.cc:237] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:10.214033 24472 raft_consensus.cc:1261] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260809 15:46:10.215968 24932 consensus_queue.cc:1035] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260809 15:46:10.215230 23394 raft_consensus.cc:2932] T 8e4ffcf83c234ae7910ddef3ec0b4bda P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } attrs { replace: true } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:10.222124 25011 ts_tablet_manager.cc:1918] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 67b3aeadfa7a432abe8af4d292e04ab9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260809 15:46:10.222798 25011 log.cc:1196] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 67b3aeadfa7a432abe8af4d292e04ab9: Deleting WAL directory at /tmp/dist-test-taskj2AJ_N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786290285951082-21289-0/minicluster-data/ts-0-root/wals/5ad8a3b25d7b49e5a2142d5709edd0c7
I20260809 15:46:10.224741 23094 catalog_manager.cc:4841] TS 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459): tablet 5ad8a3b25d7b49e5a2142d5709edd0c7 (table test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]) successfully deleted
I20260809 15:46:10.228296 24876 raft_consensus.cc:2932] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:10.231433 23519 raft_consensus.cc:1261] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260809 15:46:10.234961 24472 raft_consensus.cc:2932] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:10.237190 23095 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 2a823d35f4f348d1b52ba9e6d80b00d2 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260809 15:46:10.234632 25000 consensus_queue.cc:1035] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260809 15:46:10.247027 23118 catalog_manager.cc:5495] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 61e69695258f48baa984c9d395e8ce3f reported cstate change: config changed from index 3 to 4, VOTER 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65) evicted. New cstate: current_term: 1 leader_uuid: "c0f38c1908414566aa9115bb2cbdd85d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:10.243983 23519 raft_consensus.cc:2932] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:10.266389 23260 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a823d35f4f348d1b52ba9e6d80b00d2 with delete_type TABLET_DATA_TOMBSTONED (TS 67b3aeadfa7a432abe8af4d292e04ab9 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:53552
I20260809 15:46:10.273186 23282 consensus_queue.cc:237] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:10.292428 23404 raft_consensus.cc:1261] T 8e4ffcf83c234ae7910ddef3ec0b4bda P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Refusing update from remote peer 67b3aeadfa7a432abe8af4d292e04ab9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260809 15:46:10.295213 24472 raft_consensus.cc:1261] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Refusing update from remote peer 67b3aeadfa7a432abe8af4d292e04ab9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260809 15:46:10.296026 25007 consensus_queue.cc:1035] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260809 15:46:10.300901 25007 consensus_queue.cc:1035] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260809 15:46:10.306502 25014 raft_consensus.cc:2932] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 65c36d164b0540f099d592f38307ee6c (127.20.202.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:10.315573 25011 tablet_replica.cc:330] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 67b3aeadfa7a432abe8af4d292e04ab9: stopping tablet replica
I20260809 15:46:10.317226 25011 raft_consensus.cc:2229] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 15:46:10.317991 25011 raft_consensus.cc:2258] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 15:46:10.309391 23404 raft_consensus.cc:2932] T 8e4ffcf83c234ae7910ddef3ec0b4bda P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 65c36d164b0540f099d592f38307ee6c (127.20.202.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:10.325647 25011 ts_tablet_manager.cc:1905] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 67b3aeadfa7a432abe8af4d292e04ab9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260809 15:46:10.330260 24472 raft_consensus.cc:2932] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 65c36d164b0540f099d592f38307ee6c (127.20.202.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:10.343006 23094 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 8e4ffcf83c234ae7910ddef3ec0b4bda with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260809 15:46:10.355810 23118 catalog_manager.cc:5495] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 61e69695258f48baa984c9d395e8ce3f reported cstate change: config changed from index 3 to 4, VOTER 65c36d164b0540f099d592f38307ee6c (127.20.202.67) evicted. New cstate: current_term: 1 leader_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:10.369585 23500 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e4ffcf83c234ae7910ddef3ec0b4bda with delete_type TABLET_DATA_TOMBSTONED (TS 65c36d164b0540f099d592f38307ee6c not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:56760
I20260809 15:46:10.367277 25011 ts_tablet_manager.cc:1918] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 67b3aeadfa7a432abe8af4d292e04ab9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260809 15:46:10.377060 25011 log.cc:1196] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 67b3aeadfa7a432abe8af4d292e04ab9: Deleting WAL directory at /tmp/dist-test-taskj2AJ_N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786290285951082-21289-0/minicluster-data/ts-0-root/wals/2a823d35f4f348d1b52ba9e6d80b00d2
I20260809 15:46:10.379081 23094 catalog_manager.cc:4841] TS 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459): tablet 2a823d35f4f348d1b52ba9e6d80b00d2 (table test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]) successfully deleted
I20260809 15:46:10.381865 25020 tablet_replica.cc:330] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 65c36d164b0540f099d592f38307ee6c: stopping tablet replica
I20260809 15:46:10.383219 25020 raft_consensus.cc:2229] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 15:46:10.384127 25020 raft_consensus.cc:2258] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 15:46:10.391673 25020 ts_tablet_manager.cc:1905] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 65c36d164b0540f099d592f38307ee6c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260809 15:46:10.403409 25020 ts_tablet_manager.cc:1918] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 65c36d164b0540f099d592f38307ee6c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260809 15:46:10.403777 25020 log.cc:1196] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 65c36d164b0540f099d592f38307ee6c: Deleting WAL directory at /tmp/dist-test-taskj2AJ_N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786290285951082-21289-0/minicluster-data/ts-2-root/wals/8e4ffcf83c234ae7910ddef3ec0b4bda
I20260809 15:46:10.405469 23095 catalog_manager.cc:4841] TS 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553): tablet 8e4ffcf83c234ae7910ddef3ec0b4bda (table test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]) successfully deleted
W20260809 15:46:11.540103 23472 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 98345.3us
W20260809 15:46:11.623363 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 256248us
W20260809 15:46:11.876631 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 68973.9us
W20260809 15:46:11.907678 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 60719.9us
W20260809 15:46:11.963878 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 52624.7us
I20260809 15:46:12.334807 25027 raft_consensus.cc:482] T 763e58d6ba644eea830a6c1664d39c9b P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
I20260809 15:46:12.336867 25027 raft_consensus.cc:504] T 763e58d6ba644eea830a6c1664d39c9b P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:12.347788 25027 leader_election.cc:288] T 763e58d6ba644eea830a6c1664d39c9b P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:12.379971 23522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "763e58d6ba644eea830a6c1664d39c9b" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
W20260809 15:46:12.446527 23353 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 58603.2us
W20260809 15:46:12.490661 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 106482us
I20260809 15:46:12.504659 23273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "763e58d6ba644eea830a6c1664d39c9b" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:12.571015 25001 raft_consensus.cc:482] T 0a51c0baaa6e432997c9d3552af4f254 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
I20260809 15:46:12.576357 25001 raft_consensus.cc:504] T 0a51c0baaa6e432997c9d3552af4f254 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:12.700762 25001 leader_election.cc:288] T 0a51c0baaa6e432997c9d3552af4f254 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
W20260809 15:46:12.795208 23470 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 147275us
I20260809 15:46:12.806082 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a51c0baaa6e432997c9d3552af4f254" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
W20260809 15:46:12.956969 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 436082us
I20260809 15:46:12.965677 25034 raft_consensus.cc:482] T 213543cdbe814c89a17a963469922c1b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:46:12.966756 25034 raft_consensus.cc:504] T 213543cdbe814c89a17a963469922c1b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:12.972939 24945 raft_consensus.cc:482] T e71a05ba1d4d42bd95c3f85f9313399e P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:46:12.974360 24945 raft_consensus.cc:504] T e71a05ba1d4d42bd95c3f85f9313399e P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:12.975953 25034 leader_election.cc:288] T 213543cdbe814c89a17a963469922c1b P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
W20260809 15:46:12.983752 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 55323.7us
I20260809 15:46:12.987882 24945 leader_election.cc:288] T e71a05ba1d4d42bd95c3f85f9313399e P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:12.996591 23351 leader_election.cc:302] T 763e58d6ba644eea830a6c1664d39c9b P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:13.009918 25050 raft_consensus.cc:2726] T 763e58d6ba644eea830a6c1664d39c9b P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260809 15:46:13.027442 23353 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 62905.2us
I20260809 15:46:13.039471 23404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "213543cdbe814c89a17a963469922c1b" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:13.052887 23404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e71a05ba1d4d42bd95c3f85f9313399e" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:13.063421 23274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a51c0baaa6e432997c9d3552af4f254" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:13.084779 23522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "213543cdbe814c89a17a963469922c1b" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:13.139523 23517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e71a05ba1d4d42bd95c3f85f9313399e" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:13.150171 23351 leader_election.cc:302] T 0a51c0baaa6e432997c9d3552af4f254 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 61e69695258f48baa984c9d395e8ce3f, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:13.157867 25046 raft_consensus.cc:2726] T 0a51c0baaa6e432997c9d3552af4f254 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 15:46:13.202692 23285 consensus_queue.cc:237] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } attrs { replace: true } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } }
I20260809 15:46:13.222805 23228 leader_election.cc:302] T 213543cdbe814c89a17a963469922c1b P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:13.226307 25060 raft_consensus.cc:2726] T 213543cdbe814c89a17a963469922c1b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 15:46:13.227605 23228 leader_election.cc:302] T e71a05ba1d4d42bd95c3f85f9313399e P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:13.229158 25060 raft_consensus.cc:2726] T e71a05ba1d4d42bd95c3f85f9313399e P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260809 15:46:13.274638 23223 consensus_peers.cc:487] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9 -> Peer 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689): Couldn't send request to peer 61e69695258f48baa984c9d395e8ce3f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 907bacfa79d04a55a68c8dbe60056462. This is attempt 1: this message will repeat every 5th retry.
I20260809 15:46:13.273553 23522 raft_consensus.cc:1261] T 907bacfa79d04a55a68c8dbe60056462 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Refusing update from remote peer 67b3aeadfa7a432abe8af4d292e04ab9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 15:46:13.290854 25034 consensus_queue.cc:1035] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260809 15:46:13.337886 23402 raft_consensus.cc:1261] T 907bacfa79d04a55a68c8dbe60056462 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Refusing update from remote peer 67b3aeadfa7a432abe8af4d292e04ab9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 15:46:13.352226 25034 consensus_queue.cc:1035] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260809 15:46:13.359046 25021 raft_consensus.cc:2932] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } attrs { replace: true } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } } }
I20260809 15:46:13.362340 23527 raft_consensus.cc:2932] T 907bacfa79d04a55a68c8dbe60056462 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } attrs { replace: true } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } } }
I20260809 15:46:13.386646 23402 raft_consensus.cc:2932] T 907bacfa79d04a55a68c8dbe60056462 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } attrs { replace: true } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } } }
I20260809 15:46:13.405443 23273 consensus_queue.cc:237] T 3d880de657e544468c82157769622a26 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } attrs { replace: true } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } }
I20260809 15:46:13.418524 23115 catalog_manager.cc:5495] T 907bacfa79d04a55a68c8dbe60056462 P 65c36d164b0540f099d592f38307ee6c reported cstate change: config changed from index -1 to 2, NON_VOTER 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) added. New cstate: current_term: 1 leader_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } attrs { replace: true } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } } }
I20260809 15:46:13.440336 23527 raft_consensus.cc:1261] T 3d880de657e544468c82157769622a26 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Refusing update from remote peer 67b3aeadfa7a432abe8af4d292e04ab9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260809 15:46:13.440441 23223 consensus_peers.cc:487] T 3d880de657e544468c82157769622a26 P 67b3aeadfa7a432abe8af4d292e04ab9 -> Peer 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689): Couldn't send request to peer 61e69695258f48baa984c9d395e8ce3f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3d880de657e544468c82157769622a26. This is attempt 1: this message will repeat every 5th retry.
I20260809 15:46:13.441789 25041 consensus_queue.cc:1035] T 3d880de657e544468c82157769622a26 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 15:46:13.450237 23402 raft_consensus.cc:1261] T 3d880de657e544468c82157769622a26 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Refusing update from remote peer 67b3aeadfa7a432abe8af4d292e04ab9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 15:46:13.451655 25021 consensus_queue.cc:1035] T 3d880de657e544468c82157769622a26 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260809 15:46:13.472419 25041 raft_consensus.cc:2932] T 3d880de657e544468c82157769622a26 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } attrs { replace: true } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } } }
I20260809 15:46:13.479518 23394 raft_consensus.cc:2932] T 3d880de657e544468c82157769622a26 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } attrs { replace: true } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } } }
I20260809 15:46:13.513319 23394 consensus_queue.cc:237] T e71a05ba1d4d42bd95c3f85f9313399e P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } attrs { replace: true } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } }
I20260809 15:46:13.523392 23522 raft_consensus.cc:1261] T e71a05ba1d4d42bd95c3f85f9313399e P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 15:46:13.524542 23273 raft_consensus.cc:1261] T e71a05ba1d4d42bd95c3f85f9313399e P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260809 15:46:13.526093 23345 consensus_peers.cc:487] T e71a05ba1d4d42bd95c3f85f9313399e P c0f38c1908414566aa9115bb2cbdd85d -> Peer 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689): Couldn't send request to peer 61e69695258f48baa984c9d395e8ce3f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e71a05ba1d4d42bd95c3f85f9313399e. This is attempt 1: this message will repeat every 5th retry.
I20260809 15:46:13.527596 25046 consensus_queue.cc:1035] T e71a05ba1d4d42bd95c3f85f9313399e P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260809 15:46:13.530201 25046 consensus_queue.cc:1035] T e71a05ba1d4d42bd95c3f85f9313399e P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 15:46:13.543042 23118 catalog_manager.cc:5495] T 3d880de657e544468c82157769622a26 P 67b3aeadfa7a432abe8af4d292e04ab9 reported cstate change: config changed from index -1 to 2, NON_VOTER 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) added. New cstate: current_term: 1 leader_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260809 15:46:13.555051 23527 raft_consensus.cc:2932] T 3d880de657e544468c82157769622a26 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } attrs { replace: true } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } } }
I20260809 15:46:13.587983 25025 raft_consensus.cc:2932] T e71a05ba1d4d42bd95c3f85f9313399e P c0f38c1908414566aa9115bb2cbdd85d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } attrs { replace: true } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } } }
I20260809 15:46:13.590406 23519 raft_consensus.cc:2932] T e71a05ba1d4d42bd95c3f85f9313399e P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } attrs { replace: true } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } } }
I20260809 15:46:13.607065 23273 raft_consensus.cc:2932] T e71a05ba1d4d42bd95c3f85f9313399e P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } attrs { replace: true } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } } }
I20260809 15:46:13.631603 23116 catalog_manager.cc:5495] T e71a05ba1d4d42bd95c3f85f9313399e P 65c36d164b0540f099d592f38307ee6c reported cstate change: config changed from index -1 to 2, NON_VOTER 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) added. New cstate: current_term: 1 leader_uuid: "c0f38c1908414566aa9115bb2cbdd85d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } attrs { replace: true } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } } }
I20260809 15:46:13.633569 23519 consensus_queue.cc:237] T c0741ab44d99439abc3aa3bdd27ee160 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } attrs { replace: true } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } }
I20260809 15:46:13.672127 23285 raft_consensus.cc:1261] T c0741ab44d99439abc3aa3bdd27ee160 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260809 15:46:13.672688 23468 consensus_peers.cc:487] T c0741ab44d99439abc3aa3bdd27ee160 P 65c36d164b0540f099d592f38307ee6c -> Peer 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689): Couldn't send request to peer 61e69695258f48baa984c9d395e8ce3f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c0741ab44d99439abc3aa3bdd27ee160. This is attempt 1: this message will repeat every 5th retry.
I20260809 15:46:13.673600 23402 raft_consensus.cc:1261] T c0741ab44d99439abc3aa3bdd27ee160 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 15:46:13.675567 25053 consensus_queue.cc:1035] T c0741ab44d99439abc3aa3bdd27ee160 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 15:46:13.680461 24927 consensus_queue.cc:1035] T c0741ab44d99439abc3aa3bdd27ee160 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 15:46:13.697146 24927 raft_consensus.cc:2932] T c0741ab44d99439abc3aa3bdd27ee160 P 65c36d164b0540f099d592f38307ee6c [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } attrs { replace: true } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } } }
I20260809 15:46:13.699842 23273 raft_consensus.cc:2932] T c0741ab44d99439abc3aa3bdd27ee160 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } attrs { replace: true } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } } }
I20260809 15:46:13.707743 23402 raft_consensus.cc:2932] T c0741ab44d99439abc3aa3bdd27ee160 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } attrs { replace: true } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } } }
I20260809 15:46:13.716759 23116 catalog_manager.cc:5495] T c0741ab44d99439abc3aa3bdd27ee160 P 65c36d164b0540f099d592f38307ee6c reported cstate change: config changed from index -1 to 2, NON_VOTER 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) added. New cstate: current_term: 1 leader_uuid: "65c36d164b0540f099d592f38307ee6c" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260809 15:46:13.748765 25092 ts_tablet_manager.cc:927] T 907bacfa79d04a55a68c8dbe60056462 P 61e69695258f48baa984c9d395e8ce3f: Initiating tablet copy from peer 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459)
I20260809 15:46:13.763379 25092 tablet_copy_client.cc:316] T 907bacfa79d04a55a68c8dbe60056462 P 61e69695258f48baa984c9d395e8ce3f: tablet copy: Beginning tablet copy session from remote peer at address 127.20.202.65:35459
I20260809 15:46:13.765175 23298 tablet_copy_service.cc:140] P 67b3aeadfa7a432abe8af4d292e04ab9: Received BeginTabletCopySession request for tablet 907bacfa79d04a55a68c8dbe60056462 from peer 61e69695258f48baa984c9d395e8ce3f ({username='slave'} at 127.0.0.1:55434)
I20260809 15:46:13.765846 23298 tablet_copy_service.cc:161] P 67b3aeadfa7a432abe8af4d292e04ab9: Beginning new tablet copy session on tablet 907bacfa79d04a55a68c8dbe60056462 from peer 61e69695258f48baa984c9d395e8ce3f at {username='slave'} at 127.0.0.1:55434: session id = 61e69695258f48baa984c9d395e8ce3f-907bacfa79d04a55a68c8dbe60056462
I20260809 15:46:13.777086 23298 tablet_copy_source_session.cc:205] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9: Tablet Copy: opened 0 blocks and 1 log segments
I20260809 15:46:13.780576 25092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 907bacfa79d04a55a68c8dbe60056462. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:46:13.805680 25092 tablet_copy_client.cc:791] T 907bacfa79d04a55a68c8dbe60056462 P 61e69695258f48baa984c9d395e8ce3f: tablet copy: Starting download of 0 data blocks...
I20260809 15:46:13.806286 25092 tablet_copy_client.cc:655] T 907bacfa79d04a55a68c8dbe60056462 P 61e69695258f48baa984c9d395e8ce3f: tablet copy: Starting download of 1 WAL segments...
I20260809 15:46:13.812458 25092 tablet_copy_client.cc:531] T 907bacfa79d04a55a68c8dbe60056462 P 61e69695258f48baa984c9d395e8ce3f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260809 15:46:13.824430 25092 tablet_bootstrap.cc:492] T 907bacfa79d04a55a68c8dbe60056462 P 61e69695258f48baa984c9d395e8ce3f: Bootstrap starting.
I20260809 15:46:13.851861 25092 tablet_bootstrap.cc:492] T 907bacfa79d04a55a68c8dbe60056462 P 61e69695258f48baa984c9d395e8ce3f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 15:46:13.852792 25092 tablet_bootstrap.cc:492] T 907bacfa79d04a55a68c8dbe60056462 P 61e69695258f48baa984c9d395e8ce3f: Bootstrap complete.
I20260809 15:46:13.853389 25092 ts_tablet_manager.cc:1397] T 907bacfa79d04a55a68c8dbe60056462 P 61e69695258f48baa984c9d395e8ce3f: Time spent bootstrapping tablet: real 0.029s user 0.016s sys 0.009s
I20260809 15:46:13.856279 25092 raft_consensus.cc:348] T 907bacfa79d04a55a68c8dbe60056462 P 61e69695258f48baa984c9d395e8ce3f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } attrs { replace: true } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } }
I20260809 15:46:13.857137 25092 raft_consensus.cc:729] T 907bacfa79d04a55a68c8dbe60056462 P 61e69695258f48baa984c9d395e8ce3f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 61e69695258f48baa984c9d395e8ce3f, State: Initialized, Role: LEARNER
I20260809 15:46:13.857812 25092 consensus_queue.cc:260] T 907bacfa79d04a55a68c8dbe60056462 P 61e69695258f48baa984c9d395e8ce3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } attrs { replace: true } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } }
I20260809 15:46:13.861392 25092 ts_tablet_manager.cc:1428] T 907bacfa79d04a55a68c8dbe60056462 P 61e69695258f48baa984c9d395e8ce3f: Time spent starting tablet: real 0.008s user 0.007s sys 0.001s
I20260809 15:46:13.867952 23298 tablet_copy_service.cc:342] P 67b3aeadfa7a432abe8af4d292e04ab9: Request end of tablet copy session 61e69695258f48baa984c9d395e8ce3f-907bacfa79d04a55a68c8dbe60056462 received from {username='slave'} at 127.0.0.1:55434
I20260809 15:46:13.868328 23298 tablet_copy_service.cc:433] P 67b3aeadfa7a432abe8af4d292e04ab9: ending tablet copy session 61e69695258f48baa984c9d395e8ce3f-907bacfa79d04a55a68c8dbe60056462 on tablet 907bacfa79d04a55a68c8dbe60056462 with peer 61e69695258f48baa984c9d395e8ce3f
I20260809 15:46:13.965577 25092 ts_tablet_manager.cc:927] T 3d880de657e544468c82157769622a26 P 61e69695258f48baa984c9d395e8ce3f: Initiating tablet copy from peer 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459)
I20260809 15:46:13.978269 25092 tablet_copy_client.cc:316] T 3d880de657e544468c82157769622a26 P 61e69695258f48baa984c9d395e8ce3f: tablet copy: Beginning tablet copy session from remote peer at address 127.20.202.65:35459
I20260809 15:46:13.979815 23298 tablet_copy_service.cc:140] P 67b3aeadfa7a432abe8af4d292e04ab9: Received BeginTabletCopySession request for tablet 3d880de657e544468c82157769622a26 from peer 61e69695258f48baa984c9d395e8ce3f ({username='slave'} at 127.0.0.1:55434)
I20260809 15:46:13.980298 23298 tablet_copy_service.cc:161] P 67b3aeadfa7a432abe8af4d292e04ab9: Beginning new tablet copy session on tablet 3d880de657e544468c82157769622a26 from peer 61e69695258f48baa984c9d395e8ce3f at {username='slave'} at 127.0.0.1:55434: session id = 61e69695258f48baa984c9d395e8ce3f-3d880de657e544468c82157769622a26
I20260809 15:46:13.988600 23298 tablet_copy_source_session.cc:205] T 3d880de657e544468c82157769622a26 P 67b3aeadfa7a432abe8af4d292e04ab9: Tablet Copy: opened 0 blocks and 1 log segments
I20260809 15:46:13.995875 25092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d880de657e544468c82157769622a26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:46:14.011416 25092 tablet_copy_client.cc:791] T 3d880de657e544468c82157769622a26 P 61e69695258f48baa984c9d395e8ce3f: tablet copy: Starting download of 0 data blocks...
I20260809 15:46:14.012072 25092 tablet_copy_client.cc:655] T 3d880de657e544468c82157769622a26 P 61e69695258f48baa984c9d395e8ce3f: tablet copy: Starting download of 1 WAL segments...
I20260809 15:46:14.016481 25092 tablet_copy_client.cc:531] T 3d880de657e544468c82157769622a26 P 61e69695258f48baa984c9d395e8ce3f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260809 15:46:14.026682 25092 tablet_bootstrap.cc:492] T 3d880de657e544468c82157769622a26 P 61e69695258f48baa984c9d395e8ce3f: Bootstrap starting.
I20260809 15:46:14.061187 25092 tablet_bootstrap.cc:492] T 3d880de657e544468c82157769622a26 P 61e69695258f48baa984c9d395e8ce3f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 15:46:14.062093 25092 tablet_bootstrap.cc:492] T 3d880de657e544468c82157769622a26 P 61e69695258f48baa984c9d395e8ce3f: Bootstrap complete.
I20260809 15:46:14.062779 25092 ts_tablet_manager.cc:1397] T 3d880de657e544468c82157769622a26 P 61e69695258f48baa984c9d395e8ce3f: Time spent bootstrapping tablet: real 0.036s user 0.017s sys 0.014s
I20260809 15:46:14.066762 25092 raft_consensus.cc:348] T 3d880de657e544468c82157769622a26 P 61e69695258f48baa984c9d395e8ce3f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } attrs { replace: true } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } }
I20260809 15:46:14.067600 25092 raft_consensus.cc:729] T 3d880de657e544468c82157769622a26 P 61e69695258f48baa984c9d395e8ce3f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 61e69695258f48baa984c9d395e8ce3f, State: Initialized, Role: LEARNER
I20260809 15:46:14.068316 25092 consensus_queue.cc:260] T 3d880de657e544468c82157769622a26 P 61e69695258f48baa984c9d395e8ce3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } attrs { replace: true } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } }
I20260809 15:46:14.073671 25092 ts_tablet_manager.cc:1428] T 3d880de657e544468c82157769622a26 P 61e69695258f48baa984c9d395e8ce3f: Time spent starting tablet: real 0.011s user 0.005s sys 0.004s
I20260809 15:46:14.075518 23298 tablet_copy_service.cc:342] P 67b3aeadfa7a432abe8af4d292e04ab9: Request end of tablet copy session 61e69695258f48baa984c9d395e8ce3f-3d880de657e544468c82157769622a26 received from {username='slave'} at 127.0.0.1:55434
I20260809 15:46:14.075943 23298 tablet_copy_service.cc:433] P 67b3aeadfa7a432abe8af4d292e04ab9: ending tablet copy session 61e69695258f48baa984c9d395e8ce3f-3d880de657e544468c82157769622a26 on tablet 3d880de657e544468c82157769622a26 with peer 61e69695258f48baa984c9d395e8ce3f
I20260809 15:46:14.095882 24471 raft_consensus.cc:1203] T 907bacfa79d04a55a68c8dbe60056462 P 61e69695258f48baa984c9d395e8ce3f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260809 15:46:14.097319 25092 ts_tablet_manager.cc:927] T c0741ab44d99439abc3aa3bdd27ee160 P 61e69695258f48baa984c9d395e8ce3f: Initiating tablet copy from peer 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:14.102226 25092 tablet_copy_client.cc:316] T c0741ab44d99439abc3aa3bdd27ee160 P 61e69695258f48baa984c9d395e8ce3f: tablet copy: Beginning tablet copy session from remote peer at address 127.20.202.67:40553
I20260809 15:46:14.104794 23546 tablet_copy_service.cc:140] P 65c36d164b0540f099d592f38307ee6c: Received BeginTabletCopySession request for tablet c0741ab44d99439abc3aa3bdd27ee160 from peer 61e69695258f48baa984c9d395e8ce3f ({username='slave'} at 127.0.0.1:44018)
I20260809 15:46:14.105288 23546 tablet_copy_service.cc:161] P 65c36d164b0540f099d592f38307ee6c: Beginning new tablet copy session on tablet c0741ab44d99439abc3aa3bdd27ee160 from peer 61e69695258f48baa984c9d395e8ce3f at {username='slave'} at 127.0.0.1:44018: session id = 61e69695258f48baa984c9d395e8ce3f-c0741ab44d99439abc3aa3bdd27ee160
I20260809 15:46:14.119248 23546 tablet_copy_source_session.cc:205] T c0741ab44d99439abc3aa3bdd27ee160 P 65c36d164b0540f099d592f38307ee6c: Tablet Copy: opened 0 blocks and 1 log segments
I20260809 15:46:14.122149 25092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0741ab44d99439abc3aa3bdd27ee160. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:46:14.133469 25106 ts_tablet_manager.cc:927] T e71a05ba1d4d42bd95c3f85f9313399e P 61e69695258f48baa984c9d395e8ce3f: Initiating tablet copy from peer c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:14.144778 25106 tablet_copy_client.cc:316] T e71a05ba1d4d42bd95c3f85f9313399e P 61e69695258f48baa984c9d395e8ce3f: tablet copy: Beginning tablet copy session from remote peer at address 127.20.202.66:36263
I20260809 15:46:14.146848 23424 tablet_copy_service.cc:140] P c0f38c1908414566aa9115bb2cbdd85d: Received BeginTabletCopySession request for tablet e71a05ba1d4d42bd95c3f85f9313399e from peer 61e69695258f48baa984c9d395e8ce3f ({username='slave'} at 127.0.0.1:52856)
I20260809 15:46:14.147539 23424 tablet_copy_service.cc:161] P c0f38c1908414566aa9115bb2cbdd85d: Beginning new tablet copy session on tablet e71a05ba1d4d42bd95c3f85f9313399e from peer 61e69695258f48baa984c9d395e8ce3f at {username='slave'} at 127.0.0.1:52856: session id = 61e69695258f48baa984c9d395e8ce3f-e71a05ba1d4d42bd95c3f85f9313399e
I20260809 15:46:14.152196 25092 tablet_copy_client.cc:791] T c0741ab44d99439abc3aa3bdd27ee160 P 61e69695258f48baa984c9d395e8ce3f: tablet copy: Starting download of 0 data blocks...
I20260809 15:46:14.153792 25092 tablet_copy_client.cc:655] T c0741ab44d99439abc3aa3bdd27ee160 P 61e69695258f48baa984c9d395e8ce3f: tablet copy: Starting download of 1 WAL segments...
I20260809 15:46:14.162614 23424 tablet_copy_source_session.cc:205] T e71a05ba1d4d42bd95c3f85f9313399e P c0f38c1908414566aa9115bb2cbdd85d: Tablet Copy: opened 0 blocks and 1 log segments
I20260809 15:46:14.165481 25092 tablet_copy_client.cc:531] T c0741ab44d99439abc3aa3bdd27ee160 P 61e69695258f48baa984c9d395e8ce3f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260809 15:46:14.170043 25106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e71a05ba1d4d42bd95c3f85f9313399e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:46:14.174561 25092 tablet_bootstrap.cc:492] T c0741ab44d99439abc3aa3bdd27ee160 P 61e69695258f48baa984c9d395e8ce3f: Bootstrap starting.
I20260809 15:46:14.196349 25106 tablet_copy_client.cc:791] T e71a05ba1d4d42bd95c3f85f9313399e P 61e69695258f48baa984c9d395e8ce3f: tablet copy: Starting download of 0 data blocks...
I20260809 15:46:14.197105 25106 tablet_copy_client.cc:655] T e71a05ba1d4d42bd95c3f85f9313399e P 61e69695258f48baa984c9d395e8ce3f: tablet copy: Starting download of 1 WAL segments...
I20260809 15:46:14.209977 25106 tablet_copy_client.cc:531] T e71a05ba1d4d42bd95c3f85f9313399e P 61e69695258f48baa984c9d395e8ce3f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260809 15:46:14.222553 25106 tablet_bootstrap.cc:492] T e71a05ba1d4d42bd95c3f85f9313399e P 61e69695258f48baa984c9d395e8ce3f: Bootstrap starting.
I20260809 15:46:14.228819 25092 tablet_bootstrap.cc:492] T c0741ab44d99439abc3aa3bdd27ee160 P 61e69695258f48baa984c9d395e8ce3f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 15:46:14.229569 25092 tablet_bootstrap.cc:492] T c0741ab44d99439abc3aa3bdd27ee160 P 61e69695258f48baa984c9d395e8ce3f: Bootstrap complete.
I20260809 15:46:14.230147 25092 ts_tablet_manager.cc:1397] T c0741ab44d99439abc3aa3bdd27ee160 P 61e69695258f48baa984c9d395e8ce3f: Time spent bootstrapping tablet: real 0.056s user 0.030s sys 0.008s
I20260809 15:46:14.233197 25092 raft_consensus.cc:348] T c0741ab44d99439abc3aa3bdd27ee160 P 61e69695258f48baa984c9d395e8ce3f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } attrs { replace: true } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } }
I20260809 15:46:14.233968 25092 raft_consensus.cc:729] T c0741ab44d99439abc3aa3bdd27ee160 P 61e69695258f48baa984c9d395e8ce3f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 61e69695258f48baa984c9d395e8ce3f, State: Initialized, Role: LEARNER
I20260809 15:46:14.234575 25092 consensus_queue.cc:260] T c0741ab44d99439abc3aa3bdd27ee160 P 61e69695258f48baa984c9d395e8ce3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } attrs { replace: true } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } }
I20260809 15:46:14.238948 25092 ts_tablet_manager.cc:1428] T c0741ab44d99439abc3aa3bdd27ee160 P 61e69695258f48baa984c9d395e8ce3f: Time spent starting tablet: real 0.009s user 0.004s sys 0.004s
I20260809 15:46:14.242770 23546 tablet_copy_service.cc:342] P 65c36d164b0540f099d592f38307ee6c: Request end of tablet copy session 61e69695258f48baa984c9d395e8ce3f-c0741ab44d99439abc3aa3bdd27ee160 received from {username='slave'} at 127.0.0.1:44018
I20260809 15:46:14.243176 23546 tablet_copy_service.cc:433] P 65c36d164b0540f099d592f38307ee6c: ending tablet copy session 61e69695258f48baa984c9d395e8ce3f-c0741ab44d99439abc3aa3bdd27ee160 on tablet c0741ab44d99439abc3aa3bdd27ee160 with peer 61e69695258f48baa984c9d395e8ce3f
I20260809 15:46:14.255014 25106 tablet_bootstrap.cc:492] T e71a05ba1d4d42bd95c3f85f9313399e P 61e69695258f48baa984c9d395e8ce3f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 15:46:14.255754 25106 tablet_bootstrap.cc:492] T e71a05ba1d4d42bd95c3f85f9313399e P 61e69695258f48baa984c9d395e8ce3f: Bootstrap complete.
I20260809 15:46:14.256309 25106 ts_tablet_manager.cc:1397] T e71a05ba1d4d42bd95c3f85f9313399e P 61e69695258f48baa984c9d395e8ce3f: Time spent bootstrapping tablet: real 0.034s user 0.012s sys 0.021s
I20260809 15:46:14.258579 25106 raft_consensus.cc:348] T e71a05ba1d4d42bd95c3f85f9313399e P 61e69695258f48baa984c9d395e8ce3f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } attrs { replace: true } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } }
I20260809 15:46:14.259438 25106 raft_consensus.cc:729] T e71a05ba1d4d42bd95c3f85f9313399e P 61e69695258f48baa984c9d395e8ce3f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 61e69695258f48baa984c9d395e8ce3f, State: Initialized, Role: LEARNER
I20260809 15:46:14.260042 25106 consensus_queue.cc:260] T e71a05ba1d4d42bd95c3f85f9313399e P 61e69695258f48baa984c9d395e8ce3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } attrs { replace: true } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } }
I20260809 15:46:14.264021 25106 ts_tablet_manager.cc:1428] T e71a05ba1d4d42bd95c3f85f9313399e P 61e69695258f48baa984c9d395e8ce3f: Time spent starting tablet: real 0.007s user 0.006s sys 0.001s
I20260809 15:46:14.265645 23424 tablet_copy_service.cc:342] P c0f38c1908414566aa9115bb2cbdd85d: Request end of tablet copy session 61e69695258f48baa984c9d395e8ce3f-e71a05ba1d4d42bd95c3f85f9313399e received from {username='slave'} at 127.0.0.1:52856
I20260809 15:46:14.266008 23424 tablet_copy_service.cc:433] P c0f38c1908414566aa9115bb2cbdd85d: ending tablet copy session 61e69695258f48baa984c9d395e8ce3f-e71a05ba1d4d42bd95c3f85f9313399e on tablet e71a05ba1d4d42bd95c3f85f9313399e with peer 61e69695258f48baa984c9d395e8ce3f
I20260809 15:46:14.458992 24472 raft_consensus.cc:1203] T 3d880de657e544468c82157769622a26 P 61e69695258f48baa984c9d395e8ce3f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260809 15:46:14.481125 25034 raft_consensus.cc:1050] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9: attempting to promote NON_VOTER 61e69695258f48baa984c9d395e8ce3f to VOTER
I20260809 15:46:14.483213 25034 consensus_queue.cc:237] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } attrs { replace: true } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:14.488706 24472 raft_consensus.cc:1261] T 907bacfa79d04a55a68c8dbe60056462 P 61e69695258f48baa984c9d395e8ce3f [term 1 LEARNER]: Refusing update from remote peer 67b3aeadfa7a432abe8af4d292e04ab9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 15:46:14.488914 23519 raft_consensus.cc:1261] T 907bacfa79d04a55a68c8dbe60056462 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Refusing update from remote peer 67b3aeadfa7a432abe8af4d292e04ab9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 15:46:14.490824 25041 consensus_queue.cc:1035] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 15:46:14.492859 25119 consensus_queue.cc:1035] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 15:46:14.503404 23394 raft_consensus.cc:1261] T 907bacfa79d04a55a68c8dbe60056462 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Refusing update from remote peer 67b3aeadfa7a432abe8af4d292e04ab9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 15:46:14.505030 25119 consensus_queue.cc:1035] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 15:46:14.506804 25060 raft_consensus.cc:2932] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } attrs { replace: true } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:14.516386 23404 raft_consensus.cc:2932] T 907bacfa79d04a55a68c8dbe60056462 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } attrs { replace: true } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:14.509171 24472 raft_consensus.cc:2932] T 907bacfa79d04a55a68c8dbe60056462 P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } attrs { replace: true } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:14.522393 23519 raft_consensus.cc:2932] T 907bacfa79d04a55a68c8dbe60056462 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } attrs { replace: true } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:14.533274 23116 catalog_manager.cc:5495] T 907bacfa79d04a55a68c8dbe60056462 P c0f38c1908414566aa9115bb2cbdd85d reported cstate change: config changed from index 2 to 3, 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } attrs { replace: true } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:14.560374 24472 raft_consensus.cc:1203] T e71a05ba1d4d42bd95c3f85f9313399e P 61e69695258f48baa984c9d395e8ce3f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260809 15:46:14.563366 23273 consensus_queue.cc:237] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:14.571090 23394 raft_consensus.cc:1261] T 907bacfa79d04a55a68c8dbe60056462 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Refusing update from remote peer 67b3aeadfa7a432abe8af4d292e04ab9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260809 15:46:14.572286 25115 consensus_queue.cc:1035] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260809 15:46:14.576882 24472 raft_consensus.cc:1261] T 907bacfa79d04a55a68c8dbe60056462 P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Refusing update from remote peer 67b3aeadfa7a432abe8af4d292e04ab9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260809 15:46:14.578236 25115 consensus_queue.cc:1035] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260809 15:46:14.585271 25119 raft_consensus.cc:2932] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 65c36d164b0540f099d592f38307ee6c (127.20.202.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:14.587567 24472 raft_consensus.cc:2932] T 907bacfa79d04a55a68c8dbe60056462 P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 65c36d164b0540f099d592f38307ee6c (127.20.202.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:14.588241 23394 raft_consensus.cc:2932] T 907bacfa79d04a55a68c8dbe60056462 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 65c36d164b0540f099d592f38307ee6c (127.20.202.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:14.605638 23117 catalog_manager.cc:5495] T 907bacfa79d04a55a68c8dbe60056462 P 61e69695258f48baa984c9d395e8ce3f reported cstate change: config changed from index 3 to 4, VOTER 65c36d164b0540f099d592f38307ee6c (127.20.202.67) evicted. New cstate: current_term: 1 leader_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:14.615013 23500 tablet_service.cc:1514] Processing DeleteTablet for tablet 907bacfa79d04a55a68c8dbe60056462 with delete_type TABLET_DATA_TOMBSTONED (TS 65c36d164b0540f099d592f38307ee6c not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:56760
I20260809 15:46:14.616649 25127 tablet_replica.cc:330] T 907bacfa79d04a55a68c8dbe60056462 P 65c36d164b0540f099d592f38307ee6c: stopping tablet replica
I20260809 15:46:14.617574 25127 raft_consensus.cc:2229] T 907bacfa79d04a55a68c8dbe60056462 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 15:46:14.618312 25127 raft_consensus.cc:2258] T 907bacfa79d04a55a68c8dbe60056462 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 15:46:14.621627 25127 ts_tablet_manager.cc:1905] T 907bacfa79d04a55a68c8dbe60056462 P 65c36d164b0540f099d592f38307ee6c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260809 15:46:14.625114 23094 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 907bacfa79d04a55a68c8dbe60056462 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260809 15:46:14.637674 25127 ts_tablet_manager.cc:1918] T 907bacfa79d04a55a68c8dbe60056462 P 65c36d164b0540f099d592f38307ee6c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260809 15:46:14.638063 25127 log.cc:1196] T 907bacfa79d04a55a68c8dbe60056462 P 65c36d164b0540f099d592f38307ee6c: Deleting WAL directory at /tmp/dist-test-taskj2AJ_N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786290285951082-21289-0/minicluster-data/ts-2-root/wals/907bacfa79d04a55a68c8dbe60056462
I20260809 15:46:14.640429 23095 catalog_manager.cc:4841] TS 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553): tablet 907bacfa79d04a55a68c8dbe60056462 (table test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]) successfully deleted
I20260809 15:46:14.655412 24472 raft_consensus.cc:1203] T c0741ab44d99439abc3aa3bdd27ee160 P 61e69695258f48baa984c9d395e8ce3f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260809 15:46:14.941968 25049 raft_consensus.cc:1050] T e71a05ba1d4d42bd95c3f85f9313399e P c0f38c1908414566aa9115bb2cbdd85d: attempting to promote NON_VOTER 61e69695258f48baa984c9d395e8ce3f to VOTER
I20260809 15:46:14.944308 25049 consensus_queue.cc:237] T e71a05ba1d4d42bd95c3f85f9313399e P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } attrs { replace: true } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:14.950361 24472 raft_consensus.cc:1261] T e71a05ba1d4d42bd95c3f85f9313399e P 61e69695258f48baa984c9d395e8ce3f [term 1 LEARNER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 15:46:14.951630 25049 consensus_queue.cc:1035] T e71a05ba1d4d42bd95c3f85f9313399e P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 15:46:14.952455 23273 raft_consensus.cc:1261] T e71a05ba1d4d42bd95c3f85f9313399e P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 15:46:14.952450 23519 raft_consensus.cc:1261] T e71a05ba1d4d42bd95c3f85f9313399e P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 15:46:14.953884 25028 consensus_queue.cc:1035] T e71a05ba1d4d42bd95c3f85f9313399e P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 15:46:14.954674 25129 consensus_queue.cc:1035] T e71a05ba1d4d42bd95c3f85f9313399e P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 15:46:14.966260 25028 raft_consensus.cc:2932] T e71a05ba1d4d42bd95c3f85f9313399e P c0f38c1908414566aa9115bb2cbdd85d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } attrs { replace: true } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:14.971661 23282 raft_consensus.cc:2932] T e71a05ba1d4d42bd95c3f85f9313399e P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } attrs { replace: true } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:14.972551 24472 raft_consensus.cc:2932] T e71a05ba1d4d42bd95c3f85f9313399e P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } attrs { replace: true } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:14.988585 23117 catalog_manager.cc:5495] T e71a05ba1d4d42bd95c3f85f9313399e P 61e69695258f48baa984c9d395e8ce3f reported cstate change: config changed from index 2 to 3, 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c0f38c1908414566aa9115bb2cbdd85d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } attrs { replace: true } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:14.990814 23519 raft_consensus.cc:2932] T e71a05ba1d4d42bd95c3f85f9313399e P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } attrs { replace: true } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:15.010571 23394 consensus_queue.cc:237] T e71a05ba1d4d42bd95c3f85f9313399e P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:15.016144 24472 raft_consensus.cc:1261] T e71a05ba1d4d42bd95c3f85f9313399e P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260809 15:46:15.016610 23282 raft_consensus.cc:1261] T e71a05ba1d4d42bd95c3f85f9313399e P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260809 15:46:15.018085 24876 consensus_queue.cc:1035] T e71a05ba1d4d42bd95c3f85f9313399e P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260809 15:46:15.018901 25129 consensus_queue.cc:1035] T e71a05ba1d4d42bd95c3f85f9313399e P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260809 15:46:15.026966 23285 raft_consensus.cc:2932] T e71a05ba1d4d42bd95c3f85f9313399e P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 65c36d164b0540f099d592f38307ee6c (127.20.202.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:15.026890 25049 raft_consensus.cc:2932] T e71a05ba1d4d42bd95c3f85f9313399e P c0f38c1908414566aa9115bb2cbdd85d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 65c36d164b0540f099d592f38307ee6c (127.20.202.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:15.031761 24472 raft_consensus.cc:2932] T e71a05ba1d4d42bd95c3f85f9313399e P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 65c36d164b0540f099d592f38307ee6c (127.20.202.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:15.046124 23115 catalog_manager.cc:5495] T e71a05ba1d4d42bd95c3f85f9313399e P 61e69695258f48baa984c9d395e8ce3f reported cstate change: config changed from index 3 to 4, VOTER 65c36d164b0540f099d592f38307ee6c (127.20.202.67) evicted. New cstate: current_term: 1 leader_uuid: "c0f38c1908414566aa9115bb2cbdd85d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:15.047020 23095 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet e71a05ba1d4d42bd95c3f85f9313399e with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260809 15:46:15.055583 23500 tablet_service.cc:1514] Processing DeleteTablet for tablet e71a05ba1d4d42bd95c3f85f9313399e with delete_type TABLET_DATA_TOMBSTONED (TS 65c36d164b0540f099d592f38307ee6c not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:56760
I20260809 15:46:15.056438 25127 tablet_replica.cc:330] T e71a05ba1d4d42bd95c3f85f9313399e P 65c36d164b0540f099d592f38307ee6c: stopping tablet replica
I20260809 15:46:15.056528 25034 raft_consensus.cc:1050] T 3d880de657e544468c82157769622a26 P 67b3aeadfa7a432abe8af4d292e04ab9: attempting to promote NON_VOTER 61e69695258f48baa984c9d395e8ce3f to VOTER
I20260809 15:46:15.057312 25127 raft_consensus.cc:2229] T e71a05ba1d4d42bd95c3f85f9313399e P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 15:46:15.057904 25127 raft_consensus.cc:2258] T e71a05ba1d4d42bd95c3f85f9313399e P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 15:46:15.059594 25034 consensus_queue.cc:237] T 3d880de657e544468c82157769622a26 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } attrs { replace: true } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:15.060941 25127 ts_tablet_manager.cc:1905] T e71a05ba1d4d42bd95c3f85f9313399e P 65c36d164b0540f099d592f38307ee6c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260809 15:46:15.071431 23404 raft_consensus.cc:1261] T 3d880de657e544468c82157769622a26 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Refusing update from remote peer 67b3aeadfa7a432abe8af4d292e04ab9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 15:46:15.071506 23519 raft_consensus.cc:1261] T 3d880de657e544468c82157769622a26 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Refusing update from remote peer 67b3aeadfa7a432abe8af4d292e04ab9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 15:46:15.072011 24472 raft_consensus.cc:1261] T 3d880de657e544468c82157769622a26 P 61e69695258f48baa984c9d395e8ce3f [term 1 LEARNER]: Refusing update from remote peer 67b3aeadfa7a432abe8af4d292e04ab9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 15:46:15.074440 25119 consensus_queue.cc:1035] T 3d880de657e544468c82157769622a26 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 15:46:15.075412 25060 consensus_queue.cc:1035] T 3d880de657e544468c82157769622a26 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 15:46:15.076202 25034 consensus_queue.cc:1035] T 3d880de657e544468c82157769622a26 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260809 15:46:15.078203 25127 ts_tablet_manager.cc:1918] T e71a05ba1d4d42bd95c3f85f9313399e P 65c36d164b0540f099d592f38307ee6c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260809 15:46:15.078585 25127 log.cc:1196] T e71a05ba1d4d42bd95c3f85f9313399e P 65c36d164b0540f099d592f38307ee6c: Deleting WAL directory at /tmp/dist-test-taskj2AJ_N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786290285951082-21289-0/minicluster-data/ts-2-root/wals/e71a05ba1d4d42bd95c3f85f9313399e
I20260809 15:46:15.079890 23095 catalog_manager.cc:4841] TS 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553): tablet e71a05ba1d4d42bd95c3f85f9313399e (table test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]) successfully deleted
I20260809 15:46:15.101621 25060 raft_consensus.cc:2932] T 3d880de657e544468c82157769622a26 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } attrs { replace: true } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:15.104751 24472 raft_consensus.cc:2932] T 3d880de657e544468c82157769622a26 P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } attrs { replace: true } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:15.106859 23517 raft_consensus.cc:2932] T 3d880de657e544468c82157769622a26 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } attrs { replace: true } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:15.121320 23394 raft_consensus.cc:2932] T 3d880de657e544468c82157769622a26 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } attrs { replace: true } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:15.121996 23116 catalog_manager.cc:5495] T 3d880de657e544468c82157769622a26 P 61e69695258f48baa984c9d395e8ce3f reported cstate change: config changed from index 2 to 3, 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } attrs { replace: true } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:15.145269 23273 consensus_queue.cc:237] T 3d880de657e544468c82157769622a26 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:15.158421 24472 raft_consensus.cc:1261] T 3d880de657e544468c82157769622a26 P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Refusing update from remote peer 67b3aeadfa7a432abe8af4d292e04ab9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260809 15:46:15.159699 25060 consensus_queue.cc:1035] T 3d880de657e544468c82157769622a26 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260809 15:46:15.166101 23527 raft_consensus.cc:1261] T 3d880de657e544468c82157769622a26 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Refusing update from remote peer 67b3aeadfa7a432abe8af4d292e04ab9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260809 15:46:15.167707 25119 raft_consensus.cc:2932] T 3d880de657e544468c82157769622a26 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:15.169050 24472 raft_consensus.cc:2932] T 3d880de657e544468c82157769622a26 P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:15.167717 25060 consensus_queue.cc:1035] T 3d880de657e544468c82157769622a26 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.003s
I20260809 15:46:15.175027 23517 raft_consensus.cc:2932] T 3d880de657e544468c82157769622a26 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:15.180231 23094 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 3d880de657e544468c82157769622a26 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260809 15:46:15.182991 23116 catalog_manager.cc:5495] T 3d880de657e544468c82157769622a26 P 61e69695258f48baa984c9d395e8ce3f reported cstate change: config changed from index 3 to 4, VOTER c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66) evicted. New cstate: current_term: 1 leader_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:15.205802 23389 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d880de657e544468c82157769622a26 with delete_type TABLET_DATA_TOMBSTONED (TS c0f38c1908414566aa9115bb2cbdd85d not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:55312
I20260809 15:46:15.216123 25146 tablet_replica.cc:330] T 3d880de657e544468c82157769622a26 P c0f38c1908414566aa9115bb2cbdd85d: stopping tablet replica
I20260809 15:46:15.217224 25146 raft_consensus.cc:2229] T 3d880de657e544468c82157769622a26 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 15:46:15.217949 25146 raft_consensus.cc:2258] T 3d880de657e544468c82157769622a26 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 15:46:15.221242 25146 ts_tablet_manager.cc:1905] T 3d880de657e544468c82157769622a26 P c0f38c1908414566aa9115bb2cbdd85d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260809 15:46:15.235464 25146 ts_tablet_manager.cc:1918] T 3d880de657e544468c82157769622a26 P c0f38c1908414566aa9115bb2cbdd85d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260809 15:46:15.235800 25146 log.cc:1196] T 3d880de657e544468c82157769622a26 P c0f38c1908414566aa9115bb2cbdd85d: Deleting WAL directory at /tmp/dist-test-taskj2AJ_N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786290285951082-21289-0/minicluster-data/ts-1-root/wals/3d880de657e544468c82157769622a26
I20260809 15:46:15.239801 23095 catalog_manager.cc:4841] TS c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263): tablet 3d880de657e544468c82157769622a26 (table test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]) successfully deleted
I20260809 15:46:15.274327 25044 raft_consensus.cc:1050] T c0741ab44d99439abc3aa3bdd27ee160 P 65c36d164b0540f099d592f38307ee6c: attempting to promote NON_VOTER 61e69695258f48baa984c9d395e8ce3f to VOTER
I20260809 15:46:15.276556 25044 consensus_queue.cc:237] T c0741ab44d99439abc3aa3bdd27ee160 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } attrs { replace: true } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:15.286355 24472 raft_consensus.cc:1261] T c0741ab44d99439abc3aa3bdd27ee160 P 61e69695258f48baa984c9d395e8ce3f [term 1 LEARNER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 15:46:15.286482 23394 raft_consensus.cc:1261] T c0741ab44d99439abc3aa3bdd27ee160 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 15:46:15.288025 25128 consensus_queue.cc:1035] T c0741ab44d99439abc3aa3bdd27ee160 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260809 15:46:15.291303 25128 consensus_queue.cc:1035] T c0741ab44d99439abc3aa3bdd27ee160 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 15:46:15.295468 23273 raft_consensus.cc:1261] T c0741ab44d99439abc3aa3bdd27ee160 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 15:46:15.297523 25154 consensus_queue.cc:1035] T c0741ab44d99439abc3aa3bdd27ee160 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 15:46:15.305598 25151 raft_consensus.cc:2932] T c0741ab44d99439abc3aa3bdd27ee160 P 65c36d164b0540f099d592f38307ee6c [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } attrs { replace: true } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:15.306834 23273 raft_consensus.cc:2932] T c0741ab44d99439abc3aa3bdd27ee160 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } attrs { replace: true } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:15.308249 24472 raft_consensus.cc:2932] T c0741ab44d99439abc3aa3bdd27ee160 P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } attrs { replace: true } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:15.317863 23394 raft_consensus.cc:2932] T c0741ab44d99439abc3aa3bdd27ee160 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } attrs { replace: true } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:15.322570 23117 catalog_manager.cc:5495] T c0741ab44d99439abc3aa3bdd27ee160 P 61e69695258f48baa984c9d395e8ce3f reported cstate change: config changed from index 2 to 3, 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "65c36d164b0540f099d592f38307ee6c" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } attrs { replace: true } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:15.345899 23527 consensus_queue.cc:237] T c0741ab44d99439abc3aa3bdd27ee160 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:15.351202 23273 raft_consensus.cc:1261] T c0741ab44d99439abc3aa3bdd27ee160 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260809 15:46:15.353713 25151 consensus_queue.cc:1035] T c0741ab44d99439abc3aa3bdd27ee160 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260809 15:46:15.355728 24472 raft_consensus.cc:1261] T c0741ab44d99439abc3aa3bdd27ee160 P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260809 15:46:15.358846 25159 consensus_queue.cc:1035] T c0741ab44d99439abc3aa3bdd27ee160 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260809 15:46:15.360638 25128 raft_consensus.cc:2932] T c0741ab44d99439abc3aa3bdd27ee160 P 65c36d164b0540f099d592f38307ee6c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:15.361750 23273 raft_consensus.cc:2932] T c0741ab44d99439abc3aa3bdd27ee160 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:15.375602 23095 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet c0741ab44d99439abc3aa3bdd27ee160 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260809 15:46:15.376183 23118 catalog_manager.cc:5495] T c0741ab44d99439abc3aa3bdd27ee160 P 67b3aeadfa7a432abe8af4d292e04ab9 reported cstate change: config changed from index 3 to 4, VOTER c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66) evicted. New cstate: current_term: 1 leader_uuid: "65c36d164b0540f099d592f38307ee6c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:15.377858 24472 raft_consensus.cc:2932] T c0741ab44d99439abc3aa3bdd27ee160 P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:15.385043 23389 tablet_service.cc:1514] Processing DeleteTablet for tablet c0741ab44d99439abc3aa3bdd27ee160 with delete_type TABLET_DATA_TOMBSTONED (TS c0f38c1908414566aa9115bb2cbdd85d not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:55312
I20260809 15:46:15.386173 25146 tablet_replica.cc:330] T c0741ab44d99439abc3aa3bdd27ee160 P c0f38c1908414566aa9115bb2cbdd85d: stopping tablet replica
I20260809 15:46:15.387288 25146 raft_consensus.cc:2229] T c0741ab44d99439abc3aa3bdd27ee160 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 15:46:15.387830 25146 raft_consensus.cc:2258] T c0741ab44d99439abc3aa3bdd27ee160 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 15:46:15.390290 25146 ts_tablet_manager.cc:1905] T c0741ab44d99439abc3aa3bdd27ee160 P c0f38c1908414566aa9115bb2cbdd85d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260809 15:46:15.404234 25146 ts_tablet_manager.cc:1918] T c0741ab44d99439abc3aa3bdd27ee160 P c0f38c1908414566aa9115bb2cbdd85d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260809 15:46:15.404590 25146 log.cc:1196] T c0741ab44d99439abc3aa3bdd27ee160 P c0f38c1908414566aa9115bb2cbdd85d: Deleting WAL directory at /tmp/dist-test-taskj2AJ_N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786290285951082-21289-0/minicluster-data/ts-1-root/wals/c0741ab44d99439abc3aa3bdd27ee160
I20260809 15:46:15.405851 23095 catalog_manager.cc:4841] TS c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263): tablet c0741ab44d99439abc3aa3bdd27ee160 (table test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]) successfully deleted
I20260809 15:46:16.400149 21289 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 15:46:16.454710 23517 consensus_queue.cc:237] T 472816994d814cb198b69e5248697440 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } }
W20260809 15:46:16.460762 23468 consensus_peers.cc:487] T 472816994d814cb198b69e5248697440 P 65c36d164b0540f099d592f38307ee6c -> Peer 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689): Couldn't send request to peer 61e69695258f48baa984c9d395e8ce3f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 472816994d814cb198b69e5248697440. This is attempt 1: this message will repeat every 5th retry.
I20260809 15:46:16.461057 23404 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "538a228ddfb54bc6a36528d5ff07597b"
dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
mode: GRACEFUL
new_leader_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
from {username='slave'} at 127.0.0.1:51280
I20260809 15:46:16.461740 23404 raft_consensus.cc:595] T 538a228ddfb54bc6a36528d5ff07597b P c0f38c1908414566aa9115bb2cbdd85d [term 1 LEADER]: Received request to transfer leadership to TS 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:16.461892 23394 raft_consensus.cc:1261] T 472816994d814cb198b69e5248697440 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 15:46:16.463660 23285 raft_consensus.cc:1261] T 472816994d814cb198b69e5248697440 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 15:46:16.464846 25151 consensus_queue.cc:1035] T 472816994d814cb198b69e5248697440 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 15:46:16.466804 25044 consensus_queue.cc:1035] T 472816994d814cb198b69e5248697440 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 15:46:16.473861 25044 raft_consensus.cc:2932] T 472816994d814cb198b69e5248697440 P 65c36d164b0540f099d592f38307ee6c [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } } }
I20260809 15:46:16.478381 23394 raft_consensus.cc:2932] T 472816994d814cb198b69e5248697440 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } } }
I20260809 15:46:16.479703 23285 raft_consensus.cc:2932] T 472816994d814cb198b69e5248697440 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } } }
I20260809 15:46:16.488722 23273 consensus_queue.cc:237] T 563daf42d7784db195ea99c3cb21b370 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } attrs { replace: true } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } }
I20260809 15:46:16.497511 23116 catalog_manager.cc:5495] T 472816994d814cb198b69e5248697440 P 65c36d164b0540f099d592f38307ee6c reported cstate change: config changed from index -1 to 2, NON_VOTER 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) added. New cstate: current_term: 1 leader_uuid: "65c36d164b0540f099d592f38307ee6c" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260809 15:46:16.514506 23517 raft_consensus.cc:1261] T 563daf42d7784db195ea99c3cb21b370 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Refusing update from remote peer 67b3aeadfa7a432abe8af4d292e04ab9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260809 15:46:16.514760 23223 consensus_peers.cc:487] T 563daf42d7784db195ea99c3cb21b370 P 67b3aeadfa7a432abe8af4d292e04ab9 -> Peer 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689): Couldn't send request to peer 61e69695258f48baa984c9d395e8ce3f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 563daf42d7784db195ea99c3cb21b370. This is attempt 1: this message will repeat every 5th retry.
I20260809 15:46:16.515874 25119 consensus_queue.cc:1035] T 563daf42d7784db195ea99c3cb21b370 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 15:46:16.521169 23394 raft_consensus.cc:1261] T 563daf42d7784db195ea99c3cb21b370 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Refusing update from remote peer 67b3aeadfa7a432abe8af4d292e04ab9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 15:46:16.522514 25119 consensus_queue.cc:1035] T 563daf42d7784db195ea99c3cb21b370 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 15:46:16.524370 23273 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5712f6b6fffc4cc5a8ddfbe63e4097df"
dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
mode: GRACEFUL
new_leader_uuid: "61e69695258f48baa984c9d395e8ce3f"
from {username='slave'} at 127.0.0.1:33468
I20260809 15:46:16.524942 23273 raft_consensus.cc:595] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 LEADER]: Received request to transfer leadership to TS 61e69695258f48baa984c9d395e8ce3f
I20260809 15:46:16.527433 23273 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "763e58d6ba644eea830a6c1664d39c9b"
dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
mode: GRACEFUL
new_leader_uuid: "65c36d164b0540f099d592f38307ee6c"
from {username='slave'} at 127.0.0.1:33468
I20260809 15:46:16.527943 23273 raft_consensus.cc:595] T 763e58d6ba644eea830a6c1664d39c9b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 LEADER]: Received request to transfer leadership to TS 65c36d164b0540f099d592f38307ee6c
I20260809 15:46:16.528527 25060 raft_consensus.cc:2932] T 563daf42d7784db195ea99c3cb21b370 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } attrs { replace: true } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } } }
I20260809 15:46:16.531344 23517 raft_consensus.cc:2932] T 563daf42d7784db195ea99c3cb21b370 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } attrs { replace: true } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } } }
I20260809 15:46:16.532840 23402 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "998c6cee72cf40228c6dba064c0db2da"
dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
mode: GRACEFUL
new_leader_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
from {username='slave'} at 127.0.0.1:51280
I20260809 15:46:16.533438 23402 raft_consensus.cc:595] T 998c6cee72cf40228c6dba064c0db2da P c0f38c1908414566aa9115bb2cbdd85d [term 1 LEADER]: Received request to transfer leadership to TS 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:16.546900 23273 consensus_queue.cc:237] T 16ca412a0dae4feebe85dd01bf1a9173 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } attrs { replace: true } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } }
W20260809 15:46:16.555199 23223 consensus_peers.cc:487] T 16ca412a0dae4feebe85dd01bf1a9173 P 67b3aeadfa7a432abe8af4d292e04ab9 -> Peer 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689): Couldn't send request to peer 61e69695258f48baa984c9d395e8ce3f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 16ca412a0dae4feebe85dd01bf1a9173. This is attempt 1: this message will repeat every 5th retry.
I20260809 15:46:16.560931 23394 raft_consensus.cc:2932] T 563daf42d7784db195ea99c3cb21b370 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } attrs { replace: true } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } } }
I20260809 15:46:16.563421 23517 raft_consensus.cc:1261] T 16ca412a0dae4feebe85dd01bf1a9173 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Refusing update from remote peer 67b3aeadfa7a432abe8af4d292e04ab9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 15:46:16.577085 23402 raft_consensus.cc:1261] T 16ca412a0dae4feebe85dd01bf1a9173 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Refusing update from remote peer 67b3aeadfa7a432abe8af4d292e04ab9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 15:46:16.575762 23118 catalog_manager.cc:5495] T 563daf42d7784db195ea99c3cb21b370 P 67b3aeadfa7a432abe8af4d292e04ab9 reported cstate change: config changed from index -1 to 2, NON_VOTER 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) added. New cstate: current_term: 1 leader_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260809 15:46:16.582368 25060 consensus_queue.cc:1035] T 16ca412a0dae4feebe85dd01bf1a9173 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 15:46:16.584370 23517 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e14d118b2eaa42aa8a7cd4025aba7128"
dest_uuid: "65c36d164b0540f099d592f38307ee6c"
mode: GRACEFUL
new_leader_uuid: "61e69695258f48baa984c9d395e8ce3f"
from {username='slave'} at 127.0.0.1:48888
I20260809 15:46:16.584900 23517 raft_consensus.cc:595] T e14d118b2eaa42aa8a7cd4025aba7128 P 65c36d164b0540f099d592f38307ee6c [term 1 LEADER]: Received request to transfer leadership to TS 61e69695258f48baa984c9d395e8ce3f
I20260809 15:46:16.586019 21289 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260809 15:46:16.586305 25185 consensus_queue.cc:1035] T 16ca412a0dae4feebe85dd01bf1a9173 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 15:46:16.587627 21289 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 15:46:16.588316 21289 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 15:46:16.608196 25129 raft_consensus.cc:979] T 538a228ddfb54bc6a36528d5ff07597b P c0f38c1908414566aa9115bb2cbdd85d: : Instructing follower 67b3aeadfa7a432abe8af4d292e04ab9 to start an election
I20260809 15:46:16.608690 25160 raft_consensus.cc:1067] T 538a228ddfb54bc6a36528d5ff07597b P c0f38c1908414566aa9115bb2cbdd85d [term 1 LEADER]: Signalling peer 67b3aeadfa7a432abe8af4d292e04ab9 to start an election
I20260809 15:46:16.610316 23273 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "538a228ddfb54bc6a36528d5ff07597b"
dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
from {username='slave'} at 127.0.0.1:53562
I20260809 15:46:16.610954 23273 raft_consensus.cc:482] T 538a228ddfb54bc6a36528d5ff07597b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 15:46:16.611321 23273 raft_consensus.cc:3037] T 538a228ddfb54bc6a36528d5ff07597b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:16.610268 25185 raft_consensus.cc:2932] T 16ca412a0dae4feebe85dd01bf1a9173 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } attrs { replace: true } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } } }
I20260809 15:46:16.615227 23517 raft_consensus.cc:2932] T 16ca412a0dae4feebe85dd01bf1a9173 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } attrs { replace: true } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } } }
I20260809 15:46:16.615973 23394 raft_consensus.cc:2932] T 16ca412a0dae4feebe85dd01bf1a9173 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } attrs { replace: true } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } } }
I20260809 15:46:16.616830 23273 raft_consensus.cc:504] T 538a228ddfb54bc6a36528d5ff07597b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:16.619129 23273 leader_election.cc:288] T 538a228ddfb54bc6a36528d5ff07597b P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 2 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:16.619650 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "538a228ddfb54bc6a36528d5ff07597b" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:16.620182 23527 raft_consensus.cc:3037] T 538a228ddfb54bc6a36528d5ff07597b P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:16.623596 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "538a228ddfb54bc6a36528d5ff07597b" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:16.624107 23402 raft_consensus.cc:3032] T 538a228ddfb54bc6a36528d5ff07597b P c0f38c1908414566aa9115bb2cbdd85d [term 1 LEADER]: Stepping down as leader of term 1
I20260809 15:46:16.624411 23402 raft_consensus.cc:729] T 538a228ddfb54bc6a36528d5ff07597b P c0f38c1908414566aa9115bb2cbdd85d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:46:16.625131 23402 consensus_queue.cc:260] T 538a228ddfb54bc6a36528d5ff07597b P c0f38c1908414566aa9115bb2cbdd85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:16.626230 23402 raft_consensus.cc:3037] T 538a228ddfb54bc6a36528d5ff07597b P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:16.630548 23519 consensus_queue.cc:237] T 7660dfa2ef694927b8eda4100a2bcd9b P 65c36d164b0540f099d592f38307ee6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } }
I20260809 15:46:16.636762 23115 catalog_manager.cc:5495] T 16ca412a0dae4feebe85dd01bf1a9173 P 67b3aeadfa7a432abe8af4d292e04ab9 reported cstate change: config changed from index -1 to 2, NON_VOTER 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) added. New cstate: current_term: 1 leader_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260809 15:46:16.640308 23527 raft_consensus.cc:2454] T 538a228ddfb54bc6a36528d5ff07597b P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 2.
I20260809 15:46:16.642009 23228 leader_election.cc:302] T 538a228ddfb54bc6a36528d5ff07597b P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:46:16.643457 25060 raft_consensus.cc:2781] T 538a228ddfb54bc6a36528d5ff07597b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 15:46:16.643925 25060 raft_consensus.cc:686] T 538a228ddfb54bc6a36528d5ff07597b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 LEADER]: Becoming Leader. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Running, Role: LEADER
I20260809 15:46:16.644734 25060 consensus_queue.cc:237] T 538a228ddfb54bc6a36528d5ff07597b P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:16.645574 25160 raft_consensus.cc:979] T 998c6cee72cf40228c6dba064c0db2da P c0f38c1908414566aa9115bb2cbdd85d: : Instructing follower 67b3aeadfa7a432abe8af4d292e04ab9 to start an election
I20260809 15:46:16.646028 25160 raft_consensus.cc:1067] T 998c6cee72cf40228c6dba064c0db2da P c0f38c1908414566aa9115bb2cbdd85d [term 1 LEADER]: Signalling peer 67b3aeadfa7a432abe8af4d292e04ab9 to start an election
I20260809 15:46:16.647706 23285 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "998c6cee72cf40228c6dba064c0db2da"
dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
from {username='slave'} at 127.0.0.1:53562
I20260809 15:46:16.648150 23285 raft_consensus.cc:482] T 998c6cee72cf40228c6dba064c0db2da P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 15:46:16.648444 23285 raft_consensus.cc:3037] T 998c6cee72cf40228c6dba064c0db2da P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:16.662976 23285 raft_consensus.cc:504] T 998c6cee72cf40228c6dba064c0db2da P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:16.666139 23285 leader_election.cc:288] T 998c6cee72cf40228c6dba064c0db2da P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 2 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:16.666967 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998c6cee72cf40228c6dba064c0db2da" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:16.667538 23527 raft_consensus.cc:3037] T 998c6cee72cf40228c6dba064c0db2da P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:16.668079 23402 raft_consensus.cc:2454] T 538a228ddfb54bc6a36528d5ff07597b P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 2.
I20260809 15:46:16.676757 23527 raft_consensus.cc:2454] T 998c6cee72cf40228c6dba064c0db2da P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 2.
I20260809 15:46:16.677785 23228 leader_election.cc:302] T 998c6cee72cf40228c6dba064c0db2da P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:46:16.679620 25119 raft_consensus.cc:2781] T 998c6cee72cf40228c6dba064c0db2da P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Leader election won for term 2
W20260809 15:46:16.679857 23468 consensus_peers.cc:487] T 7660dfa2ef694927b8eda4100a2bcd9b P 65c36d164b0540f099d592f38307ee6c -> Peer 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689): Couldn't send request to peer 61e69695258f48baa984c9d395e8ce3f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7660dfa2ef694927b8eda4100a2bcd9b. This is attempt 1: this message will repeat every 5th retry.
I20260809 15:46:16.680083 25119 raft_consensus.cc:686] T 998c6cee72cf40228c6dba064c0db2da P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 LEADER]: Becoming Leader. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Running, Role: LEADER
I20260809 15:46:16.680752 23285 raft_consensus.cc:1261] T 7660dfa2ef694927b8eda4100a2bcd9b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 15:46:16.680944 25119 consensus_queue.cc:237] T 998c6cee72cf40228c6dba064c0db2da P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:16.681977 25044 consensus_queue.cc:1035] T 7660dfa2ef694927b8eda4100a2bcd9b P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 15:46:16.688813 23394 raft_consensus.cc:1261] T 7660dfa2ef694927b8eda4100a2bcd9b P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 15:46:16.690276 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998c6cee72cf40228c6dba064c0db2da" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:16.690723 23402 raft_consensus.cc:3032] T 998c6cee72cf40228c6dba064c0db2da P c0f38c1908414566aa9115bb2cbdd85d [term 1 LEADER]: Stepping down as leader of term 1
I20260809 15:46:16.690922 23402 raft_consensus.cc:729] T 998c6cee72cf40228c6dba064c0db2da P c0f38c1908414566aa9115bb2cbdd85d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:46:16.691663 23402 consensus_queue.cc:260] T 998c6cee72cf40228c6dba064c0db2da P c0f38c1908414566aa9115bb2cbdd85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:16.692171 25044 consensus_queue.cc:1035] T 7660dfa2ef694927b8eda4100a2bcd9b P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260809 15:46:16.697145 23402 raft_consensus.cc:3037] T 998c6cee72cf40228c6dba064c0db2da P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:16.704468 25180 raft_consensus.cc:979] T 763e58d6ba644eea830a6c1664d39c9b P 67b3aeadfa7a432abe8af4d292e04ab9: : Instructing follower 65c36d164b0540f099d592f38307ee6c to start an election
I20260809 15:46:16.704919 25180 raft_consensus.cc:1067] T 763e58d6ba644eea830a6c1664d39c9b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 LEADER]: Signalling peer 65c36d164b0540f099d592f38307ee6c to start an election
I20260809 15:46:16.709455 23517 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "763e58d6ba644eea830a6c1664d39c9b"
dest_uuid: "65c36d164b0540f099d592f38307ee6c"
from {username='slave'} at 127.0.0.1:56770
I20260809 15:46:16.709982 23517 raft_consensus.cc:482] T 763e58d6ba644eea830a6c1664d39c9b P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 15:46:16.710285 23517 raft_consensus.cc:3037] T 763e58d6ba644eea830a6c1664d39c9b P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:16.717041 23402 raft_consensus.cc:2454] T 998c6cee72cf40228c6dba064c0db2da P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 2.
I20260809 15:46:16.721042 23517 raft_consensus.cc:504] T 763e58d6ba644eea830a6c1664d39c9b P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:16.723043 23517 leader_election.cc:288] T 763e58d6ba644eea830a6c1664d39c9b P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 2 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:16.723886 23404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "763e58d6ba644eea830a6c1664d39c9b" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:16.723670 23115 catalog_manager.cc:5495] T 538a228ddfb54bc6a36528d5ff07597b P 67b3aeadfa7a432abe8af4d292e04ab9 reported cstate change: term changed from 1 to 2, leader changed from c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66) to 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65). New cstate: current_term: 2 leader_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } health_report { overall_health: UNKNOWN } } }
I20260809 15:46:16.724409 23404 raft_consensus.cc:3037] T 763e58d6ba644eea830a6c1664d39c9b P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:16.723114 25044 raft_consensus.cc:2932] T 7660dfa2ef694927b8eda4100a2bcd9b P 65c36d164b0540f099d592f38307ee6c [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } } }
I20260809 15:46:16.725457 23285 raft_consensus.cc:2932] T 7660dfa2ef694927b8eda4100a2bcd9b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } } }
I20260809 15:46:16.731566 23273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "763e58d6ba644eea830a6c1664d39c9b" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:16.732198 23273 raft_consensus.cc:3032] T 763e58d6ba644eea830a6c1664d39c9b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 LEADER]: Stepping down as leader of term 1
I20260809 15:46:16.740845 23273 raft_consensus.cc:729] T 763e58d6ba644eea830a6c1664d39c9b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Running, Role: LEADER
I20260809 15:46:16.740092 23404 raft_consensus.cc:2454] T 763e58d6ba644eea830a6c1664d39c9b P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 2.
I20260809 15:46:16.741564 23273 consensus_queue.cc:260] T 763e58d6ba644eea830a6c1664d39c9b P 67b3aeadfa7a432abe8af4d292e04ab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:16.742622 25060 raft_consensus.cc:979] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 67b3aeadfa7a432abe8af4d292e04ab9: : Instructing follower 61e69695258f48baa984c9d395e8ce3f to start an election
I20260809 15:46:16.742730 23273 raft_consensus.cc:3037] T 763e58d6ba644eea830a6c1664d39c9b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:16.743908 23472 leader_election.cc:302] T 763e58d6ba644eea830a6c1664d39c9b P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:46:16.746040 25151 raft_consensus.cc:2781] T 763e58d6ba644eea830a6c1664d39c9b P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Leader election won for term 2
I20260809 15:46:16.746506 25151 raft_consensus.cc:686] T 763e58d6ba644eea830a6c1664d39c9b P 65c36d164b0540f099d592f38307ee6c [term 2 LEADER]: Becoming Leader. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:46:16.746829 23115 catalog_manager.cc:5495] T 998c6cee72cf40228c6dba064c0db2da P 67b3aeadfa7a432abe8af4d292e04ab9 reported cstate change: term changed from 1 to 2, leader changed from c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66) to 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65). New cstate: current_term: 2 leader_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } health_report { overall_health: UNKNOWN } } }
I20260809 15:46:16.747370 25151 consensus_queue.cc:237] T 763e58d6ba644eea830a6c1664d39c9b P 65c36d164b0540f099d592f38307ee6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:16.749518 23273 raft_consensus.cc:2454] T 763e58d6ba644eea830a6c1664d39c9b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 2.
I20260809 15:46:16.750777 25060 raft_consensus.cc:1067] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 LEADER]: Signalling peer 61e69695258f48baa984c9d395e8ce3f to start an election
I20260809 15:46:16.752143 23404 raft_consensus.cc:2932] T 7660dfa2ef694927b8eda4100a2bcd9b P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } } }
I20260809 15:46:16.758616 24475 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5712f6b6fffc4cc5a8ddfbe63e4097df"
dest_uuid: "61e69695258f48baa984c9d395e8ce3f"
from {username='slave'} at 127.0.0.1:38722
I20260809 15:46:16.759131 24475 raft_consensus.cc:482] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 15:46:16.759462 24475 raft_consensus.cc:3037] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:16.788654 24475 raft_consensus.cc:504] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 61e69695258f48baa984c9d395e8ce3f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:16.791255 24475 leader_election.cc:288] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 2 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:16.792626 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5712f6b6fffc4cc5a8ddfbe63e4097df" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:16.793200 23527 raft_consensus.cc:3037] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:16.803717 23527 raft_consensus.cc:2454] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 2.
I20260809 15:46:16.804550 24410 leader_election.cc:302] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c; no voters:
I20260809 15:46:16.812464 23118 catalog_manager.cc:5495] T 7660dfa2ef694927b8eda4100a2bcd9b P 65c36d164b0540f099d592f38307ee6c reported cstate change: config changed from index -1 to 2, NON_VOTER 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) added. New cstate: current_term: 1 leader_uuid: "65c36d164b0540f099d592f38307ee6c" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260809 15:46:16.813680 23273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5712f6b6fffc4cc5a8ddfbe63e4097df" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:16.814229 23273 raft_consensus.cc:3032] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 LEADER]: Stepping down as leader of term 1
I20260809 15:46:16.815348 23273 raft_consensus.cc:729] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Running, Role: LEADER
I20260809 15:46:16.816677 25215 raft_consensus.cc:2781] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 61e69695258f48baa984c9d395e8ce3f [term 2 FOLLOWER]: Leader election won for term 2
I20260809 15:46:16.816363 23273 consensus_queue.cc:260] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 67b3aeadfa7a432abe8af4d292e04ab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:16.831182 23273 raft_consensus.cc:3037] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:16.832176 25215 raft_consensus.cc:686] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 61e69695258f48baa984c9d395e8ce3f [term 2 LEADER]: Becoming Leader. State: Replica: 61e69695258f48baa984c9d395e8ce3f, State: Running, Role: LEADER
I20260809 15:46:16.833142 25215 consensus_queue.cc:237] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:16.840123 23273 raft_consensus.cc:2454] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 2.
I20260809 15:46:16.844760 23118 catalog_manager.cc:5495] T 763e58d6ba644eea830a6c1664d39c9b P 65c36d164b0540f099d592f38307ee6c reported cstate change: term changed from 1 to 2, leader changed from 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65) to 65c36d164b0540f099d592f38307ee6c (127.20.202.67). New cstate: current_term: 2 leader_uuid: "65c36d164b0540f099d592f38307ee6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } health_report { overall_health: HEALTHY } } }
I20260809 15:46:16.849442 23115 catalog_manager.cc:5495] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 61e69695258f48baa984c9d395e8ce3f reported cstate change: term changed from 1 to 2, leader changed from 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65) to 61e69695258f48baa984c9d395e8ce3f (127.20.202.68). New cstate: current_term: 2 leader_uuid: "61e69695258f48baa984c9d395e8ce3f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 15:46:16.856709 25208 raft_consensus.cc:979] T e14d118b2eaa42aa8a7cd4025aba7128 P 65c36d164b0540f099d592f38307ee6c: : Instructing follower 61e69695258f48baa984c9d395e8ce3f to start an election
I20260809 15:46:16.857084 25208 raft_consensus.cc:1067] T e14d118b2eaa42aa8a7cd4025aba7128 P 65c36d164b0540f099d592f38307ee6c [term 1 LEADER]: Signalling peer 61e69695258f48baa984c9d395e8ce3f to start an election
I20260809 15:46:16.858632 24475 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e14d118b2eaa42aa8a7cd4025aba7128"
dest_uuid: "61e69695258f48baa984c9d395e8ce3f"
from {username='slave'} at 127.0.0.1:38734
I20260809 15:46:16.859091 24475 raft_consensus.cc:482] T e14d118b2eaa42aa8a7cd4025aba7128 P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 15:46:16.859367 24475 raft_consensus.cc:3037] T e14d118b2eaa42aa8a7cd4025aba7128 P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:16.864051 24475 raft_consensus.cc:504] T e14d118b2eaa42aa8a7cd4025aba7128 P 61e69695258f48baa984c9d395e8ce3f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:16.865607 24475 leader_election.cc:288] T e14d118b2eaa42aa8a7cd4025aba7128 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 2 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:16.866524 23519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e14d118b2eaa42aa8a7cd4025aba7128" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:16.867072 23519 raft_consensus.cc:3032] T e14d118b2eaa42aa8a7cd4025aba7128 P 65c36d164b0540f099d592f38307ee6c [term 1 LEADER]: Stepping down as leader of term 1
I20260809 15:46:16.867369 23519 raft_consensus.cc:729] T e14d118b2eaa42aa8a7cd4025aba7128 P 65c36d164b0540f099d592f38307ee6c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:46:16.868003 23519 consensus_queue.cc:260] T e14d118b2eaa42aa8a7cd4025aba7128 P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:16.869088 23519 raft_consensus.cc:3037] T e14d118b2eaa42aa8a7cd4025aba7128 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:16.869894 23273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e14d118b2eaa42aa8a7cd4025aba7128" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:16.870402 23273 raft_consensus.cc:3037] T e14d118b2eaa42aa8a7cd4025aba7128 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:16.874948 23519 raft_consensus.cc:2454] T e14d118b2eaa42aa8a7cd4025aba7128 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 2.
I20260809 15:46:16.876027 23273 raft_consensus.cc:2454] T e14d118b2eaa42aa8a7cd4025aba7128 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 2.
I20260809 15:46:16.876277 24410 leader_election.cc:302] T e14d118b2eaa42aa8a7cd4025aba7128 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c; no voters:
I20260809 15:46:16.877315 25215 raft_consensus.cc:2781] T e14d118b2eaa42aa8a7cd4025aba7128 P 61e69695258f48baa984c9d395e8ce3f [term 2 FOLLOWER]: Leader election won for term 2
I20260809 15:46:16.877755 25215 raft_consensus.cc:686] T e14d118b2eaa42aa8a7cd4025aba7128 P 61e69695258f48baa984c9d395e8ce3f [term 2 LEADER]: Becoming Leader. State: Replica: 61e69695258f48baa984c9d395e8ce3f, State: Running, Role: LEADER
I20260809 15:46:16.878510 25215 consensus_queue.cc:237] T e14d118b2eaa42aa8a7cd4025aba7128 P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:16.889941 23115 catalog_manager.cc:5495] T e14d118b2eaa42aa8a7cd4025aba7128 P 61e69695258f48baa984c9d395e8ce3f reported cstate change: term changed from 1 to 2, leader changed from 65c36d164b0540f099d592f38307ee6c (127.20.202.67) to 61e69695258f48baa984c9d395e8ce3f (127.20.202.68). New cstate: current_term: 2 leader_uuid: "61e69695258f48baa984c9d395e8ce3f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 15:46:16.909482 25224 ts_tablet_manager.cc:927] T 563daf42d7784db195ea99c3cb21b370 P 61e69695258f48baa984c9d395e8ce3f: Initiating tablet copy from peer 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459)
I20260809 15:46:16.911221 25224 tablet_copy_client.cc:316] T 563daf42d7784db195ea99c3cb21b370 P 61e69695258f48baa984c9d395e8ce3f: tablet copy: Beginning tablet copy session from remote peer at address 127.20.202.65:35459
I20260809 15:46:16.913383 23298 tablet_copy_service.cc:140] P 67b3aeadfa7a432abe8af4d292e04ab9: Received BeginTabletCopySession request for tablet 563daf42d7784db195ea99c3cb21b370 from peer 61e69695258f48baa984c9d395e8ce3f ({username='slave'} at 127.0.0.1:55434)
I20260809 15:46:16.913834 23298 tablet_copy_service.cc:161] P 67b3aeadfa7a432abe8af4d292e04ab9: Beginning new tablet copy session on tablet 563daf42d7784db195ea99c3cb21b370 from peer 61e69695258f48baa984c9d395e8ce3f at {username='slave'} at 127.0.0.1:55434: session id = 61e69695258f48baa984c9d395e8ce3f-563daf42d7784db195ea99c3cb21b370
I20260809 15:46:16.920295 23298 tablet_copy_source_session.cc:205] T 563daf42d7784db195ea99c3cb21b370 P 67b3aeadfa7a432abe8af4d292e04ab9: Tablet Copy: opened 0 blocks and 1 log segments
I20260809 15:46:16.922739 25224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 563daf42d7784db195ea99c3cb21b370. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:46:16.935120 25224 tablet_copy_client.cc:791] T 563daf42d7784db195ea99c3cb21b370 P 61e69695258f48baa984c9d395e8ce3f: tablet copy: Starting download of 0 data blocks...
I20260809 15:46:16.935658 25224 tablet_copy_client.cc:655] T 563daf42d7784db195ea99c3cb21b370 P 61e69695258f48baa984c9d395e8ce3f: tablet copy: Starting download of 1 WAL segments...
I20260809 15:46:16.941089 25224 tablet_copy_client.cc:531] T 563daf42d7784db195ea99c3cb21b370 P 61e69695258f48baa984c9d395e8ce3f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260809 15:46:16.950378 25232 ts_tablet_manager.cc:927] T 16ca412a0dae4feebe85dd01bf1a9173 P 61e69695258f48baa984c9d395e8ce3f: Initiating tablet copy from peer 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459)
I20260809 15:46:16.950574 25229 ts_tablet_manager.cc:927] T 472816994d814cb198b69e5248697440 P 61e69695258f48baa984c9d395e8ce3f: Initiating tablet copy from peer 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:16.951318 25224 tablet_bootstrap.cc:492] T 563daf42d7784db195ea99c3cb21b370 P 61e69695258f48baa984c9d395e8ce3f: Bootstrap starting.
I20260809 15:46:16.958679 25232 tablet_copy_client.cc:316] T 16ca412a0dae4feebe85dd01bf1a9173 P 61e69695258f48baa984c9d395e8ce3f: tablet copy: Beginning tablet copy session from remote peer at address 127.20.202.65:35459
I20260809 15:46:16.961452 25229 tablet_copy_client.cc:316] T 472816994d814cb198b69e5248697440 P 61e69695258f48baa984c9d395e8ce3f: tablet copy: Beginning tablet copy session from remote peer at address 127.20.202.67:40553
I20260809 15:46:16.962952 23546 tablet_copy_service.cc:140] P 65c36d164b0540f099d592f38307ee6c: Received BeginTabletCopySession request for tablet 472816994d814cb198b69e5248697440 from peer 61e69695258f48baa984c9d395e8ce3f ({username='slave'} at 127.0.0.1:44018)
I20260809 15:46:16.963526 23546 tablet_copy_service.cc:161] P 65c36d164b0540f099d592f38307ee6c: Beginning new tablet copy session on tablet 472816994d814cb198b69e5248697440 from peer 61e69695258f48baa984c9d395e8ce3f at {username='slave'} at 127.0.0.1:44018: session id = 61e69695258f48baa984c9d395e8ce3f-472816994d814cb198b69e5248697440
I20260809 15:46:16.991027 23298 tablet_copy_service.cc:140] P 67b3aeadfa7a432abe8af4d292e04ab9: Received BeginTabletCopySession request for tablet 16ca412a0dae4feebe85dd01bf1a9173 from peer 61e69695258f48baa984c9d395e8ce3f ({username='slave'} at 127.0.0.1:55434)
I20260809 15:46:16.992393 23298 tablet_copy_service.cc:161] P 67b3aeadfa7a432abe8af4d292e04ab9: Beginning new tablet copy session on tablet 16ca412a0dae4feebe85dd01bf1a9173 from peer 61e69695258f48baa984c9d395e8ce3f at {username='slave'} at 127.0.0.1:55434: session id = 61e69695258f48baa984c9d395e8ce3f-16ca412a0dae4feebe85dd01bf1a9173
I20260809 15:46:17.015362 25224 tablet_bootstrap.cc:492] T 563daf42d7784db195ea99c3cb21b370 P 61e69695258f48baa984c9d395e8ce3f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 15:46:17.016783 25224 tablet_bootstrap.cc:492] T 563daf42d7784db195ea99c3cb21b370 P 61e69695258f48baa984c9d395e8ce3f: Bootstrap complete.
I20260809 15:46:17.017953 25224 ts_tablet_manager.cc:1397] T 563daf42d7784db195ea99c3cb21b370 P 61e69695258f48baa984c9d395e8ce3f: Time spent bootstrapping tablet: real 0.067s user 0.017s sys 0.009s
I20260809 15:46:17.021617 23546 tablet_copy_source_session.cc:205] T 472816994d814cb198b69e5248697440 P 65c36d164b0540f099d592f38307ee6c: Tablet Copy: opened 0 blocks and 1 log segments
I20260809 15:46:17.022691 25224 raft_consensus.cc:348] T 563daf42d7784db195ea99c3cb21b370 P 61e69695258f48baa984c9d395e8ce3f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } attrs { replace: true } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } }
I20260809 15:46:17.024592 25224 raft_consensus.cc:729] T 563daf42d7784db195ea99c3cb21b370 P 61e69695258f48baa984c9d395e8ce3f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 61e69695258f48baa984c9d395e8ce3f, State: Initialized, Role: LEARNER
I20260809 15:46:17.025118 23298 tablet_copy_source_session.cc:205] T 16ca412a0dae4feebe85dd01bf1a9173 P 67b3aeadfa7a432abe8af4d292e04ab9: Tablet Copy: opened 0 blocks and 1 log segments
I20260809 15:46:17.025420 25224 consensus_queue.cc:260] T 563daf42d7784db195ea99c3cb21b370 P 61e69695258f48baa984c9d395e8ce3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } attrs { replace: true } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } }
I20260809 15:46:17.044382 25224 ts_tablet_manager.cc:1428] T 563daf42d7784db195ea99c3cb21b370 P 61e69695258f48baa984c9d395e8ce3f: Time spent starting tablet: real 0.026s user 0.011s sys 0.002s
I20260809 15:46:17.048151 25232 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16ca412a0dae4feebe85dd01bf1a9173. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:46:17.056313 23298 tablet_copy_service.cc:342] P 67b3aeadfa7a432abe8af4d292e04ab9: Request end of tablet copy session 61e69695258f48baa984c9d395e8ce3f-563daf42d7784db195ea99c3cb21b370 received from {username='slave'} at 127.0.0.1:55434
I20260809 15:46:17.056794 23298 tablet_copy_service.cc:433] P 67b3aeadfa7a432abe8af4d292e04ab9: ending tablet copy session 61e69695258f48baa984c9d395e8ce3f-563daf42d7784db195ea99c3cb21b370 on tablet 563daf42d7784db195ea99c3cb21b370 with peer 61e69695258f48baa984c9d395e8ce3f
I20260809 15:46:17.059768 25229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 472816994d814cb198b69e5248697440. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:46:17.099408 25232 tablet_copy_client.cc:791] T 16ca412a0dae4feebe85dd01bf1a9173 P 61e69695258f48baa984c9d395e8ce3f: tablet copy: Starting download of 0 data blocks...
I20260809 15:46:17.100016 25232 tablet_copy_client.cc:655] T 16ca412a0dae4feebe85dd01bf1a9173 P 61e69695258f48baa984c9d395e8ce3f: tablet copy: Starting download of 1 WAL segments...
I20260809 15:46:17.108489 25232 tablet_copy_client.cc:531] T 16ca412a0dae4feebe85dd01bf1a9173 P 61e69695258f48baa984c9d395e8ce3f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260809 15:46:17.133896 25229 tablet_copy_client.cc:791] T 472816994d814cb198b69e5248697440 P 61e69695258f48baa984c9d395e8ce3f: tablet copy: Starting download of 0 data blocks...
I20260809 15:46:17.134605 25229 tablet_copy_client.cc:655] T 472816994d814cb198b69e5248697440 P 61e69695258f48baa984c9d395e8ce3f: tablet copy: Starting download of 1 WAL segments...
I20260809 15:46:17.141276 25232 tablet_bootstrap.cc:492] T 16ca412a0dae4feebe85dd01bf1a9173 P 61e69695258f48baa984c9d395e8ce3f: Bootstrap starting.
I20260809 15:46:17.144438 25229 tablet_copy_client.cc:531] T 472816994d814cb198b69e5248697440 P 61e69695258f48baa984c9d395e8ce3f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260809 15:46:17.148422 25224 ts_tablet_manager.cc:927] T 7660dfa2ef694927b8eda4100a2bcd9b P 61e69695258f48baa984c9d395e8ce3f: Initiating tablet copy from peer 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:17.163925 25229 tablet_bootstrap.cc:492] T 472816994d814cb198b69e5248697440 P 61e69695258f48baa984c9d395e8ce3f: Bootstrap starting.
I20260809 15:46:17.178500 25224 tablet_copy_client.cc:316] T 7660dfa2ef694927b8eda4100a2bcd9b P 61e69695258f48baa984c9d395e8ce3f: tablet copy: Beginning tablet copy session from remote peer at address 127.20.202.67:40553
I20260809 15:46:17.181726 23546 tablet_copy_service.cc:140] P 65c36d164b0540f099d592f38307ee6c: Received BeginTabletCopySession request for tablet 7660dfa2ef694927b8eda4100a2bcd9b from peer 61e69695258f48baa984c9d395e8ce3f ({username='slave'} at 127.0.0.1:44018)
I20260809 15:46:17.182278 23546 tablet_copy_service.cc:161] P 65c36d164b0540f099d592f38307ee6c: Beginning new tablet copy session on tablet 7660dfa2ef694927b8eda4100a2bcd9b from peer 61e69695258f48baa984c9d395e8ce3f at {username='slave'} at 127.0.0.1:44018: session id = 61e69695258f48baa984c9d395e8ce3f-7660dfa2ef694927b8eda4100a2bcd9b
I20260809 15:46:17.192135 23402 raft_consensus.cc:1261] T 998c6cee72cf40228c6dba064c0db2da P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Refusing update from remote peer 67b3aeadfa7a432abe8af4d292e04ab9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 15:46:17.201489 25060 consensus_queue.cc:1035] T 998c6cee72cf40228c6dba064c0db2da P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260809 15:46:17.205673 23546 tablet_copy_source_session.cc:205] T 7660dfa2ef694927b8eda4100a2bcd9b P 65c36d164b0540f099d592f38307ee6c: Tablet Copy: opened 0 blocks and 1 log segments
I20260809 15:46:17.212997 25224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7660dfa2ef694927b8eda4100a2bcd9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:46:17.213445 23394 raft_consensus.cc:1261] T 763e58d6ba644eea830a6c1664d39c9b P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 15:46:17.215317 25044 consensus_queue.cc:1035] T 763e58d6ba644eea830a6c1664d39c9b P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260809 15:46:17.243978 23519 raft_consensus.cc:1261] T 998c6cee72cf40228c6dba064c0db2da P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Refusing update from remote peer 67b3aeadfa7a432abe8af4d292e04ab9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 15:46:17.246002 25060 consensus_queue.cc:1035] T 998c6cee72cf40228c6dba064c0db2da P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 15:46:17.254794 23404 raft_consensus.cc:1261] T 538a228ddfb54bc6a36528d5ff07597b P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Refusing update from remote peer 67b3aeadfa7a432abe8af4d292e04ab9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 15:46:17.265534 25060 consensus_queue.cc:1035] T 538a228ddfb54bc6a36528d5ff07597b P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 15:46:17.268424 25232 tablet_bootstrap.cc:492] T 16ca412a0dae4feebe85dd01bf1a9173 P 61e69695258f48baa984c9d395e8ce3f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 15:46:17.271708 25232 tablet_bootstrap.cc:492] T 16ca412a0dae4feebe85dd01bf1a9173 P 61e69695258f48baa984c9d395e8ce3f: Bootstrap complete.
I20260809 15:46:17.272538 25232 ts_tablet_manager.cc:1397] T 16ca412a0dae4feebe85dd01bf1a9173 P 61e69695258f48baa984c9d395e8ce3f: Time spent bootstrapping tablet: real 0.131s user 0.021s sys 0.027s
I20260809 15:46:17.281395 23526 raft_consensus.cc:1261] T 538a228ddfb54bc6a36528d5ff07597b P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Refusing update from remote peer 67b3aeadfa7a432abe8af4d292e04ab9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 15:46:17.283072 25119 consensus_queue.cc:1035] T 538a228ddfb54bc6a36528d5ff07597b P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260809 15:46:17.307801 23274 raft_consensus.cc:1261] T 763e58d6ba644eea830a6c1664d39c9b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 15:46:17.326969 25224 tablet_copy_client.cc:791] T 7660dfa2ef694927b8eda4100a2bcd9b P 61e69695258f48baa984c9d395e8ce3f: tablet copy: Starting download of 0 data blocks...
I20260809 15:46:17.340360 25232 raft_consensus.cc:348] T 16ca412a0dae4feebe85dd01bf1a9173 P 61e69695258f48baa984c9d395e8ce3f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } attrs { replace: true } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } }
I20260809 15:46:17.339020 25224 tablet_copy_client.cc:655] T 7660dfa2ef694927b8eda4100a2bcd9b P 61e69695258f48baa984c9d395e8ce3f: tablet copy: Starting download of 1 WAL segments...
I20260809 15:46:17.344518 25232 raft_consensus.cc:729] T 16ca412a0dae4feebe85dd01bf1a9173 P 61e69695258f48baa984c9d395e8ce3f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 61e69695258f48baa984c9d395e8ce3f, State: Initialized, Role: LEARNER
I20260809 15:46:17.440595 23274 raft_consensus.cc:1261] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Refusing update from remote peer 61e69695258f48baa984c9d395e8ce3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260809 15:46:17.453516 25232 consensus_queue.cc:260] T 16ca412a0dae4feebe85dd01bf1a9173 P 61e69695258f48baa984c9d395e8ce3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } attrs { replace: true } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } }
I20260809 15:46:17.527469 25229 tablet_bootstrap.cc:492] T 472816994d814cb198b69e5248697440 P 61e69695258f48baa984c9d395e8ce3f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 15:46:17.528419 25229 tablet_bootstrap.cc:492] T 472816994d814cb198b69e5248697440 P 61e69695258f48baa984c9d395e8ce3f: Bootstrap complete.
I20260809 15:46:17.529124 25229 ts_tablet_manager.cc:1397] T 472816994d814cb198b69e5248697440 P 61e69695258f48baa984c9d395e8ce3f: Time spent bootstrapping tablet: real 0.365s user 0.026s sys 0.062s
I20260809 15:46:17.531036 25232 ts_tablet_manager.cc:1428] T 16ca412a0dae4feebe85dd01bf1a9173 P 61e69695258f48baa984c9d395e8ce3f: Time spent starting tablet: real 0.258s user 0.011s sys 0.005s
I20260809 15:46:17.554950 25229 raft_consensus.cc:348] T 472816994d814cb198b69e5248697440 P 61e69695258f48baa984c9d395e8ce3f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } }
I20260809 15:46:17.555850 25229 raft_consensus.cc:729] T 472816994d814cb198b69e5248697440 P 61e69695258f48baa984c9d395e8ce3f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 61e69695258f48baa984c9d395e8ce3f, State: Initialized, Role: LEARNER
I20260809 15:46:17.556565 25229 consensus_queue.cc:260] T 472816994d814cb198b69e5248697440 P 61e69695258f48baa984c9d395e8ce3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } }
I20260809 15:46:17.603214 25229 ts_tablet_manager.cc:1428] T 472816994d814cb198b69e5248697440 P 61e69695258f48baa984c9d395e8ce3f: Time spent starting tablet: real 0.070s user 0.007s sys 0.000s
I20260809 15:46:17.632797 25215 consensus_queue.cc:1035] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260809 15:46:17.633463 23527 raft_consensus.cc:1261] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Refusing update from remote peer 61e69695258f48baa984c9d395e8ce3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260809 15:46:17.639520 23517 raft_consensus.cc:1261] T e14d118b2eaa42aa8a7cd4025aba7128 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Refusing update from remote peer 61e69695258f48baa984c9d395e8ce3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260809 15:46:17.657028 24472 raft_consensus.cc:1203] T 472816994d814cb198b69e5248697440 P 61e69695258f48baa984c9d395e8ce3f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
W20260809 15:46:17.661540 23470 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 322713us
I20260809 15:46:17.687054 24472 raft_consensus.cc:1203] T 16ca412a0dae4feebe85dd01bf1a9173 P 61e69695258f48baa984c9d395e8ce3f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260809 15:46:17.688143 23546 tablet_copy_service.cc:342] P 65c36d164b0540f099d592f38307ee6c: Request end of tablet copy session 61e69695258f48baa984c9d395e8ce3f-472816994d814cb198b69e5248697440 received from {username='slave'} at 127.0.0.1:44018
I20260809 15:46:17.688562 23546 tablet_copy_service.cc:433] P 65c36d164b0540f099d592f38307ee6c: ending tablet copy session 61e69695258f48baa984c9d395e8ce3f-472816994d814cb198b69e5248697440 on tablet 472816994d814cb198b69e5248697440 with peer 61e69695258f48baa984c9d395e8ce3f
I20260809 15:46:17.689796 25215 consensus_queue.cc:1035] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260809 15:46:17.696676 24475 raft_consensus.cc:1203] T 563daf42d7784db195ea99c3cb21b370 P 61e69695258f48baa984c9d395e8ce3f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260809 15:46:17.696607 21289 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20260809 15:46:17.778528 23223 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.68:34689, user_credentials={real_user=slave}} blocked reactor thread for 60977.3us
W20260809 15:46:17.817741 24410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 128386us
I20260809 15:46:17.778591 25253 consensus_queue.cc:1035] T 763e58d6ba644eea830a6c1664d39c9b P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.116s
I20260809 15:46:17.877765 25224 tablet_copy_client.cc:531] T 7660dfa2ef694927b8eda4100a2bcd9b P 61e69695258f48baa984c9d395e8ce3f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260809 15:46:17.872466 25215 consensus_queue.cc:1035] T e14d118b2eaa42aa8a7cd4025aba7128 P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.023s
W20260809 15:46:17.956070 23345 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.68:34689, user_credentials={real_user=slave}} blocked reactor thread for 217303us
I20260809 15:46:17.968168 25224 tablet_bootstrap.cc:492] T 7660dfa2ef694927b8eda4100a2bcd9b P 61e69695258f48baa984c9d395e8ce3f: Bootstrap starting.
W20260809 15:46:18.013877 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 119808us
I20260809 15:46:18.044023 23298 tablet_copy_service.cc:342] P 67b3aeadfa7a432abe8af4d292e04ab9: Request end of tablet copy session 61e69695258f48baa984c9d395e8ce3f-16ca412a0dae4feebe85dd01bf1a9173 received from {username='slave'} at 127.0.0.1:55434
I20260809 15:46:18.044415 23298 tablet_copy_service.cc:433] P 67b3aeadfa7a432abe8af4d292e04ab9: ending tablet copy session 61e69695258f48baa984c9d395e8ce3f-16ca412a0dae4feebe85dd01bf1a9173 on tablet 16ca412a0dae4feebe85dd01bf1a9173 with peer 61e69695258f48baa984c9d395e8ce3f
I20260809 15:46:18.046667 23274 raft_consensus.cc:1261] T e14d118b2eaa42aa8a7cd4025aba7128 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Refusing update from remote peer 61e69695258f48baa984c9d395e8ce3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
W20260809 15:46:18.053061 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 121559us
I20260809 15:46:18.090628 25215 consensus_queue.cc:1035] T e14d118b2eaa42aa8a7cd4025aba7128 P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260809 15:46:18.167682 25185 raft_consensus.cc:1050] T 563daf42d7784db195ea99c3cb21b370 P 67b3aeadfa7a432abe8af4d292e04ab9: attempting to promote NON_VOTER 61e69695258f48baa984c9d395e8ce3f to VOTER
I20260809 15:46:18.171763 25185 consensus_queue.cc:237] T 563daf42d7784db195ea99c3cb21b370 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } attrs { replace: true } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:18.186980 24472 raft_consensus.cc:1261] T 563daf42d7784db195ea99c3cb21b370 P 61e69695258f48baa984c9d395e8ce3f [term 1 LEARNER]: Refusing update from remote peer 67b3aeadfa7a432abe8af4d292e04ab9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 15:46:18.191305 23519 raft_consensus.cc:1261] T 563daf42d7784db195ea99c3cb21b370 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Refusing update from remote peer 67b3aeadfa7a432abe8af4d292e04ab9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 15:46:18.202513 25279 consensus_queue.cc:1035] T 563daf42d7784db195ea99c3cb21b370 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.008s
I20260809 15:46:18.210783 25119 consensus_queue.cc:1035] T 563daf42d7784db195ea99c3cb21b370 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 15:46:18.222518 23400 raft_consensus.cc:1261] T 563daf42d7784db195ea99c3cb21b370 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Refusing update from remote peer 67b3aeadfa7a432abe8af4d292e04ab9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 15:46:18.228029 23400 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "00a1881267a14ba8b78c81040f17a014"
dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
mode: GRACEFUL
new_leader_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
from {username='slave'} at 127.0.0.1:51280
I20260809 15:46:18.228775 23400 raft_consensus.cc:595] T 00a1881267a14ba8b78c81040f17a014 P c0f38c1908414566aa9115bb2cbdd85d [term 1 LEADER]: Received request to transfer leadership to TS 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:18.229584 25224 tablet_bootstrap.cc:492] T 7660dfa2ef694927b8eda4100a2bcd9b P 61e69695258f48baa984c9d395e8ce3f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 15:46:18.230480 25224 tablet_bootstrap.cc:492] T 7660dfa2ef694927b8eda4100a2bcd9b P 61e69695258f48baa984c9d395e8ce3f: Bootstrap complete.
I20260809 15:46:18.231127 25224 ts_tablet_manager.cc:1397] T 7660dfa2ef694927b8eda4100a2bcd9b P 61e69695258f48baa984c9d395e8ce3f: Time spent bootstrapping tablet: real 0.268s user 0.028s sys 0.030s
I20260809 15:46:18.233207 23522 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "294d4bcc1e7a477cb56455581277a60f"
dest_uuid: "65c36d164b0540f099d592f38307ee6c"
mode: GRACEFUL
new_leader_uuid: "61e69695258f48baa984c9d395e8ce3f"
from {username='slave'} at 127.0.0.1:48888
I20260809 15:46:18.233722 23522 raft_consensus.cc:595] T 294d4bcc1e7a477cb56455581277a60f P 65c36d164b0540f099d592f38307ee6c [term 1 LEADER]: Received request to transfer leadership to TS 61e69695258f48baa984c9d395e8ce3f
I20260809 15:46:18.241475 25201 consensus_queue.cc:1035] T 563daf42d7784db195ea99c3cb21b370 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260809 15:46:18.241991 23400 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "30c1740ef6874df0a332fbd6fd4b03ce"
dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
mode: GRACEFUL
new_leader_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
from {username='slave'} at 127.0.0.1:51280
I20260809 15:46:18.242552 23400 raft_consensus.cc:595] T 30c1740ef6874df0a332fbd6fd4b03ce P c0f38c1908414566aa9115bb2cbdd85d [term 1 LEADER]: Received request to transfer leadership to TS 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:18.247881 25224 raft_consensus.cc:348] T 7660dfa2ef694927b8eda4100a2bcd9b P 61e69695258f48baa984c9d395e8ce3f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } }
I20260809 15:46:18.248749 25224 raft_consensus.cc:729] T 7660dfa2ef694927b8eda4100a2bcd9b P 61e69695258f48baa984c9d395e8ce3f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 61e69695258f48baa984c9d395e8ce3f, State: Initialized, Role: LEARNER
I20260809 15:46:18.249393 25224 consensus_queue.cc:260] T 7660dfa2ef694927b8eda4100a2bcd9b P 61e69695258f48baa984c9d395e8ce3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } }
I20260809 15:46:18.251816 25224 ts_tablet_manager.cc:1428] T 7660dfa2ef694927b8eda4100a2bcd9b P 61e69695258f48baa984c9d395e8ce3f: Time spent starting tablet: real 0.020s user 0.006s sys 0.000s
I20260809 15:46:18.252532 25279 raft_consensus.cc:2932] T 563daf42d7784db195ea99c3cb21b370 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } attrs { replace: true } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:18.253665 23546 tablet_copy_service.cc:342] P 65c36d164b0540f099d592f38307ee6c: Request end of tablet copy session 61e69695258f48baa984c9d395e8ce3f-7660dfa2ef694927b8eda4100a2bcd9b received from {username='slave'} at 127.0.0.1:44018
I20260809 15:46:18.254088 23546 tablet_copy_service.cc:433] P 65c36d164b0540f099d592f38307ee6c: ending tablet copy session 61e69695258f48baa984c9d395e8ce3f-7660dfa2ef694927b8eda4100a2bcd9b on tablet 7660dfa2ef694927b8eda4100a2bcd9b with peer 61e69695258f48baa984c9d395e8ce3f
I20260809 15:46:18.259045 23274 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8e4ffcf83c234ae7910ddef3ec0b4bda"
dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
mode: GRACEFUL
new_leader_uuid: "61e69695258f48baa984c9d395e8ce3f"
from {username='slave'} at 127.0.0.1:33468
I20260809 15:46:18.263617 23274 raft_consensus.cc:595] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 LEADER]: Received request to transfer leadership to TS 61e69695258f48baa984c9d395e8ce3f
I20260809 15:46:18.264612 23522 raft_consensus.cc:2932] T 563daf42d7784db195ea99c3cb21b370 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } attrs { replace: true } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:18.271986 24472 raft_consensus.cc:2932] T 563daf42d7784db195ea99c3cb21b370 P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } attrs { replace: true } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:18.270524 23400 raft_consensus.cc:2932] T 563daf42d7784db195ea99c3cb21b370 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } attrs { replace: true } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:18.284586 23285 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "998c6cee72cf40228c6dba064c0db2da"
dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
mode: GRACEFUL
new_leader_uuid: "65c36d164b0540f099d592f38307ee6c"
from {username='slave'} at 127.0.0.1:33468
I20260809 15:46:18.285094 23285 raft_consensus.cc:595] T 998c6cee72cf40228c6dba064c0db2da P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 LEADER]: Received request to transfer leadership to TS 65c36d164b0540f099d592f38307ee6c
I20260809 15:46:18.287052 21289 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260809 15:46:18.288619 21289 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 15:46:18.285101 23115 catalog_manager.cc:5495] T 563daf42d7784db195ea99c3cb21b370 P 67b3aeadfa7a432abe8af4d292e04ab9 reported cstate change: config changed from index 2 to 3, 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 15:46:18.289361 21289 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 15:46:18.358052 24472 raft_consensus.cc:1203] T 7660dfa2ef694927b8eda4100a2bcd9b P 61e69695258f48baa984c9d395e8ce3f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260809 15:46:18.397481 25284 raft_consensus.cc:979] T 30c1740ef6874df0a332fbd6fd4b03ce P c0f38c1908414566aa9115bb2cbdd85d: : Instructing follower 67b3aeadfa7a432abe8af4d292e04ab9 to start an election
I20260809 15:46:18.398355 25194 raft_consensus.cc:1067] T 30c1740ef6874df0a332fbd6fd4b03ce P c0f38c1908414566aa9115bb2cbdd85d [term 1 LEADER]: Signalling peer 67b3aeadfa7a432abe8af4d292e04ab9 to start an election
I20260809 15:46:18.403086 23270 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "30c1740ef6874df0a332fbd6fd4b03ce"
dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
from {username='slave'} at 127.0.0.1:53562
I20260809 15:46:18.404518 23270 raft_consensus.cc:482] T 30c1740ef6874df0a332fbd6fd4b03ce P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 15:46:18.404788 23270 raft_consensus.cc:3037] T 30c1740ef6874df0a332fbd6fd4b03ce P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:18.412853 23270 raft_consensus.cc:504] T 30c1740ef6874df0a332fbd6fd4b03ce P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:18.417511 25201 raft_consensus.cc:979] T 998c6cee72cf40228c6dba064c0db2da P 67b3aeadfa7a432abe8af4d292e04ab9: : Instructing follower 65c36d164b0540f099d592f38307ee6c to start an election
I20260809 15:46:18.417721 23270 leader_election.cc:288] T 30c1740ef6874df0a332fbd6fd4b03ce P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 2 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:18.418155 25060 raft_consensus.cc:1067] T 998c6cee72cf40228c6dba064c0db2da P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 LEADER]: Signalling peer 65c36d164b0540f099d592f38307ee6c to start an election
I20260809 15:46:18.419492 23400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30c1740ef6874df0a332fbd6fd4b03ce" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:18.419961 23519 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "998c6cee72cf40228c6dba064c0db2da"
dest_uuid: "65c36d164b0540f099d592f38307ee6c"
from {username='slave'} at 127.0.0.1:56770
I20260809 15:46:18.420116 23400 raft_consensus.cc:3032] T 30c1740ef6874df0a332fbd6fd4b03ce P c0f38c1908414566aa9115bb2cbdd85d [term 1 LEADER]: Stepping down as leader of term 1
I20260809 15:46:18.420499 23519 raft_consensus.cc:482] T 998c6cee72cf40228c6dba064c0db2da P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 15:46:18.420868 23519 raft_consensus.cc:3037] T 998c6cee72cf40228c6dba064c0db2da P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:18.420632 23400 raft_consensus.cc:729] T 30c1740ef6874df0a332fbd6fd4b03ce P c0f38c1908414566aa9115bb2cbdd85d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:46:18.421483 23522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30c1740ef6874df0a332fbd6fd4b03ce" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:18.421917 23400 consensus_queue.cc:260] T 30c1740ef6874df0a332fbd6fd4b03ce P c0f38c1908414566aa9115bb2cbdd85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:18.422031 23522 raft_consensus.cc:3037] T 30c1740ef6874df0a332fbd6fd4b03ce P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:18.423712 23400 raft_consensus.cc:3037] T 30c1740ef6874df0a332fbd6fd4b03ce P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:18.437028 25060 raft_consensus.cc:979] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 67b3aeadfa7a432abe8af4d292e04ab9: : Instructing follower 61e69695258f48baa984c9d395e8ce3f to start an election
I20260809 15:46:18.437557 25060 raft_consensus.cc:1067] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 LEADER]: Signalling peer 61e69695258f48baa984c9d395e8ce3f to start an election
I20260809 15:46:18.438292 25060 raft_consensus.cc:1050] T 16ca412a0dae4feebe85dd01bf1a9173 P 67b3aeadfa7a432abe8af4d292e04ab9: attempting to promote NON_VOTER 61e69695258f48baa984c9d395e8ce3f to VOTER
I20260809 15:46:18.440711 25060 consensus_queue.cc:237] T 16ca412a0dae4feebe85dd01bf1a9173 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } attrs { replace: true } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:18.437009 23522 raft_consensus.cc:2454] T 30c1740ef6874df0a332fbd6fd4b03ce P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 2.
I20260809 15:46:18.445889 23519 raft_consensus.cc:504] T 998c6cee72cf40228c6dba064c0db2da P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:18.447516 24475 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8e4ffcf83c234ae7910ddef3ec0b4bda"
dest_uuid: "61e69695258f48baa984c9d395e8ce3f"
from {username='slave'} at 127.0.0.1:38722
I20260809 15:46:18.448052 24475 raft_consensus.cc:482] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 15:46:18.448356 24475 raft_consensus.cc:3037] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:18.455098 24475 raft_consensus.cc:504] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 61e69695258f48baa984c9d395e8ce3f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:18.458412 23400 raft_consensus.cc:2454] T 30c1740ef6874df0a332fbd6fd4b03ce P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 2.
I20260809 15:46:18.460052 23228 leader_election.cc:302] T 30c1740ef6874df0a332fbd6fd4b03ce P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:46:18.460865 25185 raft_consensus.cc:2781] T 30c1740ef6874df0a332fbd6fd4b03ce P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 15:46:18.461402 25185 raft_consensus.cc:686] T 30c1740ef6874df0a332fbd6fd4b03ce P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 LEADER]: Becoming Leader. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Running, Role: LEADER
I20260809 15:46:18.462497 23274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998c6cee72cf40228c6dba064c0db2da" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:18.463042 23274 raft_consensus.cc:3032] T 998c6cee72cf40228c6dba064c0db2da P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 LEADER]: Stepping down as leader of term 2
I20260809 15:46:18.462679 25185 consensus_queue.cc:237] T 30c1740ef6874df0a332fbd6fd4b03ce P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:18.463376 23274 raft_consensus.cc:729] T 998c6cee72cf40228c6dba064c0db2da P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Running, Role: LEADER
I20260809 15:46:18.463541 23396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998c6cee72cf40228c6dba064c0db2da" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:18.465175 23396 raft_consensus.cc:3037] T 998c6cee72cf40228c6dba064c0db2da P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:18.465011 23274 consensus_queue.cc:260] T 998c6cee72cf40228c6dba064c0db2da P 67b3aeadfa7a432abe8af4d292e04ab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:18.466152 23274 raft_consensus.cc:3037] T 998c6cee72cf40228c6dba064c0db2da P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:18.467371 23519 leader_election.cc:288] T 998c6cee72cf40228c6dba064c0db2da P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:18.473192 24475 leader_election.cc:288] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 2 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:18.474277 23285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e4ffcf83c234ae7910ddef3ec0b4bda" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:18.475237 23285 raft_consensus.cc:3032] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 LEADER]: Stepping down as leader of term 1
I20260809 15:46:18.475564 23285 raft_consensus.cc:729] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Running, Role: LEADER
I20260809 15:46:18.476255 23400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e4ffcf83c234ae7910ddef3ec0b4bda" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:18.476809 23400 raft_consensus.cc:3037] T 8e4ffcf83c234ae7910ddef3ec0b4bda P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:18.476367 23285 consensus_queue.cc:260] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 67b3aeadfa7a432abe8af4d292e04ab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:18.478770 23285 raft_consensus.cc:3037] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:18.483351 23274 raft_consensus.cc:2454] T 998c6cee72cf40228c6dba064c0db2da P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 3.
I20260809 15:46:18.483388 24472 raft_consensus.cc:1261] T 16ca412a0dae4feebe85dd01bf1a9173 P 61e69695258f48baa984c9d395e8ce3f [term 1 LEARNER]: Refusing update from remote peer 67b3aeadfa7a432abe8af4d292e04ab9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 15:46:18.484336 23396 raft_consensus.cc:2454] T 998c6cee72cf40228c6dba064c0db2da P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 3.
I20260809 15:46:18.485513 23472 leader_election.cc:302] T 998c6cee72cf40228c6dba064c0db2da P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:46:18.485636 25060 consensus_queue.cc:1035] T 16ca412a0dae4feebe85dd01bf1a9173 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 15:46:18.498571 23522 raft_consensus.cc:1261] T 16ca412a0dae4feebe85dd01bf1a9173 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Refusing update from remote peer 67b3aeadfa7a432abe8af4d292e04ab9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 15:46:18.500016 23396 raft_consensus.cc:1261] T 16ca412a0dae4feebe85dd01bf1a9173 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Refusing update from remote peer 67b3aeadfa7a432abe8af4d292e04ab9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 15:46:18.501493 23117 catalog_manager.cc:5495] T 30c1740ef6874df0a332fbd6fd4b03ce P 67b3aeadfa7a432abe8af4d292e04ab9 reported cstate change: term changed from 1 to 2, leader changed from c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66) to 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65). New cstate: current_term: 2 leader_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } health_report { overall_health: UNKNOWN } } }
I20260809 15:46:18.486922 25151 raft_consensus.cc:2781] T 998c6cee72cf40228c6dba064c0db2da P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Leader election won for term 3
I20260809 15:46:18.501920 25201 consensus_queue.cc:1035] T 16ca412a0dae4feebe85dd01bf1a9173 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 15:46:18.505780 25151 raft_consensus.cc:686] T 998c6cee72cf40228c6dba064c0db2da P 65c36d164b0540f099d592f38307ee6c [term 3 LEADER]: Becoming Leader. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:46:18.517154 25060 consensus_queue.cc:1035] T 16ca412a0dae4feebe85dd01bf1a9173 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 15:46:18.513348 25151 consensus_queue.cc:237] T 998c6cee72cf40228c6dba064c0db2da P 65c36d164b0540f099d592f38307ee6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:18.486573 23400 raft_consensus.cc:2454] T 8e4ffcf83c234ae7910ddef3ec0b4bda P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 2.
I20260809 15:46:18.531046 24410 leader_election.cc:302] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:46:18.532280 25272 raft_consensus.cc:2781] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 61e69695258f48baa984c9d395e8ce3f [term 2 FOLLOWER]: Leader election won for term 2
I20260809 15:46:18.532928 25272 raft_consensus.cc:686] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 61e69695258f48baa984c9d395e8ce3f [term 2 LEADER]: Becoming Leader. State: Replica: 61e69695258f48baa984c9d395e8ce3f, State: Running, Role: LEADER
I20260809 15:46:18.533996 25272 consensus_queue.cc:237] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:18.538720 23117 catalog_manager.cc:5495] T 998c6cee72cf40228c6dba064c0db2da P 65c36d164b0540f099d592f38307ee6c reported cstate change: term changed from 2 to 3, leader changed from 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65) to 65c36d164b0540f099d592f38307ee6c (127.20.202.67). New cstate: current_term: 3 leader_uuid: "65c36d164b0540f099d592f38307ee6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } health_report { overall_health: HEALTHY } } }
I20260809 15:46:18.519181 23285 raft_consensus.cc:2454] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 2.
I20260809 15:46:18.578768 25060 raft_consensus.cc:2932] T 16ca412a0dae4feebe85dd01bf1a9173 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } attrs { replace: true } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:18.582278 23117 catalog_manager.cc:5495] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 61e69695258f48baa984c9d395e8ce3f reported cstate change: term changed from 1 to 2, leader changed from 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65) to 61e69695258f48baa984c9d395e8ce3f (127.20.202.68). New cstate: current_term: 2 leader_uuid: "61e69695258f48baa984c9d395e8ce3f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 15:46:18.586989 24472 raft_consensus.cc:2932] T 16ca412a0dae4feebe85dd01bf1a9173 P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } attrs { replace: true } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:18.603557 23116 catalog_manager.cc:5495] T 16ca412a0dae4feebe85dd01bf1a9173 P 67b3aeadfa7a432abe8af4d292e04ab9 reported cstate change: config changed from index 2 to 3, 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 15:46:18.606387 23522 raft_consensus.cc:2932] T 16ca412a0dae4feebe85dd01bf1a9173 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } attrs { replace: true } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:18.613521 23396 raft_consensus.cc:2932] T 16ca412a0dae4feebe85dd01bf1a9173 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } attrs { replace: true } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:18.629387 23274 consensus_queue.cc:237] T 16ca412a0dae4feebe85dd01bf1a9173 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:18.634091 25254 raft_consensus.cc:979] T 294d4bcc1e7a477cb56455581277a60f P 65c36d164b0540f099d592f38307ee6c: : Instructing follower 61e69695258f48baa984c9d395e8ce3f to start an election
I20260809 15:46:18.634518 25254 raft_consensus.cc:1067] T 294d4bcc1e7a477cb56455581277a60f P 65c36d164b0540f099d592f38307ee6c [term 1 LEADER]: Signalling peer 61e69695258f48baa984c9d395e8ce3f to start an election
I20260809 15:46:18.636130 24471 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "294d4bcc1e7a477cb56455581277a60f"
dest_uuid: "61e69695258f48baa984c9d395e8ce3f"
from {username='slave'} at 127.0.0.1:38734
I20260809 15:46:18.636698 24471 raft_consensus.cc:482] T 294d4bcc1e7a477cb56455581277a60f P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 15:46:18.636871 23519 raft_consensus.cc:1261] T 16ca412a0dae4feebe85dd01bf1a9173 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Refusing update from remote peer 67b3aeadfa7a432abe8af4d292e04ab9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260809 15:46:18.637073 24471 raft_consensus.cc:3037] T 294d4bcc1e7a477cb56455581277a60f P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:18.639389 24475 raft_consensus.cc:1261] T 16ca412a0dae4feebe85dd01bf1a9173 P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Refusing update from remote peer 67b3aeadfa7a432abe8af4d292e04ab9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260809 15:46:18.640089 25185 consensus_queue.cc:1035] T 16ca412a0dae4feebe85dd01bf1a9173 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.003s
I20260809 15:46:18.646461 25208 raft_consensus.cc:1050] T 472816994d814cb198b69e5248697440 P 65c36d164b0540f099d592f38307ee6c: attempting to promote NON_VOTER 61e69695258f48baa984c9d395e8ce3f to VOTER
I20260809 15:46:18.646692 25261 consensus_queue.cc:1035] T 16ca412a0dae4feebe85dd01bf1a9173 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260809 15:46:18.643707 24471 raft_consensus.cc:504] T 294d4bcc1e7a477cb56455581277a60f P 61e69695258f48baa984c9d395e8ce3f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:18.652436 25208 consensus_queue.cc:237] T 472816994d814cb198b69e5248697440 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:18.656042 24471 leader_election.cc:288] T 294d4bcc1e7a477cb56455581277a60f P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 2 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:18.661329 25261 raft_consensus.cc:2932] T 16ca412a0dae4feebe85dd01bf1a9173 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:18.666949 24471 raft_consensus.cc:1261] T 472816994d814cb198b69e5248697440 P 61e69695258f48baa984c9d395e8ce3f [term 1 LEARNER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 15:46:18.667904 23274 raft_consensus.cc:1261] T 472816994d814cb198b69e5248697440 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 15:46:18.668527 25208 consensus_queue.cc:1035] T 472816994d814cb198b69e5248697440 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 15:46:18.669508 25151 consensus_queue.cc:1035] T 472816994d814cb198b69e5248697440 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 15:46:18.673244 23274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "294d4bcc1e7a477cb56455581277a60f" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:18.673898 23274 raft_consensus.cc:3037] T 294d4bcc1e7a477cb56455581277a60f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:18.661283 23519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "294d4bcc1e7a477cb56455581277a60f" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:18.672480 24471 raft_consensus.cc:2932] T 16ca412a0dae4feebe85dd01bf1a9173 P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:18.676226 23519 raft_consensus.cc:3032] T 294d4bcc1e7a477cb56455581277a60f P 65c36d164b0540f099d592f38307ee6c [term 1 LEADER]: Stepping down as leader of term 1
I20260809 15:46:18.676678 23519 raft_consensus.cc:729] T 294d4bcc1e7a477cb56455581277a60f P 65c36d164b0540f099d592f38307ee6c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:46:18.677162 23396 raft_consensus.cc:1261] T 472816994d814cb198b69e5248697440 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 15:46:18.677429 23519 consensus_queue.cc:260] T 294d4bcc1e7a477cb56455581277a60f P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:18.678751 23519 raft_consensus.cc:3037] T 294d4bcc1e7a477cb56455581277a60f P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:18.679335 25208 consensus_queue.cc:1035] T 472816994d814cb198b69e5248697440 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 15:46:18.682070 23274 raft_consensus.cc:2454] T 294d4bcc1e7a477cb56455581277a60f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 2.
I20260809 15:46:18.687907 24406 leader_election.cc:302] T 294d4bcc1e7a477cb56455581277a60f P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:46:18.688887 25272 raft_consensus.cc:2781] T 294d4bcc1e7a477cb56455581277a60f P 61e69695258f48baa984c9d395e8ce3f [term 2 FOLLOWER]: Leader election won for term 2
I20260809 15:46:18.689360 25272 raft_consensus.cc:686] T 294d4bcc1e7a477cb56455581277a60f P 61e69695258f48baa984c9d395e8ce3f [term 2 LEADER]: Becoming Leader. State: Replica: 61e69695258f48baa984c9d395e8ce3f, State: Running, Role: LEADER
I20260809 15:46:18.690172 25272 consensus_queue.cc:237] T 294d4bcc1e7a477cb56455581277a60f P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:18.690398 23527 raft_consensus.cc:2932] T 16ca412a0dae4feebe85dd01bf1a9173 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:18.692301 23094 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 16ca412a0dae4feebe85dd01bf1a9173 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260809 15:46:18.686933 23519 raft_consensus.cc:2454] T 294d4bcc1e7a477cb56455581277a60f P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 2.
I20260809 15:46:18.703310 23115 catalog_manager.cc:5495] T 16ca412a0dae4feebe85dd01bf1a9173 P 67b3aeadfa7a432abe8af4d292e04ab9 reported cstate change: config changed from index 3 to 4, VOTER c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66) evicted. New cstate: current_term: 1 leader_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 15:46:18.710454 25151 raft_consensus.cc:2932] T 472816994d814cb198b69e5248697440 P 65c36d164b0540f099d592f38307ee6c [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:18.712553 23285 raft_consensus.cc:2932] T 472816994d814cb198b69e5248697440 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:18.737973 24475 raft_consensus.cc:2932] T 472816994d814cb198b69e5248697440 P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:18.744760 23396 raft_consensus.cc:2932] T 472816994d814cb198b69e5248697440 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:18.751654 25254 raft_consensus.cc:1050] T 7660dfa2ef694927b8eda4100a2bcd9b P 65c36d164b0540f099d592f38307ee6c: attempting to promote NON_VOTER 61e69695258f48baa984c9d395e8ce3f to VOTER
I20260809 15:46:18.755481 23389 tablet_service.cc:1514] Processing DeleteTablet for tablet 16ca412a0dae4feebe85dd01bf1a9173 with delete_type TABLET_DATA_TOMBSTONED (TS c0f38c1908414566aa9115bb2cbdd85d not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:55312
I20260809 15:46:18.777616 23116 catalog_manager.cc:5495] T 472816994d814cb198b69e5248697440 P 67b3aeadfa7a432abe8af4d292e04ab9 reported cstate change: config changed from index 2 to 3, 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "65c36d164b0540f099d592f38307ee6c" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:18.758734 25254 consensus_queue.cc:237] T 7660dfa2ef694927b8eda4100a2bcd9b P 65c36d164b0540f099d592f38307ee6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:18.829584 25335 tablet_replica.cc:330] T 16ca412a0dae4feebe85dd01bf1a9173 P c0f38c1908414566aa9115bb2cbdd85d: stopping tablet replica
I20260809 15:46:18.836665 23274 consensus_queue.cc:237] T 563daf42d7784db195ea99c3cb21b370 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:18.844626 24475 raft_consensus.cc:1261] T 563daf42d7784db195ea99c3cb21b370 P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Refusing update from remote peer 67b3aeadfa7a432abe8af4d292e04ab9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20260809 15:46:18.855093 23472 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 95262.4us
W20260809 15:46:18.862655 23470 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 73995.9us
I20260809 15:46:18.856654 25335 raft_consensus.cc:2229] T 16ca412a0dae4feebe85dd01bf1a9173 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 15:46:18.865353 23396 raft_consensus.cc:1261] T 563daf42d7784db195ea99c3cb21b370 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Refusing update from remote peer 67b3aeadfa7a432abe8af4d292e04ab9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260809 15:46:18.869342 25335 raft_consensus.cc:2258] T 16ca412a0dae4feebe85dd01bf1a9173 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 15:46:18.877450 23116 catalog_manager.cc:5495] T 294d4bcc1e7a477cb56455581277a60f P 61e69695258f48baa984c9d395e8ce3f reported cstate change: term changed from 1 to 2, leader changed from 65c36d164b0540f099d592f38307ee6c (127.20.202.67) to 61e69695258f48baa984c9d395e8ce3f (127.20.202.68). New cstate: current_term: 2 leader_uuid: "61e69695258f48baa984c9d395e8ce3f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 15:46:18.880707 25335 ts_tablet_manager.cc:1905] T 16ca412a0dae4feebe85dd01bf1a9173 P c0f38c1908414566aa9115bb2cbdd85d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260809 15:46:18.868176 25185 consensus_queue.cc:1035] T 563daf42d7784db195ea99c3cb21b370 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.013s
I20260809 15:46:18.889389 25261 consensus_queue.cc:1035] T 563daf42d7784db195ea99c3cb21b370 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260809 15:46:18.891258 24475 raft_consensus.cc:1261] T 7660dfa2ef694927b8eda4100a2bcd9b P 61e69695258f48baa984c9d395e8ce3f [term 1 LEARNER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 15:46:18.882679 23519 consensus_queue.cc:237] T 472816994d814cb198b69e5248697440 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:18.891887 25201 raft_consensus.cc:2932] T 563daf42d7784db195ea99c3cb21b370 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 65c36d164b0540f099d592f38307ee6c (127.20.202.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:18.892787 25151 consensus_queue.cc:1035] T 7660dfa2ef694927b8eda4100a2bcd9b P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 15:46:18.913522 24472 raft_consensus.cc:1261] T 472816994d814cb198b69e5248697440 P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260809 15:46:18.918212 23400 raft_consensus.cc:2932] T 563daf42d7784db195ea99c3cb21b370 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 65c36d164b0540f099d592f38307ee6c (127.20.202.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
W20260809 15:46:18.926335 23472 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 50435.9us
W20260809 15:46:18.928548 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 94815us
I20260809 15:46:18.929739 23396 raft_consensus.cc:1261] T 472816994d814cb198b69e5248697440 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260809 15:46:18.971129 23400 raft_consensus.cc:1261] T 8e4ffcf83c234ae7910ddef3ec0b4bda P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Refusing update from remote peer 61e69695258f48baa984c9d395e8ce3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260809 15:46:18.971648 23285 raft_consensus.cc:1261] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Refusing update from remote peer 61e69695258f48baa984c9d395e8ce3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260809 15:46:18.972829 25334 consensus_queue.cc:1035] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260809 15:46:18.974103 25334 consensus_queue.cc:1035] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260809 15:46:19.000113 25352 consensus_queue.cc:1035] T 472816994d814cb198b69e5248697440 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.009s
W20260809 15:46:19.008399 25305 log.cc:924] Time spent T 563daf42d7784db195ea99c3cb21b370 P 61e69695258f48baa984c9d395e8ce3f: Append to log took a long time: real 0.073s user 0.000s sys 0.000s
W20260809 15:46:19.010267 25343 log.cc:924] Time spent T 7660dfa2ef694927b8eda4100a2bcd9b P 65c36d164b0540f099d592f38307ee6c: Append to log took a long time: real 0.140s user 0.001s sys 0.000s
I20260809 15:46:19.026897 23522 raft_consensus.cc:1261] T 30c1740ef6874df0a332fbd6fd4b03ce P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Refusing update from remote peer 67b3aeadfa7a432abe8af4d292e04ab9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 15:46:19.056053 25119 consensus_queue.cc:1035] T 30c1740ef6874df0a332fbd6fd4b03ce P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 15:46:19.060709 24475 raft_consensus.cc:2932] T 563daf42d7784db195ea99c3cb21b370 P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 65c36d164b0540f099d592f38307ee6c (127.20.202.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:19.085685 25253 consensus_queue.cc:1035] T 472816994d814cb198b69e5248697440 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260809 15:46:19.108435 23402 raft_consensus.cc:1261] T 30c1740ef6874df0a332fbd6fd4b03ce P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Refusing update from remote peer 67b3aeadfa7a432abe8af4d292e04ab9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 15:46:19.123231 23283 raft_consensus.cc:1261] T 7660dfa2ef694927b8eda4100a2bcd9b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 15:46:19.134610 23116 catalog_manager.cc:5495] T 563daf42d7784db195ea99c3cb21b370 P c0f38c1908414566aa9115bb2cbdd85d reported cstate change: config changed from index 3 to 4, VOTER 65c36d164b0540f099d592f38307ee6c (127.20.202.67) evicted. New cstate: current_term: 1 leader_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
W20260809 15:46:19.143577 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 87811.8us
I20260809 15:46:19.219796 25274 raft_consensus.cc:979] T 00a1881267a14ba8b78c81040f17a014 P c0f38c1908414566aa9115bb2cbdd85d: : Instructing follower 67b3aeadfa7a432abe8af4d292e04ab9 to start an election
I20260809 15:46:19.220341 23282 raft_consensus.cc:1261] T 998c6cee72cf40228c6dba064c0db2da P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
W20260809 15:46:19.243247 25322 log.cc:924] Time spent T 8e4ffcf83c234ae7910ddef3ec0b4bda P 61e69695258f48baa984c9d395e8ce3f: Append to log took a long time: real 0.065s user 0.000s sys 0.000s
I20260809 15:46:19.249022 23400 raft_consensus.cc:1261] T 7660dfa2ef694927b8eda4100a2bcd9b P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 15:46:19.253726 25119 consensus_queue.cc:1035] T 30c1740ef6874df0a332fbd6fd4b03ce P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 15:46:19.262044 23282 raft_consensus.cc:1261] T 294d4bcc1e7a477cb56455581277a60f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Refusing update from remote peer 61e69695258f48baa984c9d395e8ce3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260809 15:46:19.265220 25335 ts_tablet_manager.cc:1918] T 16ca412a0dae4feebe85dd01bf1a9173 P c0f38c1908414566aa9115bb2cbdd85d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260809 15:46:19.265669 25335 log.cc:1196] T 16ca412a0dae4feebe85dd01bf1a9173 P c0f38c1908414566aa9115bb2cbdd85d: Deleting WAL directory at /tmp/dist-test-taskj2AJ_N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786290285951082-21289-0/minicluster-data/ts-1-root/wals/16ca412a0dae4feebe85dd01bf1a9173
I20260809 15:46:19.267467 23396 raft_consensus.cc:1261] T 998c6cee72cf40228c6dba064c0db2da P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260809 15:46:19.249686 25284 raft_consensus.cc:1067] T 00a1881267a14ba8b78c81040f17a014 P c0f38c1908414566aa9115bb2cbdd85d [term 1 LEADER]: Signalling peer 67b3aeadfa7a432abe8af4d292e04ab9 to start an election
W20260809 15:46:19.271611 23470 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 103712us
W20260809 15:46:19.321920 23468 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.68:34689, user_credentials={real_user=slave}} blocked reactor thread for 72676.8us
W20260809 15:46:19.344153 24406 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 45755.9us
I20260809 15:46:19.350045 23094 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 563daf42d7784db195ea99c3cb21b370 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260809 15:46:19.296680 25334 consensus_queue.cc:1035] T 294d4bcc1e7a477cb56455581277a60f P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.050s
W20260809 15:46:19.357685 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 109979us
I20260809 15:46:19.322180 25151 consensus_queue.cc:1035] T 7660dfa2ef694927b8eda4100a2bcd9b P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.033s
I20260809 15:46:19.368170 25350 consensus_queue.cc:1035] T 998c6cee72cf40228c6dba064c0db2da P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.011s
I20260809 15:46:19.368677 23095 catalog_manager.cc:4841] TS c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263): tablet 16ca412a0dae4feebe85dd01bf1a9173 (table test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]) successfully deleted
W20260809 15:46:19.411813 23470 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 67131.7us
I20260809 15:46:19.427170 25366 raft_consensus.cc:2932] T 472816994d814cb198b69e5248697440 P 65c36d164b0540f099d592f38307ee6c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:19.459739 25253 consensus_queue.cc:1035] T 998c6cee72cf40228c6dba064c0db2da P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.018s
I20260809 15:46:19.460947 25337 consensus_queue.cc:1035] T 7660dfa2ef694927b8eda4100a2bcd9b P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.035s
I20260809 15:46:19.479229 23400 raft_consensus.cc:2932] T 472816994d814cb198b69e5248697440 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:19.473661 21289 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20260809 15:46:19.487438 24410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 143843us
W20260809 15:46:19.503371 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 143671us
I20260809 15:46:19.519021 23285 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00a1881267a14ba8b78c81040f17a014"
dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
from {username='slave'} at 127.0.0.1:53562
I20260809 15:46:19.519558 23285 raft_consensus.cc:482] T 00a1881267a14ba8b78c81040f17a014 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 15:46:19.519886 23285 raft_consensus.cc:3037] T 00a1881267a14ba8b78c81040f17a014 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:19.530171 24472 raft_consensus.cc:2932] T 472816994d814cb198b69e5248697440 P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:19.566358 23500 tablet_service.cc:1514] Processing DeleteTablet for tablet 563daf42d7784db195ea99c3cb21b370 with delete_type TABLET_DATA_TOMBSTONED (TS 65c36d164b0540f099d592f38307ee6c not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:56760
I20260809 15:46:19.574469 23522 raft_consensus.cc:1261] T 294d4bcc1e7a477cb56455581277a60f P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Refusing update from remote peer 61e69695258f48baa984c9d395e8ce3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
W20260809 15:46:19.597263 23353 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 137435us
I20260809 15:46:19.628867 25396 consensus_queue.cc:1035] T 294d4bcc1e7a477cb56455581277a60f P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.048s
W20260809 15:46:19.689934 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 104510us
I20260809 15:46:19.690300 23285 raft_consensus.cc:504] T 00a1881267a14ba8b78c81040f17a014 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:19.692562 23280 raft_consensus.cc:1226] T 00a1881267a14ba8b78c81040f17a014 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Rejecting Update request from peer c0f38c1908414566aa9115bb2cbdd85d for earlier term 1. Current term is 2. Ops: []
I20260809 15:46:19.694772 23285 leader_election.cc:288] T 00a1881267a14ba8b78c81040f17a014 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 2 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:19.723672 23395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00a1881267a14ba8b78c81040f17a014" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:19.724285 23395 raft_consensus.cc:3032] T 00a1881267a14ba8b78c81040f17a014 P c0f38c1908414566aa9115bb2cbdd85d [term 1 LEADER]: Stepping down as leader of term 1
I20260809 15:46:19.724633 23395 raft_consensus.cc:729] T 00a1881267a14ba8b78c81040f17a014 P c0f38c1908414566aa9115bb2cbdd85d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:46:19.720222 23115 catalog_manager.cc:5495] T 472816994d814cb198b69e5248697440 P 61e69695258f48baa984c9d395e8ce3f reported cstate change: config changed from index 3 to 4, VOTER 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65) evicted. New cstate: current_term: 1 leader_uuid: "65c36d164b0540f099d592f38307ee6c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:19.725533 23395 consensus_queue.cc:260] T 00a1881267a14ba8b78c81040f17a014 P c0f38c1908414566aa9115bb2cbdd85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:19.740067 23395 raft_consensus.cc:3037] T 00a1881267a14ba8b78c81040f17a014 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:19.768291 23095 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 472816994d814cb198b69e5248697440 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260809 15:46:19.769217 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00a1881267a14ba8b78c81040f17a014" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:19.769831 23526 raft_consensus.cc:3037] T 00a1881267a14ba8b78c81040f17a014 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:19.777352 23526 raft_consensus.cc:2454] T 00a1881267a14ba8b78c81040f17a014 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 2.
W20260809 15:46:19.780676 23223 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.68:34689, user_credentials={real_user=slave}} blocked reactor thread for 74308.2us
I20260809 15:46:19.793407 23395 raft_consensus.cc:2454] T 00a1881267a14ba8b78c81040f17a014 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 2.
I20260809 15:46:19.822336 23260 tablet_service.cc:1514] Processing DeleteTablet for tablet 472816994d814cb198b69e5248697440 with delete_type TABLET_DATA_TOMBSTONED (TS 67b3aeadfa7a432abe8af4d292e04ab9 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:53552
I20260809 15:46:19.824358 25416 tablet_replica.cc:330] T 563daf42d7784db195ea99c3cb21b370 P 65c36d164b0540f099d592f38307ee6c: stopping tablet replica
I20260809 15:46:19.825357 25416 raft_consensus.cc:2229] T 563daf42d7784db195ea99c3cb21b370 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 15:46:19.841428 23228 leader_election.cc:302] T 00a1881267a14ba8b78c81040f17a014 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
W20260809 15:46:19.890785 23472 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 46232.9us
W20260809 15:46:19.891880 23468 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.68:34689, user_credentials={real_user=slave}} blocked reactor thread for 49065.8us
W20260809 15:46:19.898229 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 56882.7us
I20260809 15:46:19.903587 25261 raft_consensus.cc:2781] T 00a1881267a14ba8b78c81040f17a014 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 15:46:19.910373 25416 raft_consensus.cc:2258] T 563daf42d7784db195ea99c3cb21b370 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 15:46:19.904314 23280 raft_consensus.cc:2932] T 7660dfa2ef694927b8eda4100a2bcd9b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:19.825155 25151 raft_consensus.cc:2932] T 7660dfa2ef694927b8eda4100a2bcd9b P 65c36d164b0540f099d592f38307ee6c [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:19.914806 23395 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "297c82ed906d4b48b184985eed3de601"
dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
mode: GRACEFUL
new_leader_uuid: "65c36d164b0540f099d592f38307ee6c"
from {username='slave'} at 127.0.0.1:51280
I20260809 15:46:19.931680 25423 tablet_replica.cc:330] T 472816994d814cb198b69e5248697440 P 67b3aeadfa7a432abe8af4d292e04ab9: stopping tablet replica
I20260809 15:46:19.933228 25423 raft_consensus.cc:2229] T 472816994d814cb198b69e5248697440 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 15:46:19.932080 23395 raft_consensus.cc:595] T 297c82ed906d4b48b184985eed3de601 P c0f38c1908414566aa9115bb2cbdd85d [term 1 LEADER]: Received request to transfer leadership to TS 65c36d164b0540f099d592f38307ee6c
I20260809 15:46:19.934615 25423 raft_consensus.cc:2258] T 472816994d814cb198b69e5248697440 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 15:46:19.927624 23394 raft_consensus.cc:2932] T 7660dfa2ef694927b8eda4100a2bcd9b P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:19.922802 25261 raft_consensus.cc:686] T 00a1881267a14ba8b78c81040f17a014 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 LEADER]: Becoming Leader. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Running, Role: LEADER
I20260809 15:46:19.937304 25423 ts_tablet_manager.cc:1905] T 472816994d814cb198b69e5248697440 P 67b3aeadfa7a432abe8af4d292e04ab9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260809 15:46:19.937049 25261 consensus_queue.cc:237] T 00a1881267a14ba8b78c81040f17a014 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:19.943132 25416 ts_tablet_manager.cc:1905] T 563daf42d7784db195ea99c3cb21b370 P 65c36d164b0540f099d592f38307ee6c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260809 15:46:19.958690 25358 raft_consensus.cc:979] T 297c82ed906d4b48b184985eed3de601 P c0f38c1908414566aa9115bb2cbdd85d: : Instructing follower 65c36d164b0540f099d592f38307ee6c to start an election
I20260809 15:46:19.959122 25358 raft_consensus.cc:1067] T 297c82ed906d4b48b184985eed3de601 P c0f38c1908414566aa9115bb2cbdd85d [term 1 LEADER]: Signalling peer 65c36d164b0540f099d592f38307ee6c to start an election
I20260809 15:46:19.967293 23394 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3924b648461e463b9d7925e5671d68ae"
dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
mode: GRACEFUL
new_leader_uuid: "61e69695258f48baa984c9d395e8ce3f"
from {username='slave'} at 127.0.0.1:51280
I20260809 15:46:19.968009 23394 raft_consensus.cc:595] T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d [term 1 LEADER]: Received request to transfer leadership to TS 61e69695258f48baa984c9d395e8ce3f
W20260809 15:46:19.974176 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 54712.8us
I20260809 15:46:19.974570 23517 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "297c82ed906d4b48b184985eed3de601"
dest_uuid: "65c36d164b0540f099d592f38307ee6c"
from {username='slave'} at 127.0.0.1:56766
I20260809 15:46:19.975466 23517 raft_consensus.cc:482] T 297c82ed906d4b48b184985eed3de601 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 15:46:19.975821 23517 raft_consensus.cc:3037] T 297c82ed906d4b48b184985eed3de601 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:19.996620 25347 raft_consensus.cc:979] T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d: : Instructing follower 61e69695258f48baa984c9d395e8ce3f to start an election
I20260809 15:46:19.997068 25347 raft_consensus.cc:1067] T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d [term 1 LEADER]: Signalling peer 61e69695258f48baa984c9d395e8ce3f to start an election
I20260809 15:46:19.998967 24471 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3924b648461e463b9d7925e5671d68ae"
dest_uuid: "61e69695258f48baa984c9d395e8ce3f"
from {username='slave'} at 127.0.0.1:38724
I20260809 15:46:20.001617 23527 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "73f6f44c259547f1b05a3a434616fa54"
dest_uuid: "65c36d164b0540f099d592f38307ee6c"
mode: GRACEFUL
new_leader_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
from {username='slave'} at 127.0.0.1:48888
I20260809 15:46:20.001995 24471 raft_consensus.cc:482] T 3924b648461e463b9d7925e5671d68ae P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 15:46:20.002147 23527 raft_consensus.cc:595] T 73f6f44c259547f1b05a3a434616fa54 P 65c36d164b0540f099d592f38307ee6c [term 1 LEADER]: Received request to transfer leadership to TS 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:20.002336 24471 raft_consensus.cc:3037] T 3924b648461e463b9d7925e5671d68ae P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:20.003463 24475 raft_consensus.cc:2932] T 7660dfa2ef694927b8eda4100a2bcd9b P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:20.008490 23283 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "907bacfa79d04a55a68c8dbe60056462"
dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
mode: GRACEFUL
new_leader_uuid: "61e69695258f48baa984c9d395e8ce3f"
from {username='slave'} at 127.0.0.1:33468
I20260809 15:46:20.009181 23283 raft_consensus.cc:595] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 LEADER]: Received request to transfer leadership to TS 61e69695258f48baa984c9d395e8ce3f
I20260809 15:46:20.021793 24471 raft_consensus.cc:504] T 3924b648461e463b9d7925e5671d68ae P 61e69695258f48baa984c9d395e8ce3f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:20.027629 23400 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d3e94b7e50754a81b3d72a07476cb7b9"
dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
mode: GRACEFUL
new_leader_uuid: "61e69695258f48baa984c9d395e8ce3f"
from {username='slave'} at 127.0.0.1:51280
I20260809 15:46:20.028156 23400 raft_consensus.cc:595] T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d [term 1 LEADER]: Received request to transfer leadership to TS 61e69695258f48baa984c9d395e8ce3f
I20260809 15:46:20.034519 24471 leader_election.cc:288] T 3924b648461e463b9d7925e5671d68ae P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 2 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:20.031424 23117 catalog_manager.cc:5495] T 7660dfa2ef694927b8eda4100a2bcd9b P c0f38c1908414566aa9115bb2cbdd85d reported cstate change: config changed from index 2 to 3, 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "65c36d164b0540f099d592f38307ee6c" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:20.038035 21289 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260809 15:46:20.038352 23400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3924b648461e463b9d7925e5671d68ae" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:20.038856 23400 raft_consensus.cc:3032] T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d [term 1 LEADER]: Stepping down as leader of term 1
I20260809 15:46:20.039136 23400 raft_consensus.cc:729] T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:46:20.039454 21289 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 15:46:20.040048 21289 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 15:46:20.039897 23400 consensus_queue.cc:260] T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:20.040967 23400 raft_consensus.cc:3037] T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:20.050297 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3924b648461e463b9d7925e5671d68ae" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:20.050817 23526 raft_consensus.cc:3037] T 3924b648461e463b9d7925e5671d68ae P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:20.056952 23400 raft_consensus.cc:2454] T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 2.
I20260809 15:46:20.064424 23526 raft_consensus.cc:2454] T 3924b648461e463b9d7925e5671d68ae P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 2.
I20260809 15:46:20.063531 24410 leader_election.cc:302] T 3924b648461e463b9d7925e5671d68ae P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:46:20.067179 25338 raft_consensus.cc:2781] T 3924b648461e463b9d7925e5671d68ae P 61e69695258f48baa984c9d395e8ce3f [term 2 FOLLOWER]: Leader election won for term 2
I20260809 15:46:20.067801 25338 raft_consensus.cc:686] T 3924b648461e463b9d7925e5671d68ae P 61e69695258f48baa984c9d395e8ce3f [term 2 LEADER]: Becoming Leader. State: Replica: 61e69695258f48baa984c9d395e8ce3f, State: Running, Role: LEADER
I20260809 15:46:20.069060 25338 consensus_queue.cc:237] T 3924b648461e463b9d7925e5671d68ae P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:20.085023 23115 catalog_manager.cc:5495] T 00a1881267a14ba8b78c81040f17a014 P 67b3aeadfa7a432abe8af4d292e04ab9 reported cstate change: term changed from 1 to 2, leader changed from c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66) to 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65). New cstate: current_term: 2 leader_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } health_report { overall_health: UNKNOWN } } }
I20260809 15:46:20.087812 23517 raft_consensus.cc:504] T 297c82ed906d4b48b184985eed3de601 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:20.090013 25423 ts_tablet_manager.cc:1918] T 472816994d814cb198b69e5248697440 P 67b3aeadfa7a432abe8af4d292e04ab9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260809 15:46:20.096504 25423 log.cc:1196] T 472816994d814cb198b69e5248697440 P 67b3aeadfa7a432abe8af4d292e04ab9: Deleting WAL directory at /tmp/dist-test-taskj2AJ_N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786290285951082-21289-0/minicluster-data/ts-0-root/wals/472816994d814cb198b69e5248697440
I20260809 15:46:20.098191 23280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "297c82ed906d4b48b184985eed3de601" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:20.099076 23280 raft_consensus.cc:3037] T 297c82ed906d4b48b184985eed3de601 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:20.099411 23094 catalog_manager.cc:4841] TS 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459): tablet 472816994d814cb198b69e5248697440 (table test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]) successfully deleted
I20260809 15:46:20.096856 23517 leader_election.cc:288] T 297c82ed906d4b48b184985eed3de601 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 2 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:20.103747 23400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "297c82ed906d4b48b184985eed3de601" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:20.104322 23400 raft_consensus.cc:3032] T 297c82ed906d4b48b184985eed3de601 P c0f38c1908414566aa9115bb2cbdd85d [term 1 LEADER]: Stepping down as leader of term 1
I20260809 15:46:20.104643 23400 raft_consensus.cc:729] T 297c82ed906d4b48b184985eed3de601 P c0f38c1908414566aa9115bb2cbdd85d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:46:20.105480 23400 consensus_queue.cc:260] T 297c82ed906d4b48b184985eed3de601 P c0f38c1908414566aa9115bb2cbdd85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:20.106662 23400 raft_consensus.cc:3037] T 297c82ed906d4b48b184985eed3de601 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:20.112022 23116 catalog_manager.cc:5495] T 3924b648461e463b9d7925e5671d68ae P 61e69695258f48baa984c9d395e8ce3f reported cstate change: term changed from 1 to 2, leader changed from c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66) to 61e69695258f48baa984c9d395e8ce3f (127.20.202.68). New cstate: current_term: 2 leader_uuid: "61e69695258f48baa984c9d395e8ce3f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 15:46:20.116099 25416 ts_tablet_manager.cc:1918] T 563daf42d7784db195ea99c3cb21b370 P 65c36d164b0540f099d592f38307ee6c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260809 15:46:20.116742 25416 log.cc:1196] T 563daf42d7784db195ea99c3cb21b370 P 65c36d164b0540f099d592f38307ee6c: Deleting WAL directory at /tmp/dist-test-taskj2AJ_N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786290285951082-21289-0/minicluster-data/ts-2-root/wals/563daf42d7784db195ea99c3cb21b370
I20260809 15:46:20.118492 23527 consensus_queue.cc:237] T 7660dfa2ef694927b8eda4100a2bcd9b P 65c36d164b0540f099d592f38307ee6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:20.118999 23400 raft_consensus.cc:2454] T 297c82ed906d4b48b184985eed3de601 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 2.
I20260809 15:46:20.120185 23472 leader_election.cc:302] T 297c82ed906d4b48b184985eed3de601 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:46:20.121083 25366 raft_consensus.cc:2781] T 297c82ed906d4b48b184985eed3de601 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Leader election won for term 2
I20260809 15:46:20.121636 25366 raft_consensus.cc:686] T 297c82ed906d4b48b184985eed3de601 P 65c36d164b0540f099d592f38307ee6c [term 2 LEADER]: Becoming Leader. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:46:20.122781 25366 consensus_queue.cc:237] T 297c82ed906d4b48b184985eed3de601 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:20.135322 23395 raft_consensus.cc:1261] T 7660dfa2ef694927b8eda4100a2bcd9b P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260809 15:46:20.135986 23280 raft_consensus.cc:2454] T 297c82ed906d4b48b184985eed3de601 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 2.
I20260809 15:46:20.137545 25369 consensus_queue.cc:1035] T 7660dfa2ef694927b8eda4100a2bcd9b P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260809 15:46:20.149214 25253 raft_consensus.cc:2932] T 7660dfa2ef694927b8eda4100a2bcd9b P 65c36d164b0540f099d592f38307ee6c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:20.151225 23395 raft_consensus.cc:2932] T 7660dfa2ef694927b8eda4100a2bcd9b P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:20.158109 23095 catalog_manager.cc:4841] TS 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553): tablet 563daf42d7784db195ea99c3cb21b370 (table test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]) successfully deleted
I20260809 15:46:20.159310 24471 raft_consensus.cc:1261] T 7660dfa2ef694927b8eda4100a2bcd9b P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260809 15:46:20.166368 25369 consensus_queue.cc:1035] T 7660dfa2ef694927b8eda4100a2bcd9b P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260809 15:46:20.176226 23116 catalog_manager.cc:5495] T 297c82ed906d4b48b184985eed3de601 P 65c36d164b0540f099d592f38307ee6c reported cstate change: term changed from 1 to 2, leader changed from c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66) to 65c36d164b0540f099d592f38307ee6c (127.20.202.67). New cstate: current_term: 2 leader_uuid: "65c36d164b0540f099d592f38307ee6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } health_report { overall_health: HEALTHY } } }
I20260809 15:46:20.180171 24471 raft_consensus.cc:2932] T 7660dfa2ef694927b8eda4100a2bcd9b P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:20.182693 23095 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 7660dfa2ef694927b8eda4100a2bcd9b with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260809 15:46:20.189112 23118 catalog_manager.cc:5495] T 7660dfa2ef694927b8eda4100a2bcd9b P c0f38c1908414566aa9115bb2cbdd85d reported cstate change: config changed from index 3 to 4, VOTER 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65) evicted. New cstate: current_term: 1 leader_uuid: "65c36d164b0540f099d592f38307ee6c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:20.201761 23260 tablet_service.cc:1514] Processing DeleteTablet for tablet 7660dfa2ef694927b8eda4100a2bcd9b with delete_type TABLET_DATA_TOMBSTONED (TS 67b3aeadfa7a432abe8af4d292e04ab9 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:53552
I20260809 15:46:20.202983 25423 tablet_replica.cc:330] T 7660dfa2ef694927b8eda4100a2bcd9b P 67b3aeadfa7a432abe8af4d292e04ab9: stopping tablet replica
I20260809 15:46:20.204526 25423 raft_consensus.cc:2229] T 7660dfa2ef694927b8eda4100a2bcd9b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 15:46:20.205477 25423 raft_consensus.cc:2258] T 7660dfa2ef694927b8eda4100a2bcd9b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 15:46:20.209728 25423 ts_tablet_manager.cc:1905] T 7660dfa2ef694927b8eda4100a2bcd9b P 67b3aeadfa7a432abe8af4d292e04ab9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260809 15:46:20.234596 25423 ts_tablet_manager.cc:1918] T 7660dfa2ef694927b8eda4100a2bcd9b P 67b3aeadfa7a432abe8af4d292e04ab9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260809 15:46:20.235016 25423 log.cc:1196] T 7660dfa2ef694927b8eda4100a2bcd9b P 67b3aeadfa7a432abe8af4d292e04ab9: Deleting WAL directory at /tmp/dist-test-taskj2AJ_N/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786290285951082-21289-0/minicluster-data/ts-0-root/wals/7660dfa2ef694927b8eda4100a2bcd9b
I20260809 15:46:20.236393 23094 catalog_manager.cc:4841] TS 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459): tablet 7660dfa2ef694927b8eda4100a2bcd9b (table test-workload [id=3d330bbcf9604c31b27b67ae5af9032e]) successfully deleted
I20260809 15:46:20.282115 25358 raft_consensus.cc:979] T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d: : Instructing follower 61e69695258f48baa984c9d395e8ce3f to start an election
I20260809 15:46:20.282580 25358 raft_consensus.cc:1067] T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d [term 1 LEADER]: Signalling peer 61e69695258f48baa984c9d395e8ce3f to start an election
I20260809 15:46:20.283812 24471 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d3e94b7e50754a81b3d72a07476cb7b9"
dest_uuid: "61e69695258f48baa984c9d395e8ce3f"
from {username='slave'} at 127.0.0.1:38724
I20260809 15:46:20.284313 24471 raft_consensus.cc:482] T d3e94b7e50754a81b3d72a07476cb7b9 P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 15:46:20.284622 24471 raft_consensus.cc:3037] T d3e94b7e50754a81b3d72a07476cb7b9 P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:20.289103 24471 raft_consensus.cc:504] T d3e94b7e50754a81b3d72a07476cb7b9 P 61e69695258f48baa984c9d395e8ce3f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:20.290454 24471 leader_election.cc:288] T d3e94b7e50754a81b3d72a07476cb7b9 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 2 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:20.291604 23517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3e94b7e50754a81b3d72a07476cb7b9" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:20.291671 23395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3e94b7e50754a81b3d72a07476cb7b9" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:20.292104 23517 raft_consensus.cc:3037] T d3e94b7e50754a81b3d72a07476cb7b9 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:20.292320 23395 raft_consensus.cc:3032] T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d [term 1 LEADER]: Stepping down as leader of term 1
I20260809 15:46:20.292649 23395 raft_consensus.cc:729] T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:46:20.293401 23395 consensus_queue.cc:260] T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:20.294337 23395 raft_consensus.cc:3037] T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:20.296729 23517 raft_consensus.cc:2454] T d3e94b7e50754a81b3d72a07476cb7b9 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 2.
I20260809 15:46:20.297662 24410 leader_election.cc:302] T d3e94b7e50754a81b3d72a07476cb7b9 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c; no voters:
I20260809 15:46:20.298487 25338 raft_consensus.cc:2781] T d3e94b7e50754a81b3d72a07476cb7b9 P 61e69695258f48baa984c9d395e8ce3f [term 2 FOLLOWER]: Leader election won for term 2
I20260809 15:46:20.298888 25338 raft_consensus.cc:686] T d3e94b7e50754a81b3d72a07476cb7b9 P 61e69695258f48baa984c9d395e8ce3f [term 2 LEADER]: Becoming Leader. State: Replica: 61e69695258f48baa984c9d395e8ce3f, State: Running, Role: LEADER
I20260809 15:46:20.299791 25338 consensus_queue.cc:237] T d3e94b7e50754a81b3d72a07476cb7b9 P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:20.300655 23395 raft_consensus.cc:2454] T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 2.
I20260809 15:46:20.308218 23118 catalog_manager.cc:5495] T d3e94b7e50754a81b3d72a07476cb7b9 P 61e69695258f48baa984c9d395e8ce3f reported cstate change: term changed from 1 to 2, leader changed from c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66) to 61e69695258f48baa984c9d395e8ce3f (127.20.202.68). New cstate: current_term: 2 leader_uuid: "61e69695258f48baa984c9d395e8ce3f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 15:46:20.358973 23400 raft_consensus.cc:1261] T 00a1881267a14ba8b78c81040f17a014 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Refusing update from remote peer 67b3aeadfa7a432abe8af4d292e04ab9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 15:46:20.359922 25425 consensus_queue.cc:1035] T 00a1881267a14ba8b78c81040f17a014 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 15:46:20.370155 23517 raft_consensus.cc:1261] T 00a1881267a14ba8b78c81040f17a014 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Refusing update from remote peer 67b3aeadfa7a432abe8af4d292e04ab9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 15:46:20.372840 25419 consensus_queue.cc:1035] T 00a1881267a14ba8b78c81040f17a014 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 15:46:20.439567 25419 raft_consensus.cc:979] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9: : Instructing follower 61e69695258f48baa984c9d395e8ce3f to start an election
I20260809 15:46:20.439960 25419 raft_consensus.cc:1067] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 LEADER]: Signalling peer 61e69695258f48baa984c9d395e8ce3f to start an election
I20260809 15:46:20.441210 24471 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "907bacfa79d04a55a68c8dbe60056462"
dest_uuid: "61e69695258f48baa984c9d395e8ce3f"
from {username='slave'} at 127.0.0.1:38722
I20260809 15:46:20.441630 24471 raft_consensus.cc:482] T 907bacfa79d04a55a68c8dbe60056462 P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 15:46:20.441864 24471 raft_consensus.cc:3037] T 907bacfa79d04a55a68c8dbe60056462 P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:20.445902 24471 raft_consensus.cc:504] T 907bacfa79d04a55a68c8dbe60056462 P 61e69695258f48baa984c9d395e8ce3f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:20.447698 24471 leader_election.cc:288] T 907bacfa79d04a55a68c8dbe60056462 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 2 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:20.448259 23280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "907bacfa79d04a55a68c8dbe60056462" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:20.448784 23280 raft_consensus.cc:3032] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 LEADER]: Stepping down as leader of term 1
I20260809 15:46:20.448734 23395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "907bacfa79d04a55a68c8dbe60056462" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:20.449134 23280 raft_consensus.cc:729] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Running, Role: LEADER
I20260809 15:46:20.449295 23395 raft_consensus.cc:3037] T 907bacfa79d04a55a68c8dbe60056462 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:20.449779 23280 consensus_queue.cc:260] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:20.450687 23280 raft_consensus.cc:3037] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:20.453598 23395 raft_consensus.cc:2454] T 907bacfa79d04a55a68c8dbe60056462 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 2.
I20260809 15:46:20.454530 24410 leader_election.cc:302] T 907bacfa79d04a55a68c8dbe60056462 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:46:20.454902 23280 raft_consensus.cc:2454] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 2.
I20260809 15:46:20.455399 25338 raft_consensus.cc:2781] T 907bacfa79d04a55a68c8dbe60056462 P 61e69695258f48baa984c9d395e8ce3f [term 2 FOLLOWER]: Leader election won for term 2
I20260809 15:46:20.455883 25338 raft_consensus.cc:686] T 907bacfa79d04a55a68c8dbe60056462 P 61e69695258f48baa984c9d395e8ce3f [term 2 LEADER]: Becoming Leader. State: Replica: 61e69695258f48baa984c9d395e8ce3f, State: Running, Role: LEADER
I20260809 15:46:20.456627 25338 consensus_queue.cc:237] T 907bacfa79d04a55a68c8dbe60056462 P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:20.462953 23117 catalog_manager.cc:5495] T 907bacfa79d04a55a68c8dbe60056462 P 61e69695258f48baa984c9d395e8ce3f reported cstate change: term changed from 1 to 2, leader changed from 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65) to 61e69695258f48baa984c9d395e8ce3f (127.20.202.68). New cstate: current_term: 2 leader_uuid: "61e69695258f48baa984c9d395e8ce3f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 15:46:20.479663 23395 raft_consensus.cc:1261] T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Refusing update from remote peer 61e69695258f48baa984c9d395e8ce3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260809 15:46:20.480670 25338 consensus_queue.cc:1035] T 3924b648461e463b9d7925e5671d68ae P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260809 15:46:20.489714 23517 raft_consensus.cc:1261] T 3924b648461e463b9d7925e5671d68ae P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Refusing update from remote peer 61e69695258f48baa984c9d395e8ce3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260809 15:46:20.491070 25338 consensus_queue.cc:1035] T 3924b648461e463b9d7925e5671d68ae P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260809 15:46:21.535977 25151 raft_consensus.cc:482] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:46:21.536459 25151 raft_consensus.cc:504] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:21.538655 25151 leader_election.cc:288] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:21.636962 25151 raft_consensus.cc:482] T 2424fd9bd7f14e508fd106cbfda786a4 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
I20260809 15:46:21.637599 25151 raft_consensus.cc:504] T 2424fd9bd7f14e508fd106cbfda786a4 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:21.639976 25151 leader_election.cc:288] T 2424fd9bd7f14e508fd106cbfda786a4 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:21.669462 25253 raft_consensus.cc:482] T 28e17dcb5efe45b8b6a8c42da03b4819 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:46:21.669988 25253 raft_consensus.cc:504] T 28e17dcb5efe45b8b6a8c42da03b4819 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:21.671867 25253 leader_election.cc:288] T 28e17dcb5efe45b8b6a8c42da03b4819 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:21.672313 25253 raft_consensus.cc:482] T 4b30f28311964a0b9149a50495e83d8f P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:46:21.672717 25253 raft_consensus.cc:504] T 4b30f28311964a0b9149a50495e83d8f P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:21.674484 25253 leader_election.cc:288] T 4b30f28311964a0b9149a50495e83d8f P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:21.698179 25151 raft_consensus.cc:482] T 4bcac4b24e7a4448abdc934bba5a624e P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:46:21.698689 25151 raft_consensus.cc:504] T 4bcac4b24e7a4448abdc934bba5a624e P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:21.700691 25253 raft_consensus.cc:482] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:46:21.701225 25253 raft_consensus.cc:504] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:21.703521 25253 leader_election.cc:288] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:21.745231 25151 leader_election.cc:288] T 4bcac4b24e7a4448abdc934bba5a624e P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:21.779688 25151 raft_consensus.cc:482] T 16ca412a0dae4feebe85dd01bf1a9173 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
I20260809 15:46:21.780181 25151 raft_consensus.cc:504] T 16ca412a0dae4feebe85dd01bf1a9173 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:21.787514 25151 leader_election.cc:288] T 16ca412a0dae4feebe85dd01bf1a9173 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:21.877525 25151 raft_consensus.cc:482] T 00a1881267a14ba8b78c81040f17a014 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
I20260809 15:46:21.878079 25151 raft_consensus.cc:504] T 00a1881267a14ba8b78c81040f17a014 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:21.880327 25151 leader_election.cc:288] T 00a1881267a14ba8b78c81040f17a014 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:22.024384 25253 raft_consensus.cc:482] T d1a134ba33ea4076bbaac0ae460c17ba P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:46:22.024964 25253 raft_consensus.cc:504] T d1a134ba33ea4076bbaac0ae460c17ba P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:22.027000 25253 leader_election.cc:288] T d1a134ba33ea4076bbaac0ae460c17ba P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:22.028107 25253 raft_consensus.cc:482] T d0721a4403a440f383318a3cea9d1442 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:46:22.028627 25253 raft_consensus.cc:504] T d0721a4403a440f383318a3cea9d1442 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:22.030747 25253 leader_election.cc:288] T d0721a4403a440f383318a3cea9d1442 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:22.091106 21289 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 15:46:22.100073 25253 raft_consensus.cc:482] T 0c29937a80724ddb9d07a27534a146f3 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:46:22.100590 25253 raft_consensus.cc:504] T 0c29937a80724ddb9d07a27534a146f3 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:22.102855 25253 leader_election.cc:288] T 0c29937a80724ddb9d07a27534a146f3 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:22.103395 25253 raft_consensus.cc:482] T 4564dfc6d4094c1b9240b367ccecf3b1 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:46:22.279364 25253 raft_consensus.cc:504] T 4564dfc6d4094c1b9240b367ccecf3b1 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:22.282107 25253 leader_election.cc:288] T 4564dfc6d4094c1b9240b367ccecf3b1 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:22.287025 25457 raft_consensus.cc:482] T 472816994d814cb198b69e5248697440 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:22.287652 25457 raft_consensus.cc:504] T 472816994d814cb198b69e5248697440 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:22.290027 25457 leader_election.cc:288] T 472816994d814cb198b69e5248697440 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
W20260809 15:46:22.254379 24410 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.135s user 0.007s sys 0.031s
I20260809 15:46:22.321818 25151 raft_consensus.cc:482] T 538a228ddfb54bc6a36528d5ff07597b P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
I20260809 15:46:22.379841 25151 raft_consensus.cc:504] T 538a228ddfb54bc6a36528d5ff07597b P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:22.322266 25366 raft_consensus.cc:482] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
W20260809 15:46:22.379755 24410 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.261s user 0.007s sys 0.031s
I20260809 15:46:22.336135 25458 raft_consensus.cc:482] T 2424fd9bd7f14e508fd106cbfda786a4 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
I20260809 15:46:22.344952 25462 raft_consensus.cc:482] T 6ae14a74a26f434db08c652e097385c8 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
I20260809 15:46:22.347429 25461 raft_consensus.cc:482] T 4564dfc6d4094c1b9240b367ccecf3b1 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:46:22.363030 25253 raft_consensus.cc:482] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:46:22.365850 25453 raft_consensus.cc:482] T cee02fff2c9c48a19f43eb358a89e7b0 P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:22.335099 23526 raft_consensus.cc:1261] T d3e94b7e50754a81b3d72a07476cb7b9 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Refusing update from remote peer 61e69695258f48baa984c9d395e8ce3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260809 15:46:22.388091 23526 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44018 (request call id 44) took 1641 ms. Trace:
I20260809 15:46:22.385445 25151 leader_election.cc:288] T 538a228ddfb54bc6a36528d5ff07597b P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:22.389276 25151 raft_consensus.cc:482] T 0da4d5283c8a46878bd39b85e5c71f66 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
I20260809 15:46:22.389770 25151 raft_consensus.cc:504] T 0da4d5283c8a46878bd39b85e5c71f66 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:22.393441 25151 leader_election.cc:288] T 0da4d5283c8a46878bd39b85e5c71f66 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:22.393990 25151 raft_consensus.cc:482] T 30c1740ef6874df0a332fbd6fd4b03ce P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
I20260809 15:46:22.394433 25151 raft_consensus.cc:504] T 30c1740ef6874df0a332fbd6fd4b03ce P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:22.428426 25151 leader_election.cc:288] T 30c1740ef6874df0a332fbd6fd4b03ce P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:22.487640 25151 raft_consensus.cc:482] T 07fcabb569414bddbd39df6cedd24e64 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
W20260809 15:46:22.488152 23468 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.68:34689, user_credentials={real_user=slave}} blocked reactor thread for 72307.1us
I20260809 15:46:22.388497 23526 rpcz_store.cc:276] 0809 15:46:20.746618 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:20.820589 (+ 73971us) service_pool.cc:227] Handling call
0809 15:46:22.334731 (+1514142us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:22.387826 (+ 53095us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:22.388004 (+ 178us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:46:22.513268 25151 raft_consensus.cc:504] T 07fcabb569414bddbd39df6cedd24e64 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:22.384469 25366 raft_consensus.cc:504] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:22.388654 25458 raft_consensus.cc:504] T 2424fd9bd7f14e508fd106cbfda786a4 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:22.567147 25151 leader_election.cc:288] T 07fcabb569414bddbd39df6cedd24e64 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:22.405479 25453 raft_consensus.cc:504] T cee02fff2c9c48a19f43eb358a89e7b0 P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:22.578115 25151 raft_consensus.cc:482] T d2d4f749fcfd4fb3bfb8115af3471598 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:46:22.587927 25151 raft_consensus.cc:504] T d2d4f749fcfd4fb3bfb8115af3471598 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:22.593143 25151 leader_election.cc:288] T d2d4f749fcfd4fb3bfb8115af3471598 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:22.512948 25478 raft_consensus.cc:482] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
W20260809 15:46:22.592139 23468 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.68:34689, user_credentials={real_user=slave}} blocked reactor thread for 103372us
I20260809 15:46:22.392757 25461 raft_consensus.cc:504] T 4564dfc6d4094c1b9240b367ccecf3b1 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:22.506505 25253 raft_consensus.cc:504] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:22.609519 25151 raft_consensus.cc:482] T 294d4bcc1e7a477cb56455581277a60f P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
I20260809 15:46:22.407774 25462 raft_consensus.cc:504] T 6ae14a74a26f434db08c652e097385c8 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:22.610816 25151 raft_consensus.cc:504] T 294d4bcc1e7a477cb56455581277a60f P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:22.576889 25457 raft_consensus.cc:482] T 953eea268e17491ca76f3a555860a512 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:22.566056 25496 raft_consensus.cc:482] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:46:22.590411 25467 raft_consensus.cc:482] T d3e94b7e50754a81b3d72a07476cb7b9 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:22.615211 25453 leader_election.cc:288] T cee02fff2c9c48a19f43eb358a89e7b0 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:22.617762 25461 leader_election.cc:288] T 4564dfc6d4094c1b9240b367ccecf3b1 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:22.618208 25453 raft_consensus.cc:482] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:46:22.615962 25477 raft_consensus.cc:482] T 998c6cee72cf40228c6dba064c0db2da P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:22.619786 25497 raft_consensus.cc:482] T f9e7cadcd24d45fcbfa84209225e8f2d P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:22.616935 25462 leader_election.cc:288] T 6ae14a74a26f434db08c652e097385c8 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:22.620306 25497 raft_consensus.cc:504] T f9e7cadcd24d45fcbfa84209225e8f2d P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:22.619593 25477 raft_consensus.cc:504] T 998c6cee72cf40228c6dba064c0db2da P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:46:22.623325 24403 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.360s user 0.015s sys 0.025s
I20260809 15:46:22.618319 25461 raft_consensus.cc:482] T cef3e7277d7d4760824a3436ed4a847d P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
W20260809 15:46:22.623755 24403 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.360s user 0.015s sys 0.025s
I20260809 15:46:22.622362 25497 leader_election.cc:288] T f9e7cadcd24d45fcbfa84209225e8f2d P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:22.618745 25453 raft_consensus.cc:504] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:22.659171 25497 raft_consensus.cc:482] T 05ad7d8fdca64e3da527280ca5be9280 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
I20260809 15:46:22.661900 25497 raft_consensus.cc:504] T 05ad7d8fdca64e3da527280ca5be9280 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:22.664072 25497 leader_election.cc:288] T 05ad7d8fdca64e3da527280ca5be9280 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:22.618645 25488 raft_consensus.cc:482] T 0a51c0baaa6e432997c9d3552af4f254 P 61e69695258f48baa984c9d395e8ce3f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
I20260809 15:46:22.606513 25478 raft_consensus.cc:504] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:22.668748 25473 raft_consensus.cc:482] T e14d118b2eaa42aa8a7cd4025aba7128 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
I20260809 15:46:22.669235 25477 leader_election.cc:288] T 998c6cee72cf40228c6dba064c0db2da P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:22.669970 25478 leader_election.cc:288] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:22.669227 25473 raft_consensus.cc:504] T e14d118b2eaa42aa8a7cd4025aba7128 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:22.670517 25478 raft_consensus.cc:482] T 0c48a3049897452383257aa757834fb5 P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
I20260809 15:46:22.636251 25457 raft_consensus.cc:504] T 953eea268e17491ca76f3a555860a512 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:22.671312 25483 raft_consensus.cc:482] T 30c1740ef6874df0a332fbd6fd4b03ce P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
I20260809 15:46:22.671782 25473 leader_election.cc:288] T e14d118b2eaa42aa8a7cd4025aba7128 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:22.671823 25483 raft_consensus.cc:504] T 30c1740ef6874df0a332fbd6fd4b03ce P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:22.672492 25473 raft_consensus.cc:482] T 73f6f44c259547f1b05a3a434616fa54 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:22.672247 25457 leader_election.cc:288] T 953eea268e17491ca76f3a555860a512 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:22.617878 25467 raft_consensus.cc:504] T d3e94b7e50754a81b3d72a07476cb7b9 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:22.671006 25478 raft_consensus.cc:504] T 0c48a3049897452383257aa757834fb5 P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:22.717628 25478 leader_election.cc:288] T 0c48a3049897452383257aa757834fb5 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:22.718076 25478 raft_consensus.cc:482] T 3296ef50be254fc29cfc77d22a8f7a23 P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
I20260809 15:46:22.718461 25478 raft_consensus.cc:504] T 3296ef50be254fc29cfc77d22a8f7a23 P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:22.720218 25478 leader_election.cc:288] T 3296ef50be254fc29cfc77d22a8f7a23 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:22.720634 25478 raft_consensus.cc:482] T c0741ab44d99439abc3aa3bdd27ee160 P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:22.721035 25478 raft_consensus.cc:504] T c0741ab44d99439abc3aa3bdd27ee160 P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:22.721495 25465 raft_consensus.cc:482] T f9e7cadcd24d45fcbfa84209225e8f2d P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:22.722060 25465 raft_consensus.cc:504] T f9e7cadcd24d45fcbfa84209225e8f2d P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:22.623888 25461 raft_consensus.cc:504] T cef3e7277d7d4760824a3436ed4a847d P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:22.670428 25477 raft_consensus.cc:482] T d2d4f749fcfd4fb3bfb8115af3471598 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:46:22.711879 25473 raft_consensus.cc:504] T 73f6f44c259547f1b05a3a434616fa54 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:22.723475 25478 leader_election.cc:288] T c0741ab44d99439abc3aa3bdd27ee160 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:22.617785 25496 raft_consensus.cc:504] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:22.757222 25458 leader_election.cc:288] T 2424fd9bd7f14e508fd106cbfda786a4 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:22.758078 23270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c48a3049897452383257aa757834fb5" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:22.758329 23285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3296ef50be254fc29cfc77d22a8f7a23" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:22.759367 23274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee02fff2c9c48a19f43eb358a89e7b0" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:22.771430 23274 raft_consensus.cc:2454] T cee02fff2c9c48a19f43eb358a89e7b0 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 1.
I20260809 15:46:22.759207 25489 raft_consensus.cc:482] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:22.760203 25478 raft_consensus.cc:482] T 3d880de657e544468c82157769622a26 P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
I20260809 15:46:22.753532 25465 leader_election.cc:288] T f9e7cadcd24d45fcbfa84209225e8f2d P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:22.780843 23285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0741ab44d99439abc3aa3bdd27ee160" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:22.781571 23285 raft_consensus.cc:2454] T c0741ab44d99439abc3aa3bdd27ee160 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 1.
I20260809 15:46:22.762426 25462 raft_consensus.cc:482] T 73f6f44c259547f1b05a3a434616fa54 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:22.774650 25507 raft_consensus.cc:482] T 8e4ffcf83c234ae7910ddef3ec0b4bda P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
I20260809 15:46:22.763298 25501 raft_consensus.cc:482] T ba56725e42f74a1baa4c04d9660727f9 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:22.767628 25515 raft_consensus.cc:482] T 472816994d814cb198b69e5248697440 P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:22.743103 25477 raft_consensus.cc:504] T d2d4f749fcfd4fb3bfb8115af3471598 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:22.769043 25497 raft_consensus.cc:482] T 2db057fd10e149a9895dff6ef8c9f038 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:22.786566 25507 raft_consensus.cc:504] T 8e4ffcf83c234ae7910ddef3ec0b4bda P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:22.773382 25519 raft_consensus.cc:482] T 213543cdbe814c89a17a963469922c1b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:46:22.774660 25471 raft_consensus.cc:482] T 3296ef50be254fc29cfc77d22a8f7a23 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
I20260809 15:46:22.725090 25488 raft_consensus.cc:504] T 0a51c0baaa6e432997c9d3552af4f254 P 61e69695258f48baa984c9d395e8ce3f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:22.786895 25501 raft_consensus.cc:504] T ba56725e42f74a1baa4c04d9660727f9 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:22.787477 25462 raft_consensus.cc:504] T 73f6f44c259547f1b05a3a434616fa54 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:46:22.838063 24406 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 76110.5us
I20260809 15:46:22.793553 25478 raft_consensus.cc:504] T 3d880de657e544468c82157769622a26 P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:22.805757 25453 leader_election.cc:288] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:22.806011 25521 raft_consensus.cc:482] T 16ca412a0dae4feebe85dd01bf1a9173 P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
I20260809 15:46:22.807863 25483 leader_election.cc:288] T 30c1740ef6874df0a332fbd6fd4b03ce P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:22.806665 25497 raft_consensus.cc:504] T 2db057fd10e149a9895dff6ef8c9f038 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:22.851914 25457 raft_consensus.cc:482] T 297c82ed906d4b48b184985eed3de601 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:22.846707 25458 raft_consensus.cc:482] T 7baa2492df0340c4a5ac241959ce1ac3 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
I20260809 15:46:22.865305 25457 raft_consensus.cc:504] T 297c82ed906d4b48b184985eed3de601 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:22.798926 25515 raft_consensus.cc:504] T 472816994d814cb198b69e5248697440 P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:22.866824 25523 raft_consensus.cc:482] T a88421dba80d4d3388aa5eb62e23853b P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:22.876682 25515 leader_election.cc:288] T 472816994d814cb198b69e5248697440 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:22.867511 25473 leader_election.cc:288] T 73f6f44c259547f1b05a3a434616fa54 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:22.863374 25461 leader_election.cc:288] T cef3e7277d7d4760824a3436ed4a847d P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:22.865509 25467 leader_election.cc:288] T d3e94b7e50754a81b3d72a07476cb7b9 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:22.877682 25515 raft_consensus.cc:482] T e71a05ba1d4d42bd95c3f85f9313399e P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:46:22.857908 25483 raft_consensus.cc:482] T f7715d71dbac447b9351fc5eb36c5e80 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:22.878247 25515 raft_consensus.cc:504] T e71a05ba1d4d42bd95c3f85f9313399e P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:22.878352 25483 raft_consensus.cc:504] T f7715d71dbac447b9351fc5eb36c5e80 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:22.785391 25489 raft_consensus.cc:504] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:22.880069 25496 leader_election.cc:288] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:22.880533 23280 raft_consensus.cc:1261] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Refusing update from remote peer 61e69695258f48baa984c9d395e8ce3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260809 15:46:22.881150 25489 leader_election.cc:288] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:22.881515 25479 raft_consensus.cc:482] T 763e58d6ba644eea830a6c1664d39c9b P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:22.881536 25497 leader_election.cc:288] T 2db057fd10e149a9895dff6ef8c9f038 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:22.881973 25501 leader_election.cc:288] T ba56725e42f74a1baa4c04d9660727f9 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:22.880168 25483 leader_election.cc:288] T f7715d71dbac447b9351fc5eb36c5e80 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:22.877725 25457 leader_election.cc:288] T 297c82ed906d4b48b184985eed3de601 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:22.888118 25527 raft_consensus.cc:482] T 0da4d5283c8a46878bd39b85e5c71f66 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
I20260809 15:46:22.890121 25477 leader_election.cc:288] T d2d4f749fcfd4fb3bfb8115af3471598 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:22.882184 25512 raft_consensus.cc:482] T 7660dfa2ef694927b8eda4100a2bcd9b P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:22.903191 25497 raft_consensus.cc:482] T 563daf42d7784db195ea99c3cb21b370 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
I20260809 15:46:22.850900 25521 raft_consensus.cc:504] T 16ca412a0dae4feebe85dd01bf1a9173 P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:22.913452 25483 raft_consensus.cc:482] T ba56725e42f74a1baa4c04d9660727f9 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:22.832330 25519 raft_consensus.cc:504] T 213543cdbe814c89a17a963469922c1b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:22.915732 24475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472816994d814cb198b69e5248697440" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:22.922950 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472816994d814cb198b69e5248697440" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:22.866873 25510 raft_consensus.cc:482] T 00a1881267a14ba8b78c81040f17a014 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
I20260809 15:46:22.925318 23522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ae14a74a26f434db08c652e097385c8" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:22.926231 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9e7cadcd24d45fcbfa84209225e8f2d" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:22.928349 23517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05ad7d8fdca64e3da527280ca5be9280" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:22.959828 23522 raft_consensus.cc:2454] T 6ae14a74a26f434db08c652e097385c8 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 1.
I20260809 15:46:22.928824 23519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "953eea268e17491ca76f3a555860a512" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:22.959957 23517 raft_consensus.cc:2454] T 05ad7d8fdca64e3da527280ca5be9280 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 1.
I20260809 15:46:22.959357 24475 raft_consensus.cc:2454] T 472816994d814cb198b69e5248697440 P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 1.
I20260809 15:46:22.930219 23520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2424fd9bd7f14e508fd106cbfda786a4" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:22.931130 23529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30c1740ef6874df0a332fbd6fd4b03ce" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:22.903641 25486 raft_consensus.cc:482] T 907bacfa79d04a55a68c8dbe60056462 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:22.915733 25525 raft_consensus.cc:482] T 0a51c0baaa6e432997c9d3552af4f254 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
I20260809 15:46:22.931885 23400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4564dfc6d4094c1b9240b367ccecf3b1" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:22.963351 23529 raft_consensus.cc:2454] T 30c1740ef6874df0a332fbd6fd4b03ce P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 2.
I20260809 15:46:22.904513 25461 raft_consensus.cc:482] T 297c82ed906d4b48b184985eed3de601 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:22.882030 25479 raft_consensus.cc:504] T 763e58d6ba644eea830a6c1664d39c9b P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:46:22.969107 24406 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 124906us
I20260809 15:46:22.965463 23404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998c6cee72cf40228c6dba064c0db2da" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:22.971450 23404 raft_consensus.cc:2454] T 998c6cee72cf40228c6dba064c0db2da P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 3.
I20260809 15:46:22.933775 25489 raft_consensus.cc:482] T 4b30f28311964a0b9149a50495e83d8f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:46:22.974785 24406 leader_election.cc:302] T cee02fff2c9c48a19f43eb358a89e7b0 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:46:22.816538 25471 raft_consensus.cc:504] T 3296ef50be254fc29cfc77d22a8f7a23 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:22.891033 25458 raft_consensus.cc:504] T 7baa2492df0340c4a5ac241959ce1ac3 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:22.978840 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "297c82ed906d4b48b184985eed3de601" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:22.979049 24406 leader_election.cc:302] T c0741ab44d99439abc3aa3bdd27ee160 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:46:22.961508 25488 leader_election.cc:288] T 0a51c0baaa6e432997c9d3552af4f254 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:22.989894 25470 raft_consensus.cc:482] T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:22.990151 25460 raft_consensus.cc:482] T d3be1e356b254d22b995be32f1c12f6f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:46:22.974088 25477 raft_consensus.cc:482] T e71a05ba1d4d42bd95c3f85f9313399e P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:46:22.960566 25497 raft_consensus.cc:504] T 563daf42d7784db195ea99c3cb21b370 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:22.967144 25483 raft_consensus.cc:504] T ba56725e42f74a1baa4c04d9660727f9 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:22.934659 25523 raft_consensus.cc:504] T a88421dba80d4d3388aa5eb62e23853b P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:22.970618 23520 raft_consensus.cc:2454] T 2424fd9bd7f14e508fd106cbfda786a4 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 2.
I20260809 15:46:23.021862 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7715d71dbac447b9351fc5eb36c5e80" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:23.063719 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2db057fd10e149a9895dff6ef8c9f038" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:22.973594 25510 raft_consensus.cc:504] T 00a1881267a14ba8b78c81040f17a014 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:23.078966 25539 raft_consensus.cc:482] T 7660dfa2ef694927b8eda4100a2bcd9b P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:22.976245 25507 leader_election.cc:288] T 8e4ffcf83c234ae7910ddef3ec0b4bda P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:23.075884 25473 raft_consensus.cc:482] T 28e17dcb5efe45b8b6a8c42da03b4819 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:46:23.085950 23404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3296ef50be254fc29cfc77d22a8f7a23" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:22.990294 25478 leader_election.cc:288] T 3d880de657e544468c82157769622a26 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:22.981626 25501 raft_consensus.cc:482] T a88421dba80d4d3388aa5eb62e23853b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:23.095633 24475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a823d35f4f348d1b52ba9e6d80b00d2" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:22.978122 25489 raft_consensus.cc:504] T 4b30f28311964a0b9149a50495e83d8f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:22.991041 25530 raft_consensus.cc:482] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:46:23.039073 25500 raft_consensus.cc:482] T 998c6cee72cf40228c6dba064c0db2da P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:23.039915 25457 raft_consensus.cc:482] T df20671a7b874dbb98c0ef28371c1308 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:23.077328 25515 leader_election.cc:288] T e71a05ba1d4d42bd95c3f85f9313399e P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:22.962859 25527 raft_consensus.cc:504] T 0da4d5283c8a46878bd39b85e5c71f66 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:23.051167 25555 raft_consensus.cc:482] T a3622f6e3e3641cea1960e3a31498bb7 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
I20260809 15:46:23.029189 25488 raft_consensus.cc:2781] T c0741ab44d99439abc3aa3bdd27ee160 P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260809 15:46:23.147571 23404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2d4f749fcfd4fb3bfb8115af3471598" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:23.143934 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad8a3b25d7b49e5a2142d5709edd0c7" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
W20260809 15:46:23.115420 23353 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 153635us
I20260809 15:46:23.145941 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bcac4b24e7a4448abdc934bba5a624e" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:23.086436 25529 raft_consensus.cc:482] T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
I20260809 15:46:23.079874 25462 leader_election.cc:288] T 73f6f44c259547f1b05a3a434616fa54 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:22.977106 25461 raft_consensus.cc:504] T 297c82ed906d4b48b184985eed3de601 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:23.151903 23404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "213543cdbe814c89a17a963469922c1b" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:23.079430 25518 raft_consensus.cc:482] T 563daf42d7784db195ea99c3cb21b370 P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
I20260809 15:46:22.965643 25512 raft_consensus.cc:504] T 7660dfa2ef694927b8eda4100a2bcd9b P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:23.142184 25533 raft_consensus.cc:2781] T cee02fff2c9c48a19f43eb358a89e7b0 P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260809 15:46:23.096261 25519 leader_election.cc:288] T 213543cdbe814c89a17a963469922c1b P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:23.159552 23274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a51c0baaa6e432997c9d3552af4f254" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:23.161522 23270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d880de657e544468c82157769622a26" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:23.154712 24463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16ca412a0dae4feebe85dd01bf1a9173" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:23.163506 23399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "907bacfa79d04a55a68c8dbe60056462" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:23.163818 25458 leader_election.cc:288] T 7baa2492df0340c4a5ac241959ce1ac3 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:23.162827 23283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e71a05ba1d4d42bd95c3f85f9313399e" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:23.147971 25488 raft_consensus.cc:482] T c0741ab44d99439abc3aa3bdd27ee160 P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Starting leader election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:23.148188 25507 raft_consensus.cc:482] T 538a228ddfb54bc6a36528d5ff07597b P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
I20260809 15:46:23.109654 25473 raft_consensus.cc:504] T 28e17dcb5efe45b8b6a8c42da03b4819 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:23.167196 23274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16ca412a0dae4feebe85dd01bf1a9173" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:23.169065 23285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f9d6565aa4c45a3b3f01210d0eb71f7" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:23.155023 25479 leader_election.cc:288] T 763e58d6ba644eea830a6c1664d39c9b P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:23.164037 25533 raft_consensus.cc:482] T cee02fff2c9c48a19f43eb358a89e7b0 P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Starting leader election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:23.171872 25471 leader_election.cc:288] T 3296ef50be254fc29cfc77d22a8f7a23 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:23.165253 25542 consensus_queue.cc:1035] T 907bacfa79d04a55a68c8dbe60056462 P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.055s
I20260809 15:46:22.991858 25525 raft_consensus.cc:504] T 0a51c0baaa6e432997c9d3552af4f254 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:23.169116 24463 raft_consensus.cc:2454] T 16ca412a0dae4feebe85dd01bf1a9173 P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 1.
I20260809 15:46:23.170704 25528 raft_consensus.cc:482] T 953eea268e17491ca76f3a555860a512 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:23.170773 25536 raft_consensus.cc:482] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:46:23.171101 25551 raft_consensus.cc:482] T c5c2e425a74b4e64ae70b5def750434c P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:23.171494 25521 leader_election.cc:288] T 16ca412a0dae4feebe85dd01bf1a9173 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:22.977314 25486 raft_consensus.cc:504] T 907bacfa79d04a55a68c8dbe60056462 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:23.077600 25477 raft_consensus.cc:504] T e71a05ba1d4d42bd95c3f85f9313399e P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:23.144876 25457 raft_consensus.cc:504] T df20671a7b874dbb98c0ef28371c1308 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:23.153728 25500 raft_consensus.cc:504] T 998c6cee72cf40228c6dba064c0db2da P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:23.190723 25457 leader_election.cc:288] T df20671a7b874dbb98c0ef28371c1308 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:23.163329 25519 raft_consensus.cc:482] T d0721a4403a440f383318a3cea9d1442 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:46:23.149073 25501 raft_consensus.cc:504] T a88421dba80d4d3388aa5eb62e23853b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:23.191185 25500 leader_election.cc:288] T 998c6cee72cf40228c6dba064c0db2da P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:22.990408 25470 raft_consensus.cc:504] T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:23.181732 25527 leader_election.cc:288] T 0da4d5283c8a46878bd39b85e5c71f66 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:23.181558 25540 raft_consensus.cc:482] T 0c29937a80724ddb9d07a27534a146f3 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:46:23.184144 25532 raft_consensus.cc:482] T c5c2e425a74b4e64ae70b5def750434c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:23.184993 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e14d118b2eaa42aa8a7cd4025aba7128" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:23.188125 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "907bacfa79d04a55a68c8dbe60056462" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:23.169607 25488 raft_consensus.cc:3037] T c0741ab44d99439abc3aa3bdd27ee160 P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:23.165414 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3e94b7e50754a81b3d72a07476cb7b9" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:23.192788 23468 leader_election.cc:302] T 16ca412a0dae4feebe85dd01bf1a9173 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c; no voters:
I20260809 15:46:23.176322 25533 raft_consensus.cc:3037] T cee02fff2c9c48a19f43eb358a89e7b0 P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:23.194458 23283 raft_consensus.cc:2454] T e71a05ba1d4d42bd95c3f85f9313399e P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 1.
I20260809 15:46:23.196553 25548 raft_consensus.cc:482] T 99bdb93503db4de4a1cfb0161ea0c07c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:46:23.172667 25507 raft_consensus.cc:504] T 538a228ddfb54bc6a36528d5ff07597b P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:23.197687 23400 raft_consensus.cc:1261] T 907bacfa79d04a55a68c8dbe60056462 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Refusing update from remote peer 61e69695258f48baa984c9d395e8ce3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260809 15:46:23.153931 25530 raft_consensus.cc:504] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:23.207644 25510 leader_election.cc:288] T 00a1881267a14ba8b78c81040f17a014 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:23.197477 25540 raft_consensus.cc:504] T 0c29937a80724ddb9d07a27534a146f3 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:23.208159 25523 leader_election.cc:288] T a88421dba80d4d3388aa5eb62e23853b P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:23.207662 25486 leader_election.cc:288] T 907bacfa79d04a55a68c8dbe60056462 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:23.204528 25481 raft_consensus.cc:482] T df20671a7b874dbb98c0ef28371c1308 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:23.141752 25460 raft_consensus.cc:504] T d3be1e356b254d22b995be32f1c12f6f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:23.222500 23394 raft_consensus.cc:1261] T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Refusing update from remote peer 61e69695258f48baa984c9d395e8ce3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260809 15:46:23.223120 23353 leader_election.cc:302] T 6ae14a74a26f434db08c652e097385c8 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:46:23.161381 25529 raft_consensus.cc:504] T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:23.201433 25461 leader_election.cc:288] T 297c82ed906d4b48b184985eed3de601 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:23.225569 25503 raft_consensus.cc:482] T 07fcabb569414bddbd39df6cedd24e64 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:46:23.116506 25539 raft_consensus.cc:504] T 7660dfa2ef694927b8eda4100a2bcd9b P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:23.228025 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e71a05ba1d4d42bd95c3f85f9313399e" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:23.157546 25555 raft_consensus.cc:504] T a3622f6e3e3641cea1960e3a31498bb7 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:23.228546 24472 raft_consensus.cc:2454] T e71a05ba1d4d42bd95c3f85f9313399e P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 1.
I20260809 15:46:23.198515 25519 raft_consensus.cc:504] T d0721a4403a440f383318a3cea9d1442 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:23.206161 25483 leader_election.cc:288] T ba56725e42f74a1baa4c04d9660727f9 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:23.206395 25497 leader_election.cc:288] T 563daf42d7784db195ea99c3cb21b370 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:23.232333 23223 leader_election.cc:302] T e71a05ba1d4d42bd95c3f85f9313399e P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:46:23.164618 25518 raft_consensus.cc:504] T 563daf42d7784db195ea99c3cb21b370 P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:23.193434 25528 raft_consensus.cc:504] T 953eea268e17491ca76f3a555860a512 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:23.205413 25532 raft_consensus.cc:504] T c5c2e425a74b4e64ae70b5def750434c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:23.223094 25500 raft_consensus.cc:482] T 0acc47ce21c3422698d1ef7277c3ed63 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
I20260809 15:46:23.244385 23270 raft_consensus.cc:1261] T 297c82ed906d4b48b184985eed3de601 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 15:46:23.257982 23353 leader_election.cc:302] T 05ad7d8fdca64e3da527280ca5be9280 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:46:23.226395 25501 leader_election.cc:288] T a88421dba80d4d3388aa5eb62e23853b P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:23.228295 25477 leader_election.cc:288] T e71a05ba1d4d42bd95c3f85f9313399e P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:23.205099 25531 raft_consensus.cc:482] T ae6d2c08a7a94f58ac8dd698e3fd456c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:46:23.261059 23353 leader_election.cc:302] T 30c1740ef6874df0a332fbd6fd4b03ce P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:46:23.225677 25481 raft_consensus.cc:504] T df20671a7b874dbb98c0ef28371c1308 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:23.263154 25457 raft_consensus.cc:2781] T 30c1740ef6874df0a332fbd6fd4b03ce P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260809 15:46:23.236721 23472 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.20.202.66:36263 timed out after 1.560s (SENT) [suppressed 4 similar messages]
I20260809 15:46:23.231577 25489 leader_election.cc:288] T 4b30f28311964a0b9149a50495e83d8f P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:23.263545 25457 raft_consensus.cc:482] T 30c1740ef6874df0a332fbd6fd4b03ce P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting leader election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
I20260809 15:46:23.256520 25507 leader_election.cc:288] T 538a228ddfb54bc6a36528d5ff07597b P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:23.255373 25524 raft_consensus.cc:482] T 763e58d6ba644eea830a6c1664d39c9b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:23.235476 25461 raft_consensus.cc:482] T 2db057fd10e149a9895dff6ef8c9f038 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:23.260339 25527 raft_consensus.cc:2781] T 6ae14a74a26f434db08c652e097385c8 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260809 15:46:23.260157 25462 raft_consensus.cc:2781] T 05ad7d8fdca64e3da527280ca5be9280 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260809 15:46:23.199175 25551 raft_consensus.cc:504] T c5c2e425a74b4e64ae70b5def750434c P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:23.256000 25573 raft_consensus.cc:482] T d1a134ba33ea4076bbaac0ae460c17ba P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:46:23.255568 25574 raft_consensus.cc:482] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:46:23.259230 25500 raft_consensus.cc:504] T 0acc47ce21c3422698d1ef7277c3ed63 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:23.222076 25548 raft_consensus.cc:504] T 99bdb93503db4de4a1cfb0161ea0c07c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:23.193295 25536 raft_consensus.cc:504] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:23.226089 25503 raft_consensus.cc:504] T 07fcabb569414bddbd39df6cedd24e64 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:23.265937 25457 raft_consensus.cc:3037] T 30c1740ef6874df0a332fbd6fd4b03ce P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:23.479631 25470 leader_election.cc:288] T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:23.464519 25460 leader_election.cc:288] T d3be1e356b254d22b995be32f1c12f6f P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:23.465687 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3be1e356b254d22b995be32f1c12f6f" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
W20260809 15:46:23.680542 23223 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.68:34689, user_credentials={real_user=slave}} blocked reactor thread for 115751us
I20260809 15:46:23.353343 23273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9e7cadcd24d45fcbfa84209225e8f2d" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:23.810537 23273 raft_consensus.cc:2454] T f9e7cadcd24d45fcbfa84209225e8f2d P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 1.
I20260809 15:46:23.814745 25574 raft_consensus.cc:504] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:23.494805 25530 leader_election.cc:288] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:23.465870 25529 leader_election.cc:288] T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:23.808780 25527 raft_consensus.cc:482] T 6ae14a74a26f434db08c652e097385c8 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
W20260809 15:46:23.618700 24506 debug-util.cc:398] Leaking SignalData structure 0x7b0800140760 after lost signal to thread 21293
I20260809 15:46:23.490132 25571 raft_consensus.cc:2781] T e71a05ba1d4d42bd95c3f85f9313399e P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260809 15:46:23.480278 25525 leader_election.cc:288] T 0a51c0baaa6e432997c9d3552af4f254 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
W20260809 15:46:23.822371 23472 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 585647us
I20260809 15:46:23.522823 25539 leader_election.cc:288] T 7660dfa2ef694927b8eda4100a2bcd9b P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:23.820004 23285 raft_consensus.cc:2454] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 1.
I20260809 15:46:23.819933 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e4ffcf83c234ae7910ddef3ec0b4bda" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:23.816969 23353 leader_election.cc:302] T 2424fd9bd7f14e508fd106cbfda786a4 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:46:23.274034 25569 raft_consensus.cc:482] T f7715d71dbac447b9351fc5eb36c5e80 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:23.825230 23393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3aa6a9a52804df0ba9b0ca3c82f38c3" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:23.820667 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cef3e7277d7d4760824a3436ed4a847d" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:23.810521 25462 raft_consensus.cc:482] T 05ad7d8fdca64e3da527280ca5be9280 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
I20260809 15:46:23.831597 24463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "563daf42d7784db195ea99c3cb21b370" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:23.819213 25528 leader_election.cc:288] T 953eea268e17491ca76f3a555860a512 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:23.831984 25462 raft_consensus.cc:3037] T 05ad7d8fdca64e3da527280ca5be9280 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:23.832726 25551 leader_election.cc:288] T c5c2e425a74b4e64ae70b5def750434c P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:23.802107 25519 leader_election.cc:288] T d0721a4403a440f383318a3cea9d1442 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:23.826642 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3296ef50be254fc29cfc77d22a8f7a23" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:23.810057 25548 leader_election.cc:288] T 99bdb93503db4de4a1cfb0161ea0c07c P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:23.283318 25473 leader_election.cc:288] T 28e17dcb5efe45b8b6a8c42da03b4819 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:23.494065 25512 leader_election.cc:288] T 7660dfa2ef694927b8eda4100a2bcd9b P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:23.277112 25555 leader_election.cc:288] T a3622f6e3e3641cea1960e3a31498bb7 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:23.490335 25559 raft_consensus.cc:482] T cef3e7277d7d4760824a3436ed4a847d P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:23.630072 25488 raft_consensus.cc:504] T c0741ab44d99439abc3aa3bdd27ee160 P 61e69695258f48baa984c9d395e8ce3f [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:23.490016 25518 leader_election.cc:288] T 563daf42d7784db195ea99c3cb21b370 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:23.819676 25527 raft_consensus.cc:3037] T 6ae14a74a26f434db08c652e097385c8 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:23.818827 25574 leader_election.cc:288] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:23.818328 23399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00a1881267a14ba8b78c81040f17a014" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:23.852844 25505 raft_consensus.cc:482] T 3924b648461e463b9d7925e5671d68ae P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
I20260809 15:46:23.836359 23273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0da4d5283c8a46878bd39b85e5c71f66" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:23.273741 25461 raft_consensus.cc:504] T 2db057fd10e149a9895dff6ef8c9f038 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:23.819074 25503 leader_election.cc:288] T 07fcabb569414bddbd39df6cedd24e64 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:23.843617 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "907bacfa79d04a55a68c8dbe60056462" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:23.633576 25533 raft_consensus.cc:504] T cee02fff2c9c48a19f43eb358a89e7b0 P 61e69695258f48baa984c9d395e8ce3f [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:23.819622 25573 raft_consensus.cc:504] T d1a134ba33ea4076bbaac0ae460c17ba P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:23.847587 25472 raft_consensus.cc:482] T 7baa2492df0340c4a5ac241959ce1ac3 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
I20260809 15:46:23.282944 25524 raft_consensus.cc:504] T 763e58d6ba644eea830a6c1664d39c9b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:46:23.851329 23468 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.68:34689, user_credentials={real_user=slave}} blocked reactor thread for 658564us
I20260809 15:46:23.841090 24463 raft_consensus.cc:2454] T 563daf42d7784db195ea99c3cb21b370 P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 1.
I20260809 15:46:23.838129 24455 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3924b648461e463b9d7925e5671d68ae" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:23.853335 25505 raft_consensus.cc:504] T 3924b648461e463b9d7925e5671d68ae P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:23.820133 25571 raft_consensus.cc:482] T e71a05ba1d4d42bd95c3f85f9313399e P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:46:23.837029 25539 raft_consensus.cc:2781] T 2424fd9bd7f14e508fd106cbfda786a4 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260809 15:46:23.853904 23399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b30f28311964a0b9149a50495e83d8f" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:23.835377 25369 raft_consensus.cc:482] T 0c48a3049897452383257aa757834fb5 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
I20260809 15:46:23.862028 24475 raft_consensus.cc:1226] T c0741ab44d99439abc3aa3bdd27ee160 P 61e69695258f48baa984c9d395e8ce3f [term 2 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 1. Current term is 2. Ops: []
I20260809 15:46:23.854060 25467 raft_consensus.cc:482] T a3622f6e3e3641cea1960e3a31498bb7 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
I20260809 15:46:23.848078 25490 raft_consensus.cc:482] T 0acc47ce21c3422698d1ef7277c3ed63 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
I20260809 15:46:23.888665 23517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0da4d5283c8a46878bd39b85e5c71f66" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:23.890296 23517 raft_consensus.cc:2454] T 0da4d5283c8a46878bd39b85e5c71f66 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 1.
I20260809 15:46:23.862264 23228 leader_election.cc:302] T 998c6cee72cf40228c6dba064c0db2da P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:46:23.866556 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73f6f44c259547f1b05a3a434616fa54" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
W20260809 15:46:23.838788 24506 debug-util.cc:398] Leaking SignalData structure 0x7b08002e2200 after lost signal to thread 23176
I20260809 15:46:23.842253 24453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3e94b7e50754a81b3d72a07476cb7b9" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:23.870069 23522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df20671a7b874dbb98c0ef28371c1308" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:23.870774 23345 leader_election.cc:302] T 472816994d814cb198b69e5248697440 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c
I20260809 15:46:23.869040 25533 leader_election.cc:288] T cee02fff2c9c48a19f43eb358a89e7b0 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 2 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:23.856760 25500 leader_election.cc:288] T 0acc47ce21c3422698d1ef7277c3ed63 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:23.282608 25531 raft_consensus.cc:504] T ae6d2c08a7a94f58ac8dd698e3fd456c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:23.895730 23351 leader_election.cc:302] T f9e7cadcd24d45fcbfa84209225e8f2d P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c
I20260809 15:46:23.896260 23345 leader_election.cc:302] T 563daf42d7784db195ea99c3cb21b370 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:46:23.896422 23273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3622f6e3e3641cea1960e3a31498bb7" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:23.876921 25571 raft_consensus.cc:3037] T e71a05ba1d4d42bd95c3f85f9313399e P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:23.869144 23393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bcac4b24e7a4448abdc934bba5a624e" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:23.917155 23270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df20671a7b874dbb98c0ef28371c1308" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
W20260809 15:46:23.975373 25577 log.cc:924] Time spent T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9: Append to log took a long time: real 0.123s user 0.001s sys 0.000s
I20260809 15:46:23.975823 23399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f9d6565aa4c45a3b3f01210d0eb71f7" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:23.976450 23273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e4ffcf83c234ae7910ddef3ec0b4bda" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:23.977029 23399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e17dcb5efe45b8b6a8c42da03b4819" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:23.977502 23273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ae14a74a26f434db08c652e097385c8" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:23.978166 23399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a823d35f4f348d1b52ba9e6d80b00d2" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:23.978538 23273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7715d71dbac447b9351fc5eb36c5e80" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:23.877702 25506 raft_consensus.cc:2781] T 16ca412a0dae4feebe85dd01bf1a9173 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260809 15:46:23.932897 23517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998c6cee72cf40228c6dba064c0db2da" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:23.980778 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00a1881267a14ba8b78c81040f17a014" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:23.921077 23228 leader_election.cc:302] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:23.983032 23274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a88421dba80d4d3388aa5eb62e23853b" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:23.983608 23274 raft_consensus.cc:2454] T a88421dba80d4d3388aa5eb62e23853b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 1.
I20260809 15:46:23.982415 23526 raft_consensus.cc:2454] T 00a1881267a14ba8b78c81040f17a014 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 2.
I20260809 15:46:23.898520 25483 raft_consensus.cc:2781] T 472816994d814cb198b69e5248697440 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260809 15:46:23.987248 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4564dfc6d4094c1b9240b367ccecf3b1" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:23.878043 25539 raft_consensus.cc:482] T 2424fd9bd7f14e508fd106cbfda786a4 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting leader election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
W20260809 15:46:23.988334 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 67348.3us
I20260809 15:46:23.919108 23270 raft_consensus.cc:2454] T df20671a7b874dbb98c0ef28371c1308 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 1.
I20260809 15:46:23.918754 25551 raft_consensus.cc:2781] T f9e7cadcd24d45fcbfa84209225e8f2d P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260809 15:46:23.895961 25460 raft_consensus.cc:2781] T 998c6cee72cf40228c6dba064c0db2da P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260809 15:46:23.891126 25467 raft_consensus.cc:504] T a3622f6e3e3641cea1960e3a31498bb7 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:23.993156 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a88421dba80d4d3388aa5eb62e23853b" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:23.928113 25366 leader_election.cc:288] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:23.997031 25587 raft_consensus.cc:2726] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 15:46:23.997368 24410 leader_election.cc:302] T 3296ef50be254fc29cfc77d22a8f7a23 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f; no voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:23.985756 25506 raft_consensus.cc:482] T 16ca412a0dae4feebe85dd01bf1a9173 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Starting leader election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
I20260809 15:46:24.025322 23270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2424fd9bd7f14e508fd106cbfda786a4" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:23.927403 25555 raft_consensus.cc:2781] T 563daf42d7784db195ea99c3cb21b370 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260809 15:46:23.865427 25559 raft_consensus.cc:504] T cef3e7277d7d4760824a3436ed4a847d P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:24.028239 23517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998c6cee72cf40228c6dba064c0db2da" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:23.989995 25483 raft_consensus.cc:482] T 472816994d814cb198b69e5248697440 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
W20260809 15:46:23.996088 24506 debug-util.cc:398] Leaking SignalData structure 0x7b080035a680 after lost signal to thread 23320
I20260809 15:46:23.889230 25369 raft_consensus.cc:504] T 0c48a3049897452383257aa757834fb5 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
W20260809 15:46:24.029150 23345 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.68:34689, user_credentials={real_user=slave}} blocked reactor thread for 132944us
I20260809 15:46:23.992341 25539 raft_consensus.cc:3037] T 2424fd9bd7f14e508fd106cbfda786a4 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:24.035064 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "538a228ddfb54bc6a36528d5ff07597b" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:24.036398 23526 raft_consensus.cc:2454] T 538a228ddfb54bc6a36528d5ff07597b P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 2.
I20260809 15:46:23.974005 25536 leader_election.cc:288] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:24.028985 25555 raft_consensus.cc:482] T 563daf42d7784db195ea99c3cb21b370 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
I20260809 15:46:23.867121 25472 raft_consensus.cc:504] T 7baa2492df0340c4a5ac241959ce1ac3 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:23.996677 25460 raft_consensus.cc:482] T 998c6cee72cf40228c6dba064c0db2da P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:24.038563 23270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2db057fd10e149a9895dff6ef8c9f038" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:24.039870 23270 raft_consensus.cc:2454] T 2db057fd10e149a9895dff6ef8c9f038 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 1.
I20260809 15:46:24.026625 25506 raft_consensus.cc:3037] T 16ca412a0dae4feebe85dd01bf1a9173 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:23.992478 25551 raft_consensus.cc:482] T f9e7cadcd24d45fcbfa84209225e8f2d P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:23.854616 25569 raft_consensus.cc:504] T f7715d71dbac447b9351fc5eb36c5e80 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:24.034132 25483 raft_consensus.cc:3037] T 472816994d814cb198b69e5248697440 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:23.998095 24455 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f9d6565aa4c45a3b3f01210d0eb71f7" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:24.046955 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9e7cadcd24d45fcbfa84209225e8f2d" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:24.034884 25533 raft_consensus.cc:2726] T 3296ef50be254fc29cfc77d22a8f7a23 P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 15:46:24.047582 23270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "297c82ed906d4b48b184985eed3de601" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:24.031083 25366 raft_consensus.cc:482] T e14d118b2eaa42aa8a7cd4025aba7128 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
I20260809 15:46:24.054803 23270 raft_consensus.cc:2454] T 297c82ed906d4b48b184985eed3de601 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 2.
I20260809 15:46:24.048127 25461 leader_election.cc:288] T 2db057fd10e149a9895dff6ef8c9f038 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:24.048242 23520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3924b648461e463b9d7925e5671d68ae" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:24.049414 23396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73f6f44c259547f1b05a3a434616fa54" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:24.039146 25488 leader_election.cc:288] T c0741ab44d99439abc3aa3bdd27ee160 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 2 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:24.052145 23404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad8a3b25d7b49e5a2142d5709edd0c7" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:23.998597 25573 leader_election.cc:288] T d1a134ba33ea4076bbaac0ae460c17ba P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:24.056504 23402 raft_consensus.cc:2454] T cef3e7277d7d4760824a3436ed4a847d P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 1.
I20260809 15:46:24.051349 23283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30c1740ef6874df0a332fbd6fd4b03ce" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:24.057621 25151 leader_election.cc:288] T 294d4bcc1e7a477cb56455581277a60f P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:24.038982 25555 raft_consensus.cc:3037] T 563daf42d7784db195ea99c3cb21b370 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:24.042652 23393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472816994d814cb198b69e5248697440" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:24.040843 23518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e14d118b2eaa42aa8a7cd4025aba7128" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:24.063205 23399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad8a3b25d7b49e5a2142d5709edd0c7" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:24.042287 25460 raft_consensus.cc:3037] T 998c6cee72cf40228c6dba064c0db2da P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:46:24.056646 23396 raft_consensus.cc:2454] T 73f6f44c259547f1b05a3a434616fa54 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 1.
I20260809 15:46:24.061964 23520 raft_consensus.cc:2454] T 3924b648461e463b9d7925e5671d68ae P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 2.
I20260809 15:46:24.045327 25551 raft_consensus.cc:3037] T f9e7cadcd24d45fcbfa84209225e8f2d P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:23.892983 25490 raft_consensus.cc:504] T 0acc47ce21c3422698d1ef7277c3ed63 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:24.032661 23392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba56725e42f74a1baa4c04d9660727f9" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
W20260809 15:46:24.041800 24410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 44517.8us
I20260809 15:46:24.061579 25366 raft_consensus.cc:504] T e14d118b2eaa42aa8a7cd4025aba7128 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:24.053648 24406 leader_election.cc:302] T e71a05ba1d4d42bd95c3f85f9313399e P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:46:24.068450 23270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "953eea268e17491ca76f3a555860a512" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:24.040889 25505 leader_election.cc:288] T 3924b648461e463b9d7925e5671d68ae P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:24.070338 23404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2424fd9bd7f14e508fd106cbfda786a4" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:24.070282 23396 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55332 (request call id 1072) took 1055 ms. Trace:
W20260809 15:46:24.066095 24506 debug-util.cc:398] Leaking SignalData structure 0x7b080024d9e0 after lost signal to thread 23443
I20260809 15:46:24.070729 23273 raft_consensus.cc:2454] T f7715d71dbac447b9351fc5eb36c5e80 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 1.
I20260809 15:46:24.067473 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2d4f749fcfd4fb3bfb8115af3471598" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:24.071684 25151 consensus_queue.cc:1035] T 297c82ed906d4b48b184985eed3de601 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 15:46:24.072631 23392 raft_consensus.cc:2454] T ba56725e42f74a1baa4c04d9660727f9 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 1.
I20260809 15:46:24.072764 23520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3622f6e3e3641cea1960e3a31498bb7" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:24.072933 25524 leader_election.cc:288] T 763e58d6ba644eea830a6c1664d39c9b P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:24.063761 25462 raft_consensus.cc:504] T 05ad7d8fdca64e3da527280ca5be9280 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:24.074633 23283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998c6cee72cf40228c6dba064c0db2da" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:24.076329 23396 rpcz_store.cc:276] 0809 15:46:23.012634 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:23.012754 (+ 120us) service_pool.cc:227] Handling call
0809 15:46:24.049291 (+1036537us) spinlock_profiling.cc:243] Waited 15.5 ms on lock 0x7b58001d7c98. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e613cf11 00007f34e62c0851 00007f34e62c0702 00007f34ef1b0b09 00007f34ee0ec730 00007f34ee1b279b 00007f34ee1b2821 00007f34ee1621b0 00007f34ee161a94 00007f34e7849c04 00007f34e7849b92
0809 15:46:24.068155 (+ 18864us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":35612032}
I20260809 15:46:24.066372 25540 leader_election.cc:288] T 0c29937a80724ddb9d07a27534a146f3 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:24.074327 23270 raft_consensus.cc:2454] T 953eea268e17491ca76f3a555860a512 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 1.
I20260809 15:46:24.081231 23392 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55332 (request call id 1074) took 1007 ms. Trace:
I20260809 15:46:24.081598 25472 leader_election.cc:288] T 7baa2492df0340c4a5ac241959ce1ac3 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:24.081908 23520 raft_consensus.cc:2454] T a3622f6e3e3641cea1960e3a31498bb7 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 1.
I20260809 15:46:24.082026 23518 raft_consensus.cc:2454] T e14d118b2eaa42aa8a7cd4025aba7128 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 2.
I20260809 15:46:24.085726 23404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30c1740ef6874df0a332fbd6fd4b03ce" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:24.087550 25253 leader_election.cc:288] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:24.077160 25488 raft_consensus.cc:2781] T e71a05ba1d4d42bd95c3f85f9313399e P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260809 15:46:24.074127 25531 leader_election.cc:288] T ae6d2c08a7a94f58ac8dd698e3fd456c P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:24.089715 23396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1a134ba33ea4076bbaac0ae460c17ba" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:24.090294 23270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "763e58d6ba644eea830a6c1664d39c9b" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:24.091776 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba56725e42f74a1baa4c04d9660727f9" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:24.090929 23528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "763e58d6ba644eea830a6c1664d39c9b" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:24.094671 23404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0721a4403a440f383318a3cea9d1442" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:24.087950 25559 leader_election.cc:288] T cef3e7277d7d4760824a3436ed4a847d P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:24.095732 23520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a88421dba80d4d3388aa5eb62e23853b" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
W20260809 15:46:24.097406 23227 service_pool.cc:137] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:53554 dropped due to backpressure. The service queue is full; it has 50 items.
I20260809 15:46:24.097843 23520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0721a4403a440f383318a3cea9d1442" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:24.098287 23528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "297c82ed906d4b48b184985eed3de601" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:24.089643 25488 raft_consensus.cc:482] T e71a05ba1d4d42bd95c3f85f9313399e P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Starting leader election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:46:24.099932 25488 raft_consensus.cc:3037] T e71a05ba1d4d42bd95c3f85f9313399e P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:24.100564 23286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3296ef50be254fc29cfc77d22a8f7a23" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:24.101998 23396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a51c0baaa6e432997c9d3552af4f254" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:24.102962 23270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "538a228ddfb54bc6a36528d5ff07597b" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
W20260809 15:46:24.098951 24506 debug-util.cc:398] Leaking SignalData structure 0x7b08002e1740 after lost signal to thread 23567
I20260809 15:46:24.106243 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b30f28311964a0b9149a50495e83d8f" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:24.110146 25569 leader_election.cc:288] T f7715d71dbac447b9351fc5eb36c5e80 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:24.111397 25490 leader_election.cc:288] T 0acc47ce21c3422698d1ef7277c3ed63 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:24.111682 23402 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55332 (request call id 1071) took 1129 ms. Trace:
I20260809 15:46:24.113112 23404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0da4d5283c8a46878bd39b85e5c71f66" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:24.113828 23404 raft_consensus.cc:2454] T 0da4d5283c8a46878bd39b85e5c71f66 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 1.
I20260809 15:46:24.100466 23528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99bdb93503db4de4a1cfb0161ea0c07c" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:24.112001 23402 rpcz_store.cc:276] 0809 15:46:22.982466 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:22.982582 (+ 116us) service_pool.cc:227] Handling call
0809 15:46:23.820540 (+837958us) spinlock_profiling.cc:243] Waited 9.46 ms on lock 0x7b58001d7c98. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e613cf11 00007f34e62c0851 00007f34e62c0702 00007f34ef1b0b09 00007f34ee0ec730 00007f34ee1b279b 00007f34ee1b2821 00007f34ee1621b0 00007f34ee161a94 00007f34e7849c04 00007f34e7849b92
0809 15:46:24.111535 (+290995us) spinlock_profiling.cc:243] Waited 226 ms on lock 0x7b54000a7dd4. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e92242ae 00007f34ee161c27 00007f34e7849c04 00007f34e7849b92 00007f34e7849ac1 00007f34e7849a3c 00007f34e7848cf2 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:46:24.111629 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":541796992}
I20260809 15:46:24.097134 25467 leader_election.cc:288] T a3622f6e3e3641cea1960e3a31498bb7 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:24.122083 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b30f28311964a0b9149a50495e83d8f" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:24.126246 23520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "213543cdbe814c89a17a963469922c1b" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
W20260809 15:46:24.109268 24506 debug-util.cc:398] Leaking SignalData structure 0x7b0800309560 after lost signal to thread 23568
I20260809 15:46:24.127846 23396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c29937a80724ddb9d07a27534a146f3" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:24.130136 23528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c29937a80724ddb9d07a27534a146f3" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:24.130298 23396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "297c82ed906d4b48b184985eed3de601" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:24.130895 23396 raft_consensus.cc:2454] T 297c82ed906d4b48b184985eed3de601 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 2.
I20260809 15:46:24.131058 25366 leader_election.cc:288] T e14d118b2eaa42aa8a7cd4025aba7128 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:24.099673 25369 leader_election.cc:288] T 0c48a3049897452383257aa757834fb5 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:24.097877 23227 service_pool.cc:140] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:53554 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53562 (request call id 1430)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53562 (request call id 1423)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53554 (request call id 1055)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53562 (request call id 1439)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53554 (request call id 1053)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53554 (request call id 1057)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53562 (request call id 1435)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53554 (request call id 1062)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53562 (request call id 1425)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53562 (request call id 1431)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53554 (request call id 1061)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53554 (request call id 1058)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53562 (request call id 1434)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55434 (request call id 71)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53554 (request call id 1054)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53554 (request call id 1052)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53554 (request call id 1059)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55434 (request call id 73)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55434 (request call id 74)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53554 (request call id 1056)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53554 (request call id 1051)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53554 (request call id 1060)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53562 (request call id 1407)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53562 (request call id 1408)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53562 (request call id 1410)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53562 (request call id 1411)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53562 (request call id 1414)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53562 (request call id 1415)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53562 (request call id 1420)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53562 (request call id 1422)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53562 (request call id 1424)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53562 (request call id 1427)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55434 (request call id 70)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55434 (request call id 72)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53554 (request call id 1041)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53554 (request call id 1042)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53554 (request call id 1043)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53554 (request call id 1044)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53554 (request call id 1045)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53554 (request call id 1046)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53554 (request call id 1047)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53554 (request call id 1048)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53554 (request call id 1049)
I20260809 15:46:24.140969 23404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a88421dba80d4d3388aa5eb62e23853b" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:24.153252 23404 raft_consensus.cc:2454] T a88421dba80d4d3388aa5eb62e23853b P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 1.
I20260809 15:46:24.157212 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7660dfa2ef694927b8eda4100a2bcd9b" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:24.083964 23392 rpcz_store.cc:276] 0809 15:46:23.073732 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:23.079209 (+ 5477us) service_pool.cc:227] Handling call
0809 15:46:23.892372 (+813163us) spinlock_profiling.cc:243] Waited 7.09 ms on lock 0x7b58001d7c98. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e613cf11 00007f34e62c0851 00007f34e62c0702 00007f34ef1b0b09 00007f34ee0ec730 00007f34ee1b279b 00007f34ee1b2821 00007f34ee1621b0 00007f34ee161a94 00007f34e7849c04 00007f34e7849b92
0809 15:46:24.081175 (+188803us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":16303872}
I20260809 15:46:24.159173 23280 consensus_queue.cc:237] T 0acc47ce21c3422698d1ef7277c3ed63 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } attrs { replace: true } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } }
I20260809 15:46:24.156477 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0721a4403a440f383318a3cea9d1442" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:24.160698 24410 leader_election.cc:302] T 0a51c0baaa6e432997c9d3552af4f254 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f; no voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:24.161590 25533 consensus_queue.cc:1035] T d3e94b7e50754a81b3d72a07476cb7b9 P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260809 15:46:24.162005 23276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a51c0baaa6e432997c9d3552af4f254" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:24.130499 25481 leader_election.cc:288] T df20671a7b874dbb98c0ef28371c1308 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:24.138499 23396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07fcabb569414bddbd39df6cedd24e64" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:24.204496 23404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a823d35f4f348d1b52ba9e6d80b00d2" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:24.206558 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1a134ba33ea4076bbaac0ae460c17ba" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:24.131253 23528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2db057fd10e149a9895dff6ef8c9f038" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:24.138039 23520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5c2e425a74b4e64ae70b5def750434c" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:24.203438 24451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7660dfa2ef694927b8eda4100a2bcd9b" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:24.212656 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4564dfc6d4094c1b9240b367ccecf3b1" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:24.215672 23404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f9d6565aa4c45a3b3f01210d0eb71f7" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
W20260809 15:46:24.210690 24506 debug-util.cc:398] Leaking SignalData structure 0x7b080026ffc0 after lost signal to thread 24513
I20260809 15:46:24.211117 25496 raft_consensus.cc:2726] T 0a51c0baaa6e432997c9d3552af4f254 P 61e69695258f48baa984c9d395e8ce3f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 15:46:24.221171 23520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3e94b7e50754a81b3d72a07476cb7b9" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:24.223007 23520 raft_consensus.cc:2454] T d3e94b7e50754a81b3d72a07476cb7b9 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 2.
I20260809 15:46:24.223853 23520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cef3e7277d7d4760824a3436ed4a847d" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:24.224368 23400 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52856 (request call id 41) took 1251 ms. Trace:
I20260809 15:46:24.225081 23396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2d4f749fcfd4fb3bfb8115af3471598" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:24.225244 23394 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52856 (request call id 40) took 1261 ms. Trace:
I20260809 15:46:24.226857 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae6d2c08a7a94f58ac8dd698e3fd456c" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:24.224695 23400 rpcz_store.cc:276] 0809 15:46:22.973294 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:22.973553 (+ 259us) service_pool.cc:227] Handling call
0809 15:46:22.973791 (+ 238us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:23.204067 (+230276us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:24.224146 (+1020079us) spinlock_profiling.cc:243] Waited 224 ms on lock 0x7b540031b0d4. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e9220f51 00007f34e921f99e 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772 00007f34e764a94c 00007f34e7649d86
0809 15:46:24.224296 (+ 150us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":514194688}
I20260809 15:46:24.227432 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae6d2c08a7a94f58ac8dd698e3fd456c" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:24.212544 23392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99bdb93503db4de4a1cfb0161ea0c07c" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
W20260809 15:46:24.220616 24506 debug-util.cc:398] Leaking SignalData structure 0x7b08002267c0 after lost signal to thread 24516
I20260809 15:46:24.225919 23394 rpcz_store.cc:276] 0809 15:46:22.964107 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:22.964261 (+ 154us) service_pool.cc:227] Handling call
0809 15:46:22.964520 (+ 259us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:23.223095 (+258575us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:24.225039 (+1001944us) spinlock_profiling.cc:243] Waited 258 ms on lock 0x7b54000023d4. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e9220f51 00007f34e921f99e 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772 00007f34e764a94c 00007f34e7649d86
0809 15:46:24.225185 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":592684416}
I20260809 15:46:24.235402 23404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3e94b7e50754a81b3d72a07476cb7b9" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:24.235899 23394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e71a05ba1d4d42bd95c3f85f9313399e" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:24.236027 23404 raft_consensus.cc:2454] T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 2.
I20260809 15:46:24.231890 23400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c29937a80724ddb9d07a27534a146f3" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:24.227185 24451 raft_consensus.cc:2454] T 7660dfa2ef694927b8eda4100a2bcd9b P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 1.
I20260809 15:46:24.232132 23353 leader_election.cc:302] T 0da4d5283c8a46878bd39b85e5c71f66 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters: 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:24.238847 23353 leader_election.cc:302] T 00a1881267a14ba8b78c81040f17a014 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:46:24.239599 23394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2db057fd10e149a9895dff6ef8c9f038" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:24.239771 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3924b648461e463b9d7925e5671d68ae" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:24.240267 23394 raft_consensus.cc:2454] T 2db057fd10e149a9895dff6ef8c9f038 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 1.
I20260809 15:46:24.234495 24453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5712f6b6fffc4cc5a8ddfbe63e4097df" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:24.241539 23396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3be1e356b254d22b995be32f1c12f6f" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:24.222095 25462 leader_election.cc:288] T 05ad7d8fdca64e3da527280ca5be9280 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 2 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
W20260809 15:46:24.219898 24410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 59297.3us
I20260809 15:46:24.242292 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "538a228ddfb54bc6a36528d5ff07597b" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:24.252887 23394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7660dfa2ef694927b8eda4100a2bcd9b" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:24.259099 23396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00a1881267a14ba8b78c81040f17a014" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:24.262336 23402 raft_consensus.cc:2454] T 538a228ddfb54bc6a36528d5ff07597b P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 2.
I20260809 15:46:24.263109 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b30f28311964a0b9149a50495e83d8f" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:24.244722 23228 leader_election.cc:302] T 73f6f44c259547f1b05a3a434616fa54 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:46:24.262092 23394 raft_consensus.cc:2454] T 7660dfa2ef694927b8eda4100a2bcd9b P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 1.
I20260809 15:46:24.265363 23394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0721a4403a440f383318a3cea9d1442" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:24.265756 23228 leader_election.cc:302] T cef3e7277d7d4760824a3436ed4a847d P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:46:24.263922 25593 consensus_queue.cc:1046] T c0741ab44d99439abc3aa3bdd27ee160 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.030s
I20260809 15:46:24.266921 23392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3aa6a9a52804df0ba9b0ca3c82f38c3" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:24.220839 25527 raft_consensus.cc:504] T 6ae14a74a26f434db08c652e097385c8 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:24.267642 23400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30c1740ef6874df0a332fbd6fd4b03ce" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:24.266080 23394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0acc47ce21c3422698d1ef7277c3ed63" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:24.268179 23404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07fcabb569414bddbd39df6cedd24e64" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:24.269734 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7baa2492df0340c4a5ac241959ce1ac3" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:24.243844 25500 raft_consensus.cc:2781] T 00a1881267a14ba8b78c81040f17a014 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260809 15:46:24.240903 25555 raft_consensus.cc:504] T 563daf42d7784db195ea99c3cb21b370 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:24.314145 25526 raft_consensus.cc:482] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
I20260809 15:46:24.314929 23392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1a134ba33ea4076bbaac0ae460c17ba" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:24.315128 24451 raft_consensus.cc:1226] T cee02fff2c9c48a19f43eb358a89e7b0 P 61e69695258f48baa984c9d395e8ce3f [term 2 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 1. Current term is 2. Ops: []
I20260809 15:46:24.241406 23395 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55332 (request call id 1065) took 1309 ms. Trace:
I20260809 15:46:24.315636 25460 raft_consensus.cc:504] T 998c6cee72cf40228c6dba064c0db2da P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:24.316161 23396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bcac4b24e7a4448abdc934bba5a624e" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:24.298796 25573 raft_consensus.cc:2781] T cef3e7277d7d4760824a3436ed4a847d P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260809 15:46:24.267134 25369 raft_consensus.cc:3032] T c0741ab44d99439abc3aa3bdd27ee160 P 65c36d164b0540f099d592f38307ee6c [term 1 LEADER]: Stepping down as leader of term 1
I20260809 15:46:24.241303 25529 raft_consensus.cc:2781] T 0da4d5283c8a46878bd39b85e5c71f66 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260809 15:46:24.317989 23394 raft_consensus.cc:2454] T 0acc47ce21c3422698d1ef7277c3ed63 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 1.
I20260809 15:46:24.268379 23228 leader_election.cc:302] T 297c82ed906d4b48b184985eed3de601 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:46:24.330711 23400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c29937a80724ddb9d07a27534a146f3" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:24.269754 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73f6f44c259547f1b05a3a434616fa54" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:24.321497 24455 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3aa6a9a52804df0ba9b0ca3c82f38c3" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:24.338325 23394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "563daf42d7784db195ea99c3cb21b370" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:24.317037 23528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cef3e7277d7d4760824a3436ed4a847d" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:24.299083 25465 raft_consensus.cc:482] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
I20260809 15:46:24.322480 25527 leader_election.cc:288] T 6ae14a74a26f434db08c652e097385c8 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 2 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:24.328727 23345 leader_election.cc:302] T 7660dfa2ef694927b8eda4100a2bcd9b P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:46:24.330199 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7715d71dbac447b9351fc5eb36c5e80" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:24.334275 23400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3924b648461e463b9d7925e5671d68ae" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:24.332455 23472 leader_election.cc:302] T d3e94b7e50754a81b3d72a07476cb7b9 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters: 61e69695258f48baa984c9d395e8ce3f
W20260809 15:46:24.321730 25457 consensus_meta.cc:287] T 30c1740ef6874df0a332fbd6fd4b03ce P c0f38c1908414566aa9115bb2cbdd85d: Time spent flushing consensus metadata: real 0.514s user 0.014s sys 0.006s
I20260809 15:46:24.339175 23394 raft_consensus.cc:2379] T 563daf42d7784db195ea99c3cb21b370 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 61e69695258f48baa984c9d395e8ce3f in current term 2: Already voted for candidate c0f38c1908414566aa9115bb2cbdd85d in this term.
W20260809 15:46:24.339437 23353 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 97628.2us
W20260809 15:46:24.338721 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 70401.7us
I20260809 15:46:24.318264 23276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b30f28311964a0b9149a50495e83d8f" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
W20260809 15:46:24.358234 23285 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55434 (request call id 63) took 1192 ms (client timeout 1511 ms). Trace:
I20260809 15:46:24.343086 23283 raft_consensus.cc:2379] T 998c6cee72cf40228c6dba064c0db2da P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d in current term 4: Already voted for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in this term.
I20260809 15:46:24.437517 23274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05ad7d8fdca64e3da527280ca5be9280" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:24.439333 25607 consensus_queue.cc:1035] T 907bacfa79d04a55a68c8dbe60056462 P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.006s
W20260809 15:46:24.843773 23274 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53562 (request call id 1409) took 1575 ms (client timeout 2023 ms). Trace:
I20260809 15:46:24.844347 23283 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53562 (request call id 1433) took 1209 ms. Trace:
W20260809 15:46:24.844101 23274 rpcz_store.cc:269] 0809 15:46:23.267983 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:24.437243 (+1169260us) service_pool.cc:227] Handling call
0809 15:46:24.843703 (+406460us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:46:24.844707 23283 rpcz_store.cc:276] 0809 15:46:23.634786 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:24.074428 (+439642us) service_pool.cc:227] Handling call
0809 15:46:24.844182 (+769754us) spinlock_profiling.cc:243] Waited 260 ms on lock 0x7b54000415d4. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e92242ae 00007f34ee161c27 00007f34e7849c04 00007f34e7849b92 00007f34e7849ac1 00007f34e7849a3c 00007f34e7848cf2 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:46:24.844296 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":598461824}
I20260809 15:46:24.845319 23274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30c1740ef6874df0a332fbd6fd4b03ce" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:24.443990 25610 consensus_queue.cc:1035] T d3e94b7e50754a81b3d72a07476cb7b9 P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.002s
I20260809 15:46:24.846328 23274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba56725e42f74a1baa4c04d9660727f9" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:24.346778 23392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e17dcb5efe45b8b6a8c42da03b4819" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:24.848181 23392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e71a05ba1d4d42bd95c3f85f9313399e" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:24.849208 23392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e17dcb5efe45b8b6a8c42da03b4819" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:24.850211 23392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a823d35f4f348d1b52ba9e6d80b00d2" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:24.348169 23394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df20671a7b874dbb98c0ef28371c1308" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:24.851714 23117 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:56222 (request call id 236) took 2244 ms. Trace:
I20260809 15:46:24.852135 23394 raft_consensus.cc:2454] T df20671a7b874dbb98c0ef28371c1308 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 1.
I20260809 15:46:24.852077 23117 rpcz_store.cc:276] 0809 15:46:22.607548 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:22.607696 (+ 148us) service_pool.cc:227] Handling call
0809 15:46:24.851632 (+2243936us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1033468,"reported_tablets":1}
I20260809 15:46:24.852829 23394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad8a3b25d7b49e5a2142d5709edd0c7" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:24.853791 23394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ced718c1a5c4da79cb8c1ebd1435b87" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:24.854589 23394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1a134ba33ea4076bbaac0ae460c17ba" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:24.334149 23395 rpcz_store.cc:276] 0809 15:46:22.929387 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:22.929802 (+ 415us) service_pool.cc:227] Handling call
0809 15:46:22.930071 (+ 269us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:23.050243 (+120172us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:46:23.050277 (+ 34us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 15:46:23.050319 (+ 42us) raft_consensus.cc:1631] Marking committed up to 4
0809 15:46:23.050349 (+ 30us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:24.228391 (+1178042us) spinlock_profiling.cc:243] Waited 120 ms on lock 0x7b54003197d4. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e9220f51 00007f34e921f99e 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772 00007f34e764a94c 00007f34e7649d86
0809 15:46:24.231683 (+ 3292us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:24.237753 (+ 6070us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":275214208}
I20260809 15:46:24.349781 23396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3622f6e3e3641cea1960e3a31498bb7" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:24.445343 25609 raft_consensus.cc:482] T 4bcac4b24e7a4448abdc934bba5a624e P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:46:24.349283 23400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "953eea268e17491ca76f3a555860a512" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:24.858217 25609 raft_consensus.cc:504] T 4bcac4b24e7a4448abdc934bba5a624e P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:24.858784 23400 raft_consensus.cc:2454] T 953eea268e17491ca76f3a555860a512 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 1.
I20260809 15:46:24.859522 23400 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "07fcabb569414bddbd39df6cedd24e64"
dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
mode: GRACEFUL
new_leader_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
from {username='slave'} at 127.0.0.1:51280
I20260809 15:46:24.860023 23400 raft_consensus.cc:595] T 07fcabb569414bddbd39df6cedd24e64 P c0f38c1908414566aa9115bb2cbdd85d [term 1 LEADER]: Received request to transfer leadership to TS 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:24.851188 23392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5c2e425a74b4e64ae70b5def750434c" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:24.860277 25609 leader_election.cc:288] T 4bcac4b24e7a4448abdc934bba5a624e P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:24.860703 23392 raft_consensus.cc:2454] T c5c2e425a74b4e64ae70b5def750434c P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 1.
I20260809 15:46:24.340766 23404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0da4d5283c8a46878bd39b85e5c71f66" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:24.346834 23472 leader_election.cc:302] T 538a228ddfb54bc6a36528d5ff07597b P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:46:24.419129 23397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9e7cadcd24d45fcbfa84209225e8f2d" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:24.421561 23518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d880de657e544468c82157769622a26" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:24.865458 23518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad8a3b25d7b49e5a2142d5709edd0c7" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:24.865938 23518 raft_consensus.cc:2454] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 1.
I20260809 15:46:24.866683 23518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e17dcb5efe45b8b6a8c42da03b4819" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:24.867157 23518 raft_consensus.cc:2454] T 28e17dcb5efe45b8b6a8c42da03b4819 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 1.
I20260809 15:46:24.867990 23518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "953eea268e17491ca76f3a555860a512" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:24.868880 23518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0acc47ce21c3422698d1ef7277c3ed63" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:24.351801 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a823d35f4f348d1b52ba9e6d80b00d2" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:24.360147 23528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c48a3049897452383257aa757834fb5" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:24.870657 23528 raft_consensus.cc:2454] T 0c48a3049897452383257aa757834fb5 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 1.
I20260809 15:46:24.871091 23518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee02fff2c9c48a19f43eb358a89e7b0" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:24.871582 23528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05ad7d8fdca64e3da527280ca5be9280" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
W20260809 15:46:24.443842 24410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 57095.4us
I20260809 15:46:24.417578 24451 raft_consensus.cc:1226] T c0741ab44d99439abc3aa3bdd27ee160 P 61e69695258f48baa984c9d395e8ce3f [term 2 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 1. Current term is 2. Ops: []
I20260809 15:46:24.416574 24452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a51c0baaa6e432997c9d3552af4f254" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:24.368350 24455 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "563daf42d7784db195ea99c3cb21b370" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f"
I20260809 15:46:24.356137 24453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c48a3049897452383257aa757834fb5" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:24.322530 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "294d4bcc1e7a477cb56455581277a60f" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:24.350977 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e14d118b2eaa42aa8a7cd4025aba7128" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:24.349259 25366 raft_consensus.cc:2781] T d3e94b7e50754a81b3d72a07476cb7b9 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260809 15:46:24.875234 24410 leader_election.cc:302] T 7660dfa2ef694927b8eda4100a2bcd9b P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:46:24.884603 25366 raft_consensus.cc:482] T d3e94b7e50754a81b3d72a07476cb7b9 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:46:24.885118 25366 raft_consensus.cc:3037] T d3e94b7e50754a81b3d72a07476cb7b9 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:24.877476 24452 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38724 (request call id 356) took 1044 ms. Trace:
I20260809 15:46:24.887964 24452 rpcz_store.cc:276] 0809 15:46:23.833399 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:23.835193 (+ 1794us) service_pool.cc:227] Handling call
0809 15:46:24.390348 (+555155us) spinlock_profiling.cc:243] Waited 2.87 ms on lock 0x7b5800005498. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e613cf11 00007f34e62c0851 00007f34e62c0702 00007f34ef1b0b09 00007f34ee0ec730 00007f34ee1b279b 00007f34ee1b2821 00007f34ee1621b0 00007f34ee161a94 00007f34e7849c04 00007f34e7849b92
0809 15:46:24.877354 (+487006us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6607104}
I20260809 15:46:24.889427 25609 raft_consensus.cc:2781] T 7660dfa2ef694927b8eda4100a2bcd9b P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260809 15:46:24.890003 25609 raft_consensus.cc:482] T 7660dfa2ef694927b8eda4100a2bcd9b P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Starting leader election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:24.333401 25369 raft_consensus.cc:729] T c0741ab44d99439abc3aa3bdd27ee160 P 65c36d164b0540f099d592f38307ee6c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:46:24.349464 25457 raft_consensus.cc:504] T 30c1740ef6874df0a332fbd6fd4b03ce P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:24.894265 25369 consensus_queue.cc:260] T c0741ab44d99439abc3aa3bdd27ee160 P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:24.897037 25457 leader_election.cc:288] T 30c1740ef6874df0a332fbd6fd4b03ce P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:24.368615 25460 leader_election.cc:288] T 998c6cee72cf40228c6dba064c0db2da P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:24.344403 25465 raft_consensus.cc:504] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:24.382458 25467 consensus_queue.cc:1046] T cee02fff2c9c48a19f43eb358a89e7b0 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.012s
I20260809 15:46:24.448885 25478 raft_consensus.cc:482] T d3be1e356b254d22b995be32f1c12f6f P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:46:24.908099 25478 raft_consensus.cc:504] T d3be1e356b254d22b995be32f1c12f6f P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:24.422691 25483 raft_consensus.cc:504] T 472816994d814cb198b69e5248697440 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:24.914188 25483 leader_election.cc:288] T 472816994d814cb198b69e5248697440 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 2 election: Requested vote from peers 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:24.327059 25488 raft_consensus.cc:504] T e71a05ba1d4d42bd95c3f85f9313399e P 61e69695258f48baa984c9d395e8ce3f [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:24.917032 25488 leader_election.cc:288] T e71a05ba1d4d42bd95c3f85f9313399e P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 2 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:24.495606 25490 raft_consensus.cc:482] T ae6d2c08a7a94f58ac8dd698e3fd456c P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:46:24.320317 25500 raft_consensus.cc:482] T 00a1881267a14ba8b78c81040f17a014 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting leader election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
I20260809 15:46:24.921443 25500 raft_consensus.cc:3037] T 00a1881267a14ba8b78c81040f17a014 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:24.920199 25616 consensus_queue.cc:1046] T c0741ab44d99439abc3aa3bdd27ee160 P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.002s
I20260809 15:46:24.387566 25506 raft_consensus.cc:504] T 16ca412a0dae4feebe85dd01bf1a9173 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:24.925843 25506 leader_election.cc:288] T 16ca412a0dae4feebe85dd01bf1a9173 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 2 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:24.341595 25519 raft_consensus.cc:2781] T 297c82ed906d4b48b184985eed3de601 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260809 15:46:24.331419 25526 raft_consensus.cc:504] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:24.930119 25526 leader_election.cc:288] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:24.339295 25529 raft_consensus.cc:482] T 0da4d5283c8a46878bd39b85e5c71f66 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
I20260809 15:46:24.931629 24463 raft_consensus.cc:1226] T e71a05ba1d4d42bd95c3f85f9313399e P 61e69695258f48baa984c9d395e8ce3f [term 2 FOLLOWER]: Rejecting Update request from peer c0f38c1908414566aa9115bb2cbdd85d for earlier term 1. Current term is 2. Ops: []
I20260809 15:46:24.369153 25532 leader_election.cc:288] T c5c2e425a74b4e64ae70b5def750434c P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:24.275429 25539 raft_consensus.cc:504] T 2424fd9bd7f14e508fd106cbfda786a4 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:24.935195 24448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472816994d814cb198b69e5248697440" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f"
I20260809 15:46:24.936187 25539 leader_election.cc:288] T 2424fd9bd7f14e508fd106cbfda786a4 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:24.359278 25555 leader_election.cc:288] T 563daf42d7784db195ea99c3cb21b370 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 2 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:24.349274 25559 raft_consensus.cc:2781] T 7660dfa2ef694927b8eda4100a2bcd9b P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260809 15:46:24.939363 25559 raft_consensus.cc:482] T 7660dfa2ef694927b8eda4100a2bcd9b P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:24.939963 25559 raft_consensus.cc:3037] T 7660dfa2ef694927b8eda4100a2bcd9b P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:24.940063 25465 leader_election.cc:288] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:24.940857 25478 leader_election.cc:288] T d3be1e356b254d22b995be32f1c12f6f P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:24.441217 25551 raft_consensus.cc:504] T f9e7cadcd24d45fcbfa84209225e8f2d P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:24.269892 25569 raft_consensus.cc:2781] T 73f6f44c259547f1b05a3a434616fa54 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260809 15:46:24.948758 25569 raft_consensus.cc:482] T 73f6f44c259547f1b05a3a434616fa54 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:24.949304 25569 raft_consensus.cc:3037] T 73f6f44c259547f1b05a3a434616fa54 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:24.949905 23397 raft_consensus.cc:2379] T f9e7cadcd24d45fcbfa84209225e8f2d P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 in current term 2: Already voted for candidate c0f38c1908414566aa9115bb2cbdd85d in this term.
I20260809 15:46:24.950340 25559 raft_consensus.cc:504] T 7660dfa2ef694927b8eda4100a2bcd9b P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:24.951709 23400 raft_consensus.cc:1226] T 7660dfa2ef694927b8eda4100a2bcd9b P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 1. Current term is 2. Ops: []
W20260809 15:46:24.951804 23397 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55332 (request call id 1070) took 1975 ms (client timeout 1516 ms). Trace:
W20260809 15:46:24.952150 23397 rpcz_store.cc:269] 0809 15:46:22.979210 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:22.981414 (+ 2204us) service_pool.cc:227] Handling call
0809 15:46:24.390536 (+1409122us) spinlock_profiling.cc:243] Waited 23.9 ms on lock 0x7b58001d7c98. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e613cf11 00007f34e62c0851 00007f34e62c0702 00007f34ef1b0b09 00007f34ee0ec730 00007f34ee1b279b 00007f34ee1b2821 00007f34ee1621b0 00007f34ee161a94 00007f34e7849c04 00007f34e7849b92
0809 15:46:24.951628 (+561092us) spinlock_profiling.cc:243] Waited 85.2 ms on lock 0x7b54000a91d4. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e92242ae 00007f34ee161c27 00007f34e7849c04 00007f34e7849b92 00007f34e7849ac1 00007f34e7849a3c 00007f34e7848cf2 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:46:24.951752 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":250810112}
I20260809 15:46:24.952711 23400 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55320 (request call id 970) took 1773 ms. Trace:
I20260809 15:46:24.953024 23400 rpcz_store.cc:276] 0809 15:46:23.179217 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:24.860798 (+1681581us) service_pool.cc:227] Handling call
0809 15:46:24.861059 (+ 261us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:24.952034 (+ 90975us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:24.952521 (+ 487us) spinlock_profiling.cc:243] Waited 90.5 ms on lock 0x7b540001d6d4. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e9220f51 00007f34e921f99e 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772 00007f34e764a94c 00007f34e7649d86
0809 15:46:24.952662 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":208109952}
I20260809 15:46:24.333999 25571 raft_consensus.cc:504] T e71a05ba1d4d42bd95c3f85f9313399e P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:24.957005 24463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7660dfa2ef694927b8eda4100a2bcd9b" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f"
I20260809 15:46:24.957881 25571 leader_election.cc:288] T e71a05ba1d4d42bd95c3f85f9313399e P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 2 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:24.333204 25573 raft_consensus.cc:482] T cef3e7277d7d4760824a3436ed4a847d P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:24.959332 25573 raft_consensus.cc:3037] T cef3e7277d7d4760824a3436ed4a847d P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:24.960988 25559 leader_election.cc:288] T 7660dfa2ef694927b8eda4100a2bcd9b P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 2 election: Requested vote from peers 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:24.485891 25593 raft_consensus.cc:482] T 213543cdbe814c89a17a963469922c1b P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:46:24.963626 25593 raft_consensus.cc:504] T 213543cdbe814c89a17a963469922c1b P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:24.966147 25593 leader_election.cc:288] T 213543cdbe814c89a17a963469922c1b P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:24.674175 23521 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56766 (request call id 1517) took 1743 ms. Trace:
I20260809 15:46:24.701437 25481 raft_consensus.cc:482] T 294d4bcc1e7a477cb56455581277a60f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
I20260809 15:46:24.967841 25481 raft_consensus.cc:504] T 294d4bcc1e7a477cb56455581277a60f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:24.970381 25481 leader_election.cc:288] T 294d4bcc1e7a477cb56455581277a60f P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:24.721150 23399 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52856 (request call id 47) took 1228 ms. Trace:
I20260809 15:46:24.735289 23228 leader_election.cc:302] T a88421dba80d4d3388aa5eb62e23853b P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters:
W20260809 15:46:24.972906 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 237691us
I20260809 15:46:24.973517 23228 leader_election.cc:302] T ba56725e42f74a1baa4c04d9660727f9 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:46:24.974560 23228 leader_election.cc:302] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:24.737582 23277 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53554 (request call id 1039) took 1495 ms. Trace:
I20260809 15:46:24.975391 23277 rpcz_store.cc:276] 0809 15:46:23.242126 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:23.243031 (+ 905us) service_pool.cc:227] Handling call
0809 15:46:23.243342 (+ 311us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:23.919168 (+675826us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:46:23.919198 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 15:46:23.919236 (+ 38us) raft_consensus.cc:1631] Marking committed up to 2
0809 15:46:23.919286 (+ 50us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:24.736972 (+817686us) spinlock_profiling.cc:243] Waited 675 ms on lock 0x7b54001388d4. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e9220f51 00007f34e921f99e 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772 00007f34e764a94c 00007f34e7649d86
0809 15:46:24.737004 (+ 32us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:24.737092 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1553585536}
I20260809 15:46:24.976862 23277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0da4d5283c8a46878bd39b85e5c71f66" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:24.977918 23277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0acc47ce21c3422698d1ef7277c3ed63" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:24.979486 25532 raft_consensus.cc:2781] T a88421dba80d4d3388aa5eb62e23853b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260809 15:46:24.980050 25532 raft_consensus.cc:482] T a88421dba80d4d3388aa5eb62e23853b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:24.980515 25532 raft_consensus.cc:3037] T a88421dba80d4d3388aa5eb62e23853b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:24.738616 23520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07fcabb569414bddbd39df6cedd24e64" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:24.983558 23520 raft_consensus.cc:2454] T 07fcabb569414bddbd39df6cedd24e64 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 1.
I20260809 15:46:24.984793 23520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0741ab44d99439abc3aa3bdd27ee160" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:24.748360 23273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73f6f44c259547f1b05a3a434616fa54" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:24.767086 23282 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53562 (request call id 1400) took 1523 ms. Trace:
I20260809 15:46:24.985868 23282 rpcz_store.cc:276] 0809 15:46:23.244015 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:23.244110 (+ 95us) service_pool.cc:227] Handling call
0809 15:46:23.244355 (+ 245us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:23.865411 (+621056us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:46:23.865439 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 15:46:23.865474 (+ 35us) raft_consensus.cc:1631] Marking committed up to 1
0809 15:46:23.865499 (+ 25us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:24.766901 (+901402us) spinlock_profiling.cc:243] Waited 621 ms on lock 0x7b540008aa54. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e9220f51 00007f34e921f99e 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772 00007f34e764a94c 00007f34e7649d86
0809 15:46:24.766941 (+ 40us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:24.767030 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1427935616}
I20260809 15:46:24.987057 23282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00a1881267a14ba8b78c81040f17a014" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:24.987843 23282 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53562 (request call id 1434) took 1352 ms. Trace:
I20260809 15:46:24.988129 23282 rpcz_store.cc:276] 0809 15:46:23.635334 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:24.986837 (+1351503us) service_pool.cc:227] Handling call
0809 15:46:24.987792 (+ 955us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:46:24.988832 23282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "563daf42d7784db195ea99c3cb21b370" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:24.989462 23282 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55434 (request call id 71) took 1166 ms. Trace:
I20260809 15:46:24.989777 23282 rpcz_store.cc:276] 0809 15:46:23.823515 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:24.988634 (+1165119us) service_pool.cc:227] Handling call
0809 15:46:24.989422 (+ 788us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:46:24.990507 23282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16ca412a0dae4feebe85dd01bf1a9173" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:24.772771 23522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba56725e42f74a1baa4c04d9660727f9" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:24.773289 23280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0721a4403a440f383318a3cea9d1442" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:24.786229 23270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00a1881267a14ba8b78c81040f17a014" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:24.788072 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee02fff2c9c48a19f43eb358a89e7b0" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:24.991724 23522 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56766 (request call id 1537) took 1074 ms. Trace:
I20260809 15:46:24.992100 23522 rpcz_store.cc:276] 0809 15:46:23.917032 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:23.917167 (+ 135us) service_pool.cc:227] Handling call
0809 15:46:24.772625 (+855458us) spinlock_profiling.cc:243] Waited 1.7 ms on lock 0x7b5800004298. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e613cf11 00007f34e62c0851 00007f34e62c0702 00007f34ef1b0b09 00007f34ee0ec730 00007f34ee1b279b 00007f34ee1b2821 00007f34ee1621b0 00007f34ee161a94 00007f34e7849c04 00007f34e7849b92
0809 15:46:24.991647 (+219022us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3916672}
I20260809 15:46:24.992563 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df20671a7b874dbb98c0ef28371c1308" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:24.993002 23522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "763e58d6ba644eea830a6c1664d39c9b" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:24.993536 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7660dfa2ef694927b8eda4100a2bcd9b" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:24.994024 23522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0741ab44d99439abc3aa3bdd27ee160" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:24.994499 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3aa6a9a52804df0ba9b0ca3c82f38c3" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:24.995582 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bcac4b24e7a4448abdc934bba5a624e" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:24.817467 23529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7baa2492df0340c4a5ac241959ce1ac3" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:24.820118 23116 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:37326 (request call id 141) took 2024 ms. Trace:
I20260809 15:46:24.820353 25590 raft_consensus.cc:482] T 05ad7d8fdca64e3da527280ca5be9280 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
I20260809 15:46:24.949422 25551 leader_election.cc:288] T f9e7cadcd24d45fcbfa84209225e8f2d P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 2 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:24.997339 23529 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56766 (request call id 1532) took 1130 ms. Trace:
I20260809 15:46:24.998435 23529 rpcz_store.cc:276] 0809 15:46:23.867024 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:23.867507 (+ 483us) service_pool.cc:227] Handling call
0809 15:46:24.817321 (+949814us) spinlock_profiling.cc:243] Waited 11.7 ms on lock 0x7b5800004298. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e613cf11 00007f34e62c0851 00007f34e62c0702 00007f34ef1b0b09 00007f34ee0ec730 00007f34ee1b279b 00007f34ee1b2821 00007f34ee1621b0 00007f34ee161a94 00007f34e7849c04 00007f34e7849b92
0809 15:46:24.997285 (+179964us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":27020032}
I20260809 15:46:24.822151 23227 service_pool.cc:140] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:53554 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53562 (request call id 1435)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53554 (request call id 1062)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53562 (request call id 1425)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53562 (request call id 1431)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53554 (request call id 1061)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53554 (request call id 1071)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53554 (request call id 1058)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53562 (request call id 1434)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55434 (request call id 71)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53554 (request call id 1054)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53554 (request call id 1052)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53554 (request call id 1059)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53554 (request call id 1070)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55434 (request call id 73)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53554 (request call id 1067)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55434 (request call id 74)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53554 (request call id 1056)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53554 (request call id 1051)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53554 (request call id 1073)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53554 (request call id 1069)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53554 (request call id 1063)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53554 (request call id 1060)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53554 (request call id 1064)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53554 (request call id 1066)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53554 (request call id 1074)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53562 (request call id 1440)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53562 (request call id 1407)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53562 (request call id 1408)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53562 (request call id 1410)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53562 (request call id 1411)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53562 (request call id 1414)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53562 (request call id 1415)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53562 (request call id 1420)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53562 (request call id 1422)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53562 (request call id 1424)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53562 (request call id 1427)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55434 (request call id 70)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55434 (request call id 72)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53554 (request call id 1041)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53554 (request call id 1042)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53554 (request call id 1043)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53554 (request call id 1044)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53554 (request call id 1045)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53554 (request call id 1046)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53554 (request call id 1047)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53554 (request call id 1048)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53554 (request call id 1049)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53554 (request call id 1065)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53554 (request call id 1068)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53554 (request call id 1072)
I20260809 15:46:24.826399 23286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "563daf42d7784db195ea99c3cb21b370" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:24.828069 23276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "907bacfa79d04a55a68c8dbe60056462" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:24.829599 23519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472816994d814cb198b69e5248697440" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
W20260809 15:46:24.833969 23223 consensus_peers.cc:487] T 0acc47ce21c3422698d1ef7277c3ed63 P 67b3aeadfa7a432abe8af4d292e04ab9 -> Peer 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689): Couldn't send request to peer 61e69695258f48baa984c9d395e8ce3f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0acc47ce21c3422698d1ef7277c3ed63. This is attempt 1: this message will repeat every 5th retry.
W20260809 15:46:25.000953 23223 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.68:34689, user_credentials={real_user=slave}} blocked reactor thread for 167145us
W20260809 15:46:24.842018 23285 rpcz_store.cc:269] 0809 15:46:23.158884 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:23.159011 (+ 127us) service_pool.cc:227] Handling call
0809 15:46:24.350821 (+1191810us) spinlock_profiling.cc:243] Waited 650 ms on lock 0x7b54003a9dd4. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e92242ae 00007f34ee161c27 00007f34e7849c04 00007f34e7849b92 00007f34e7849ac1 00007f34e7849a3c 00007f34e7848cf2 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:46:24.350912 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1494271232}
I20260809 15:46:24.348035 23353 leader_election.cc:302] T 538a228ddfb54bc6a36528d5ff07597b P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:46:24.845695 23283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7baa2492df0340c4a5ac241959ce1ac3" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:25.002031 23285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e17dcb5efe45b8b6a8c42da03b4819" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:25.002625 23285 raft_consensus.cc:2454] T 28e17dcb5efe45b8b6a8c42da03b4819 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 1.
W20260809 15:46:25.002389 23353 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 654371us
I20260809 15:46:25.002753 23283 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53562 (request call id 1425) took 1480 ms. Trace:
I20260809 15:46:25.003410 25526 raft_consensus.cc:2726] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 15:46:25.003566 25481 raft_consensus.cc:482] T c0741ab44d99439abc3aa3bdd27ee160 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:25.004027 23353 leader_election.cc:302] T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters: 61e69695258f48baa984c9d395e8ce3f
I20260809 15:46:25.004177 25481 raft_consensus.cc:504] T c0741ab44d99439abc3aa3bdd27ee160 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:25.005091 23353 leader_election.cc:302] T a3622f6e3e3641cea1960e3a31498bb7 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:46:24.855469 23394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2424fd9bd7f14e508fd106cbfda786a4" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:25.006006 23394 raft_consensus.cc:2379] T 2424fd9bd7f14e508fd106cbfda786a4 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c in current term 3: Already voted for candidate c0f38c1908414566aa9115bb2cbdd85d in this term.
I20260809 15:46:25.006325 25481 leader_election.cc:288] T c0741ab44d99439abc3aa3bdd27ee160 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:25.006913 25555 raft_consensus.cc:2781] T 538a228ddfb54bc6a36528d5ff07597b P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260809 15:46:25.007153 23228 leader_election.cc:302] T 2db057fd10e149a9895dff6ef8c9f038 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:46:25.007329 25555 raft_consensus.cc:482] T 538a228ddfb54bc6a36528d5ff07597b P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting leader election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
I20260809 15:46:25.007642 25551 raft_consensus.cc:2781] T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260809 15:46:25.007730 25555 raft_consensus.cc:3037] T 538a228ddfb54bc6a36528d5ff07597b P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:24.846932 23274 raft_consensus.cc:2454] T ba56725e42f74a1baa4c04d9660727f9 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 1.
I20260809 15:46:24.343048 23402 raft_consensus.cc:2454] T f7715d71dbac447b9351fc5eb36c5e80 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 1.
I20260809 15:46:24.857826 23396 raft_consensus.cc:2454] T a3622f6e3e3641cea1960e3a31498bb7 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 1.
W20260809 15:46:24.863476 23472 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 516662us
I20260809 15:46:24.863634 25583 raft_consensus.cc:2781] T 538a228ddfb54bc6a36528d5ff07597b P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260809 15:46:25.010102 25583 raft_consensus.cc:482] T 538a228ddfb54bc6a36528d5ff07597b P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Starting leader election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
I20260809 15:46:25.010684 25583 raft_consensus.cc:3037] T 538a228ddfb54bc6a36528d5ff07597b P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:25.010819 23283 rpcz_store.cc:276] 0809 15:46:23.522159 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:24.845484 (+1323325us) service_pool.cc:227] Handling call
0809 15:46:25.002703 (+157219us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:46:24.872639 23518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ced718c1a5c4da79cb8c1ebd1435b87" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:25.011680 23283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4564dfc6d4094c1b9240b367ccecf3b1" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:25.012283 23283 raft_consensus.cc:2454] T 4564dfc6d4094c1b9240b367ccecf3b1 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 1.
I20260809 15:46:25.012727 23283 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53554 (request call id 1059) took 1021 ms. Trace:
I20260809 15:46:25.013010 23283 rpcz_store.cc:276] 0809 15:46:23.991065 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:25.011466 (+1020401us) service_pool.cc:227] Handling call
0809 15:46:25.012681 (+ 1215us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:46:25.013770 23283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7baa2492df0340c4a5ac241959ce1ac3" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:24.903579 25366 raft_consensus.cc:504] T d3e94b7e50754a81b3d72a07476cb7b9 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:25.014355 25483 raft_consensus.cc:2781] T a3622f6e3e3641cea1960e3a31498bb7 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260809 15:46:24.870121 23527 raft_consensus.cc:2454] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 1.
I20260809 15:46:24.930794 23393 raft_consensus.cc:2379] T 472816994d814cb198b69e5248697440 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 61e69695258f48baa984c9d395e8ce3f in current term 2: Already voted for candidate c0f38c1908414566aa9115bb2cbdd85d in this term.
I20260809 15:46:25.003558 23276 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53562 (request call id 1435) took 1368 ms. Trace:
I20260809 15:46:24.927896 25519 raft_consensus.cc:482] T 297c82ed906d4b48b184985eed3de601 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:46:25.015679 23276 rpcz_store.cc:276] 0809 15:46:23.635589 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:24.827819 (+1192230us) service_pool.cc:227] Handling call
0809 15:46:25.003504 (+175685us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:46:24.997448 23116 rpcz_store.cc:276] 0809 15:46:22.809503 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:22.811477 (+ 1974us) service_pool.cc:227] Handling call
0809 15:46:24.820035 (+2008558us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":1}
I20260809 15:46:24.918661 25490 raft_consensus.cc:504] T ae6d2c08a7a94f58ac8dd698e3fd456c P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:25.020830 25460 raft_consensus.cc:2781] T 2db057fd10e149a9895dff6ef8c9f038 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260809 15:46:25.021212 25460 raft_consensus.cc:482] T 2db057fd10e149a9895dff6ef8c9f038 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:25.021600 25460 raft_consensus.cc:3037] T 2db057fd10e149a9895dff6ef8c9f038 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:24.995072 23522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16ca412a0dae4feebe85dd01bf1a9173" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:24.962646 25559 consensus_queue.cc:1046] T e71a05ba1d4d42bd95c3f85f9313399e P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260809 15:46:24.812678 23517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7715d71dbac447b9351fc5eb36c5e80" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:24.966698 23521 rpcz_store.cc:276] 0809 15:46:22.930122 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:22.930680 (+ 558us) service_pool.cc:227] Handling call
0809 15:46:22.931081 (+ 401us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:24.062762 (+1131681us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:46:24.062793 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 15:46:24.062832 (+ 39us) raft_consensus.cc:1631] Marking committed up to 4
0809 15:46:24.062859 (+ 27us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:24.673578 (+610719us) spinlock_profiling.cc:243] Waited 1.13 s on lock 0x7b54001fc054. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e9220f51 00007f34e921f99e 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772 00007f34e764a94c 00007f34e7649d86
0809 15:46:24.673629 (+ 51us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:24.673778 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2602294528}
I20260809 15:46:25.023972 25559 raft_consensus.cc:3032] T e71a05ba1d4d42bd95c3f85f9313399e P c0f38c1908414566aa9115bb2cbdd85d [term 1 LEADER]: Stepping down as leader of term 1
I20260809 15:46:25.024380 23517 raft_consensus.cc:1226] T d3e94b7e50754a81b3d72a07476cb7b9 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Rejecting Update request from peer 61e69695258f48baa984c9d395e8ce3f for earlier term 2. Current term is 3. Ops: [2.5-2.5]
I20260809 15:46:24.991658 23280 raft_consensus.cc:2454] T d0721a4403a440f383318a3cea9d1442 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 1.
I20260809 15:46:24.970973 23399 rpcz_store.cc:276] 0809 15:46:23.492998 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:24.034821 (+541823us) service_pool.cc:227] Handling call
0809 15:46:24.720729 (+685908us) spinlock_profiling.cc:243] Waited 1.89 ms on lock 0x7b540013eb28. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34ef1978b1 00007f34ef199434 00007f34ee162437 00007f34ee161bc8 00007f34e7849c04 00007f34e7849b92 00007f34e7849ac1 00007f34e7849a3c 00007f34e7848cf2 00007f34e764a94c 00007f34e7649d86
0809 15:46:24.721075 (+ 346us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4349312}
I20260809 15:46:25.024353 25559 raft_consensus.cc:729] T e71a05ba1d4d42bd95c3f85f9313399e P c0f38c1908414566aa9115bb2cbdd85d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:46:24.996619 23270 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53554 (request call id 1055) took 1019 ms. Trace:
W20260809 15:46:25.020270 23393 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52856 (request call id 46) took 1527 ms (client timeout 1775 ms). Trace:
I20260809 15:46:25.025743 23270 rpcz_store.cc:276] 0809 15:46:23.977174 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:24.785976 (+808802us) service_pool.cc:227] Handling call
0809 15:46:24.996570 (+210594us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:46:25.025861 25559 consensus_queue.cc:260] T e71a05ba1d4d42bd95c3f85f9313399e P c0f38c1908414566aa9115bb2cbdd85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:25.027216 23270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "294d4bcc1e7a477cb56455581277a60f" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:25.024900 23280 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53554 (request call id 1057) took 1047 ms. Trace:
I20260809 15:46:25.027766 23280 rpcz_store.cc:276] 0809 15:46:23.981165 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:24.772605 (+791440us) service_pool.cc:227] Handling call
0809 15:46:25.024851 (+252246us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:46:25.000435 23286 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53562 (request call id 1430) took 1366 ms. Trace:
I20260809 15:46:25.008010 23228 leader_election.cc:302] T d3be1e356b254d22b995be32f1c12f6f P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:24.997642 25590 raft_consensus.cc:504] T 05ad7d8fdca64e3da527280ca5be9280 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:25.031050 23283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1a134ba33ea4076bbaac0ae460c17ba" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:25.013862 23518 raft_consensus.cc:1226] T 16ca412a0dae4feebe85dd01bf1a9173 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Rejecting Update request from peer 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 1. Current term is 2. Ops: []
I20260809 15:46:24.931636 25529 raft_consensus.cc:3037] T 0da4d5283c8a46878bd39b85e5c71f66 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:25.015997 25366 leader_election.cc:288] T d3e94b7e50754a81b3d72a07476cb7b9 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:25.017333 25571 raft_consensus.cc:2781] T ba56725e42f74a1baa4c04d9660727f9 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260809 15:46:25.018303 24448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5712f6b6fffc4cc5a8ddfbe63e4097df" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:25.029259 23286 rpcz_store.cc:276] 0809 15:46:23.634077 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:24.826024 (+1191947us) service_pool.cc:227] Handling call
0809 15:46:25.000340 (+174316us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:46:25.018666 24463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e4ffcf83c234ae7910ddef3ec0b4bda" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:24.958103 25569 raft_consensus.cc:504] T 73f6f44c259547f1b05a3a434616fa54 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:25.032577 25559 raft_consensus.cc:3037] T e71a05ba1d4d42bd95c3f85f9313399e P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:25.019325 23276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee02fff2c9c48a19f43eb358a89e7b0" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:25.033610 23276 raft_consensus.cc:3037] T cee02fff2c9c48a19f43eb358a89e7b0 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:25.019695 24452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e71a05ba1d4d42bd95c3f85f9313399e" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f"
I20260809 15:46:25.018603 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "294d4bcc1e7a477cb56455581277a60f" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:25.017246 23392 raft_consensus.cc:1226] T 472816994d814cb198b69e5248697440 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 1. Current term is 2. Ops: []
I20260809 15:46:24.997069 23526 raft_consensus.cc:2454] T 4bcac4b24e7a4448abdc934bba5a624e P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 1.
I20260809 15:46:24.979080 23277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c29937a80724ddb9d07a27534a146f3" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:25.028542 23280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0741ab44d99439abc3aa3bdd27ee160" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:25.010891 23472 leader_election.cc:302] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:25.027789 23270 raft_consensus.cc:2454] T 294d4bcc1e7a477cb56455581277a60f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 2.
W20260809 15:46:25.026646 23393 rpcz_store.cc:269] 0809 15:46:23.492624 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:24.027784 (+535160us) service_pool.cc:227] Handling call
0809 15:46:25.016545 (+988761us) spinlock_profiling.cc:243] Waited 865 ms on lock 0x7b5400140ad4. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e92242ae 00007f34ee161c27 00007f34e7849c04 00007f34e7849b92 00007f34e7849ac1 00007f34e7849a3c 00007f34e7848cf2 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:46:25.020212 (+ 3667us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1989012352}
I20260809 15:46:25.030083 23274 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53562 (request call id 1431) took 1382 ms. Trace:
I20260809 15:46:24.857719 23395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "763e58d6ba644eea830a6c1664d39c9b" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:25.045449 23402 raft_consensus.cc:1261] T 297c82ed906d4b48b184985eed3de601 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 15:46:25.006719 23353 leader_election.cc:302] T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters: 61e69695258f48baa984c9d395e8ce3f
I20260809 15:46:25.046000 23402 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55320 (request call id 974) took 1839 ms. Trace:
I20260809 15:46:25.046695 23472 leader_election.cc:302] T 0acc47ce21c3422698d1ef7277c3ed63 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:46:25.016316 25519 raft_consensus.cc:3037] T 297c82ed906d4b48b184985eed3de601 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:25.046291 23402 rpcz_store.cc:276] 0809 15:46:23.206216 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:25.009456 (+1803240us) service_pool.cc:227] Handling call
0809 15:46:25.043080 (+ 33624us) spinlock_profiling.cc:243] Waited 28.2 ms on lock 0x7b58001d7c98. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e613cf11 00007f34e62c0851 00007f34e62c0702 00007f34ef1b0b09 00007f34ee0ec730 00007f34ee1b279b 00007f34ee1b2821 00007f34ee1762b0 00007f34ee1617f9 00007f34ee1612a2 00007f34e7847684
0809 15:46:25.043223 (+ 143us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:25.045784 (+ 2561us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:25.045953 (+ 169us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":64866688}
I20260809 15:46:25.034556 23283 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53554 (request call id 1056) took 1057 ms. Trace:
I20260809 15:46:24.890461 25609 raft_consensus.cc:3037] T 7660dfa2ef694927b8eda4100a2bcd9b P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:25.003074 23285 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53554 (request call id 1052) took 1026 ms. Trace:
I20260809 15:46:24.991204 23282 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53554 (request call id 1054) took 1014 ms. Trace:
I20260809 15:46:25.039821 23273 raft_consensus.cc:2379] T 73f6f44c259547f1b05a3a434616fa54 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d in current term 2: Already voted for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in this term.
I20260809 15:46:25.023094 25490 leader_election.cc:288] T ae6d2c08a7a94f58ac8dd698e3fd456c P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:25.014806 25483 raft_consensus.cc:482] T a3622f6e3e3641cea1960e3a31498bb7 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
I20260809 15:46:25.056543 25483 raft_consensus.cc:3037] T a3622f6e3e3641cea1960e3a31498bb7 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:25.042479 23400 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55320 (request call id 972) took 1845 ms. Trace:
I20260809 15:46:25.057904 23400 rpcz_store.cc:276] 0809 15:46:23.197393 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:24.953791 (+1756398us) service_pool.cc:227] Handling call
0809 15:46:25.042026 (+ 88235us) spinlock_profiling.cc:243] Waited 84.1 ms on lock 0x7b58001d7c98. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e613cf11 00007f34e62c0851 00007f34e62c0702 00007f34ef1b0b09 00007f34ee0ec730 00007f34ee1b279b 00007f34ee1b2821 00007f34ee1762b0 00007f34ee1617f9 00007f34ee1612a2 00007f34e7847684
0809 15:46:25.042193 (+ 167us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:25.042298 (+ 105us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:46:25.042312 (+ 14us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 15:46:25.042336 (+ 24us) raft_consensus.cc:1631] Marking committed up to 1
0809 15:46:25.042351 (+ 15us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:25.042380 (+ 29us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:25.042441 (+ 61us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":193348480}
I20260809 15:46:25.059585 23400 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55320 (request call id 990) took 1243 ms. Trace:
I20260809 15:46:25.044358 23280 raft_consensus.cc:3037] T c0741ab44d99439abc3aa3bdd27ee160 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:25.044533 23394 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55320 (request call id 973) took 1846 ms. Trace:
I20260809 15:46:25.040959 23228 leader_election.cc:302] T df20671a7b874dbb98c0ef28371c1308 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:46:25.045181 23270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c48a3049897452383257aa757834fb5" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:25.060557 23394 rpcz_store.cc:276] 0809 15:46:23.197681 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:25.006536 (+1808855us) service_pool.cc:227] Handling call
0809 15:46:25.043961 (+ 37425us) spinlock_profiling.cc:243] Waited 31.2 ms on lock 0x7b58001d7c98. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e613cf11 00007f34e62c0851 00007f34e62c0702 00007f34ef1b0b09 00007f34ee0ec730 00007f34ee1b279b 00007f34ee1b2821 00007f34ee1762b0 00007f34ee1617f9 00007f34ee1612a2 00007f34e7847684
0809 15:46:25.044119 (+ 158us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:25.044262 (+ 143us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:46:25.044283 (+ 21us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 15:46:25.044317 (+ 34us) raft_consensus.cc:1631] Marking committed up to 1
0809 15:46:25.044344 (+ 27us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:25.044389 (+ 45us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:25.044475 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":71666432}
I20260809 15:46:25.048817 23274 rpcz_store.cc:276] 0809 15:46:23.634322 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:24.846117 (+1211795us) service_pool.cc:227] Handling call
0809 15:46:25.016742 (+170625us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:46:25.048636 23402 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55320 (request call id 977) took 1817 ms. Trace:
I20260809 15:46:25.044382 23277 raft_consensus.cc:2454] T 0c29937a80724ddb9d07a27534a146f3 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 1.
I20260809 15:46:25.051349 23396 raft_consensus.cc:1226] T f9e7cadcd24d45fcbfa84209225e8f2d P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 1. Current term is 2. Ops: []
I20260809 15:46:25.059880 23400 rpcz_store.cc:276] 0809 15:46:23.815757 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:25.058892 (+1243135us) service_pool.cc:227] Handling call
0809 15:46:25.059153 (+ 261us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:25.059314 (+ 161us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:46:25.059337 (+ 23us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 15:46:25.059372 (+ 35us) raft_consensus.cc:1631] Marking committed up to 1
0809 15:46:25.059400 (+ 28us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:25.059447 (+ 47us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:25.059539 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:46:25.062294 23394 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55320 (request call id 996) took 1244 ms. Trace:
I20260809 15:46:25.062621 23277 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53554 (request call id 1058) took 1073 ms. Trace:
I20260809 15:46:25.063050 23396 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55320 (request call id 975) took 1837 ms. Trace:
I20260809 15:46:25.062783 23394 rpcz_store.cc:276] 0809 15:46:23.817383 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:25.061628 (+1244245us) service_pool.cc:227] Handling call
0809 15:46:25.061854 (+ 226us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:25.062006 (+ 152us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:46:25.062052 (+ 46us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 15:46:25.062090 (+ 38us) raft_consensus.cc:1631] Marking committed up to 1
0809 15:46:25.062118 (+ 28us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:25.062165 (+ 47us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:25.062251 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:46:25.061861 23228 leader_election.cc:302] T 953eea268e17491ca76f3a555860a512 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:46:25.063424 23400 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55320 (request call id 998) took 1245 ms. Trace:
I20260809 15:46:25.037819 23397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bcac4b24e7a4448abdc934bba5a624e" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:25.063897 23396 rpcz_store.cc:276] 0809 15:46:23.225134 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:25.010134 (+1785000us) service_pool.cc:227] Handling call
0809 15:46:25.045386 (+ 35252us) spinlock_profiling.cc:243] Waited 27.7 ms on lock 0x7b58001d7c98. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e613cf11 00007f34e62c0851 00007f34e62c0702 00007f34ef1b0b09 00007f34ee0ec730 00007f34ee1b279b 00007f34ee1b2821 00007f34ee1762b0 00007f34ee1617f9 00007f34ee1612a2 00007f34e7847684
0809 15:46:25.045556 (+ 170us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:25.062579 (+ 17023us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:25.063001 (+ 422us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":63635200}
I20260809 15:46:25.062062 23402 rpcz_store.cc:276] 0809 15:46:23.231389 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:25.047996 (+1816607us) service_pool.cc:227] Handling call
0809 15:46:25.048245 (+ 249us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:25.048386 (+ 141us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:46:25.048410 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 15:46:25.048443 (+ 33us) raft_consensus.cc:1631] Marking committed up to 2
0809 15:46:25.048469 (+ 26us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:25.048515 (+ 46us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:25.048597 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:46:25.064888 23400 rpcz_store.cc:276] 0809 15:46:23.817907 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:25.062725 (+1244818us) service_pool.cc:227] Handling call
0809 15:46:25.062960 (+ 235us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:25.063105 (+ 145us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:46:25.063127 (+ 22us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 15:46:25.063160 (+ 33us) raft_consensus.cc:1631] Marking committed up to 1
0809 15:46:25.063187 (+ 27us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:25.063230 (+ 43us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:25.063383 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:46:25.050149 23283 rpcz_store.cc:276] 0809 15:46:23.977458 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:25.030754 (+1053296us) service_pool.cc:227] Handling call
0809 15:46:25.034511 (+ 3757us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:46:25.053004 25539 raft_consensus.cc:2781] T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260809 15:46:25.054081 25369 raft_consensus.cc:3037] T c0741ab44d99439abc3aa3bdd27ee160 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:25.054757 23395 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55320 (request call id 978) took 1799 ms. Trace:
I20260809 15:46:25.055722 25569 leader_election.cc:288] T 73f6f44c259547f1b05a3a434616fa54 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 2 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:25.056613 25590 leader_election.cc:288] T 05ad7d8fdca64e3da527280ca5be9280 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
W20260809 15:46:25.060184 23273 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53562 (request call id 1423) took 1538 ms (client timeout 1961 ms). Trace:
I20260809 15:46:25.074702 23470 leader_election.cc:302] T 4b30f28311964a0b9149a50495e83d8f P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d
W20260809 15:46:25.074951 23273 rpcz_store.cc:269] 0809 15:46:23.521674 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:24.748007 (+1226333us) service_pool.cc:227] Handling call
0809 15:46:25.059243 (+311236us) spinlock_profiling.cc:243] Waited 48.3 ms on lock 0x7b5400366d54. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e92242ae 00007f34ee161c27 00007f34e7849c04 00007f34e7849b92 00007f34e7849ac1 00007f34e7849a3c 00007f34e7848cf2 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:46:25.060137 (+ 894us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":111187200}
I20260809 15:46:25.075799 23470 leader_election.cc:302] T 30c1740ef6874df0a332fbd6fd4b03ce P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:25.075975 23273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2d4f749fcfd4fb3bfb8115af3471598" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:25.076515 23273 raft_consensus.cc:2454] T d2d4f749fcfd4fb3bfb8115af3471598 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 1.
W20260809 15:46:25.077226 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 169882us
I20260809 15:46:25.077344 23273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5712f6b6fffc4cc5a8ddfbe63e4097df" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:25.077927 23273 raft_consensus.cc:2454] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 2.
I20260809 15:46:25.063001 23277 rpcz_store.cc:276] 0809 15:46:23.989667 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:24.978819 (+989152us) service_pool.cc:227] Handling call
0809 15:46:25.062573 (+ 83754us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:46:25.078210 23470 leader_election.cc:302] T 4564dfc6d4094c1b9240b367ccecf3b1 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters: c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:25.066322 23400 raft_consensus.cc:1226] T 563daf42d7784db195ea99c3cb21b370 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Rejecting Update request from peer 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 1. Current term is 2. Ops: []
I20260809 15:46:25.078884 23400 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55332 (request call id 1093) took 1011 ms. Trace:
I20260809 15:46:25.050616 23285 rpcz_store.cc:276] 0809 15:46:23.976380 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:25.001734 (+1025354us) service_pool.cc:227] Handling call
0809 15:46:25.003030 (+ 1296us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:46:25.051044 23282 rpcz_store.cc:276] 0809 15:46:23.976931 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:24.990304 (+1013373us) service_pool.cc:227] Handling call
0809 15:46:24.991158 (+ 854us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:46:25.008083 25551 raft_consensus.cc:482] T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting leader election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
I20260809 15:46:24.996891 25506 raft_consensus.cc:3032] T cee02fff2c9c48a19f43eb358a89e7b0 P 65c36d164b0540f099d592f38307ee6c [term 1 LEADER]: Stepping down as leader of term 1
I20260809 15:46:25.082162 25551 raft_consensus.cc:3037] T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:25.082410 25506 raft_consensus.cc:729] T cee02fff2c9c48a19f43eb358a89e7b0 P 65c36d164b0540f099d592f38307ee6c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:46:25.082720 23277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5c2e425a74b4e64ae70b5def750434c" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:25.060731 23472 leader_election.cc:302] T 4bcac4b24e7a4448abdc934bba5a624e P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:25.068260 23270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "538a228ddfb54bc6a36528d5ff07597b" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:25.061914 23274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3622f6e3e3641cea1960e3a31498bb7" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:25.083812 23397 raft_consensus.cc:1226] T 05ad7d8fdca64e3da527280ca5be9280 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Rejecting Update request from peer 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 1. Current term is 2. Ops: []
I20260809 15:46:25.084522 23274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ae14a74a26f434db08c652e097385c8" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:25.061414 25467 raft_consensus.cc:2781] T 0acc47ce21c3422698d1ef7277c3ed63 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260809 15:46:25.032604 23286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2424fd9bd7f14e508fd106cbfda786a4" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:25.070173 23528 raft_consensus.cc:3037] T 05ad7d8fdca64e3da527280ca5be9280 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:25.083297 25571 raft_consensus.cc:482] T ba56725e42f74a1baa4c04d9660727f9 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:25.090826 25571 raft_consensus.cc:3037] T ba56725e42f74a1baa4c04d9660727f9 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:25.088464 25467 raft_consensus.cc:482] T 0acc47ce21c3422698d1ef7277c3ed63 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Starting leader election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
I20260809 15:46:25.091877 25467 raft_consensus.cc:3037] T 0acc47ce21c3422698d1ef7277c3ed63 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:25.040233 25526 raft_consensus.cc:2726] T d3be1e356b254d22b995be32f1c12f6f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 15:46:25.074190 23283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07fcabb569414bddbd39df6cedd24e64" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:25.098618 23283 raft_consensus.cc:2454] T 07fcabb569414bddbd39df6cedd24e64 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 1.
W20260809 15:46:25.050315 23468 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.68:34689, user_credentials={real_user=slave}} blocked reactor thread for 173177us
I20260809 15:46:25.079206 23400 rpcz_store.cc:276] 0809 15:46:24.067114 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:25.065921 (+998807us) service_pool.cc:227] Handling call
0809 15:46:25.066179 (+ 258us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:25.078699 (+ 12520us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:25.078845 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:46:25.081477 25490 raft_consensus.cc:2726] T 4b30f28311964a0b9149a50495e83d8f P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 15:46:25.083307 23277 raft_consensus.cc:2454] T c5c2e425a74b4e64ae70b5def750434c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 1.
I20260809 15:46:25.087632 23282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "563daf42d7784db195ea99c3cb21b370" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:25.088151 23274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9e7cadcd24d45fcbfa84209225e8f2d" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:25.112205 23274 raft_consensus.cc:3037] T f9e7cadcd24d45fcbfa84209225e8f2d P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:25.081676 23285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cef3e7277d7d4760824a3436ed4a847d" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:25.087440 23270 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53554 (request call id 1060) took 1096 ms. Trace:
I20260809 15:46:25.089324 23397 raft_consensus.cc:1226] T 30c1740ef6874df0a332fbd6fd4b03ce P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Rejecting Update request from peer 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 2. Current term is 3. Ops: []
I20260809 15:46:25.113559 23270 rpcz_store.cc:276] 0809 15:46:23.991351 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:25.068101 (+1076750us) service_pool.cc:227] Handling call
0809 15:46:25.087393 (+ 19292us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:46:25.114560 23270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "213543cdbe814c89a17a963469922c1b" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:25.115106 23270 raft_consensus.cc:2454] T 213543cdbe814c89a17a963469922c1b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 1.
I20260809 15:46:25.091430 23393 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55320 (request call id 991) took 1275 ms. Trace:
I20260809 15:46:25.115932 23270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05ad7d8fdca64e3da527280ca5be9280" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:25.115686 23393 rpcz_store.cc:276] 0809 15:46:23.816030 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:25.068745 (+1252715us) service_pool.cc:227] Handling call
0809 15:46:25.090608 (+ 21863us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:25.090872 (+ 264us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:46:25.090927 (+ 55us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 15:46:25.091009 (+ 82us) raft_consensus.cc:1631] Marking committed up to 1
0809 15:46:25.091065 (+ 56us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:25.091141 (+ 76us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:25.091366 (+ 225us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:46:25.116983 23270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16ca412a0dae4feebe85dd01bf1a9173" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:25.072700 23228 leader_election.cc:302] T c5c2e425a74b4e64ae70b5def750434c P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:46:25.051015 25593 raft_consensus.cc:2726] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260809 15:46:25.118438 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 45802.2us
I20260809 15:46:25.119467 25569 raft_consensus.cc:2781] T c5c2e425a74b4e64ae70b5def750434c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260809 15:46:25.120070 25569 raft_consensus.cc:482] T c5c2e425a74b4e64ae70b5def750434c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:25.120630 25569 raft_consensus.cc:3037] T c5c2e425a74b4e64ae70b5def750434c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:25.067394 25539 raft_consensus.cc:482] T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:46:25.122573 25539 raft_consensus.cc:3037] T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:25.122167 23282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae6d2c08a7a94f58ac8dd698e3fd456c" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:25.124823 23282 raft_consensus.cc:2454] T ae6d2c08a7a94f58ac8dd698e3fd456c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 1.
I20260809 15:46:25.127120 23282 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "538a228ddfb54bc6a36528d5ff07597b"
dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
mode: GRACEFUL
new_leader_uuid: "65c36d164b0540f099d592f38307ee6c"
from {username='slave'} at 127.0.0.1:33468
I20260809 15:46:25.129292 23282 raft_consensus.cc:595] T 538a228ddfb54bc6a36528d5ff07597b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 LEADER]: Received request to transfer leadership to TS 65c36d164b0540f099d592f38307ee6c
I20260809 15:46:25.131959 23282 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53562 (request call id 1407) took 1864 ms. Trace:
I20260809 15:46:25.135378 23282 rpcz_store.cc:276] 0809 15:46:23.267531 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:25.131225 (+1863694us) service_pool.cc:227] Handling call
0809 15:46:25.131548 (+ 323us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:25.131699 (+ 151us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:46:25.131722 (+ 23us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 15:46:25.131756 (+ 34us) raft_consensus.cc:1631] Marking committed up to 1
0809 15:46:25.131782 (+ 26us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:25.131827 (+ 45us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:25.131911 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:46:25.138192 23282 raft_consensus.cc:1226] T e71a05ba1d4d42bd95c3f85f9313399e P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Rejecting Update request from peer c0f38c1908414566aa9115bb2cbdd85d for earlier term 1. Current term is 2. Ops: []
I20260809 15:46:25.118067 23270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0acc47ce21c3422698d1ef7277c3ed63" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:25.141407 23270 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53562 (request call id 1410) took 1872 ms. Trace:
I20260809 15:46:25.141855 23270 rpcz_store.cc:276] 0809 15:46:23.268612 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:25.140763 (+1872151us) service_pool.cc:227] Handling call
0809 15:46:25.141012 (+ 249us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:25.141152 (+ 140us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:46:25.141174 (+ 22us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 15:46:25.141210 (+ 36us) raft_consensus.cc:1631] Marking committed up to 1
0809 15:46:25.141236 (+ 26us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:25.141280 (+ 44us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:25.141360 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:46:25.143783 23270 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53562 (request call id 1411) took 1828 ms. Trace:
I20260809 15:46:25.145191 23282 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53562 (request call id 1408) took 1877 ms. Trace:
I20260809 15:46:25.148406 23282 rpcz_store.cc:276] 0809 15:46:23.267767 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:25.137790 (+1870023us) service_pool.cc:227] Handling call
0809 15:46:25.138047 (+ 257us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:25.144982 (+ 6935us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:25.145146 (+ 164us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:46:25.144177 23270 rpcz_store.cc:276] 0809 15:46:23.335564 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:25.142958 (+1807394us) service_pool.cc:227] Handling call
0809 15:46:25.143334 (+ 376us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:25.143522 (+ 188us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:46:25.143545 (+ 23us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 15:46:25.143589 (+ 44us) raft_consensus.cc:1631] Marking committed up to 1
0809 15:46:25.143615 (+ 26us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:25.143659 (+ 44us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:25.143739 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:46:25.156316 23282 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53562 (request call id 1414) took 1667 ms. Trace:
I20260809 15:46:25.158815 23282 rpcz_store.cc:276] 0809 15:46:23.489191 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:25.151165 (+1661974us) service_pool.cc:227] Handling call
0809 15:46:25.151440 (+ 275us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:25.151586 (+ 146us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:46:25.156045 (+ 4459us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 15:46:25.156091 (+ 46us) raft_consensus.cc:1631] Marking committed up to 1
0809 15:46:25.156125 (+ 34us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:25.156173 (+ 48us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:25.156271 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:46:25.169137 23273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e14d118b2eaa42aa8a7cd4025aba7128" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:25.072244 25576 raft_consensus.cc:2781] T df20671a7b874dbb98c0ef28371c1308 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260809 15:46:25.169358 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 70143.8us
I20260809 15:46:25.169724 23273 raft_consensus.cc:2454] T e14d118b2eaa42aa8a7cd4025aba7128 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 2.
I20260809 15:46:25.170722 23283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2424fd9bd7f14e508fd106cbfda786a4" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
W20260809 15:46:25.066844 24410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 173418us
W20260809 15:46:25.172189 23353 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 69091.4us
I20260809 15:46:25.084184 25477 raft_consensus.cc:2781] T 953eea268e17491ca76f3a555860a512 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260809 15:46:25.173699 23283 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53562 (request call id 1424) took 1651 ms. Trace:
I20260809 15:46:25.175516 23283 rpcz_store.cc:276] 0809 15:46:23.521916 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:25.173042 (+1651126us) service_pool.cc:227] Handling call
0809 15:46:25.173305 (+ 263us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:25.173445 (+ 140us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:46:25.173468 (+ 23us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 15:46:25.173502 (+ 34us) raft_consensus.cc:1631] Marking committed up to 1
0809 15:46:25.173529 (+ 27us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:25.173570 (+ 41us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:25.173654 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:46:25.177016 23283 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53562 (request call id 1427) took 1654 ms. Trace:
I20260809 15:46:25.177317 23283 rpcz_store.cc:276] 0809 15:46:23.522627 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:25.176365 (+1653738us) service_pool.cc:227] Handling call
0809 15:46:25.176618 (+ 253us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:25.176762 (+ 144us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:46:25.176784 (+ 22us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 15:46:25.176817 (+ 33us) raft_consensus.cc:1631] Marking committed up to 4
0809 15:46:25.176843 (+ 26us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:25.176885 (+ 42us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:25.176971 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:46:25.073089 23395 rpcz_store.cc:276] 0809 15:46:23.255686 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:25.053861 (+1798175us) service_pool.cc:227] Handling call
0809 15:46:25.054187 (+ 326us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:25.054351 (+ 164us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:46:25.054376 (+ 25us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 15:46:25.054445 (+ 69us) raft_consensus.cc:1631] Marking committed up to 1
0809 15:46:25.054496 (+ 51us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:25.054599 (+ 103us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:25.054704 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:46:25.181253 23286 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53554 (request call id 1051) took 1205 ms. Trace:
I20260809 15:46:25.181699 23286 rpcz_store.cc:276] 0809 15:46:23.976111 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:25.032246 (+1056135us) service_pool.cc:227] Handling call
0809 15:46:25.181193 (+148947us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:46:25.183629 23286 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55434 (request call id 72) took 1357 ms. Trace:
I20260809 15:46:25.183995 23277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30c1740ef6874df0a332fbd6fd4b03ce" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:25.185799 24452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16ca412a0dae4feebe85dd01bf1a9173" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f"
I20260809 15:46:25.172425 25576 raft_consensus.cc:482] T df20671a7b874dbb98c0ef28371c1308 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:25.186232 23518 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7660dfa2ef694927b8eda4100a2bcd9b"
dest_uuid: "65c36d164b0540f099d592f38307ee6c"
mode: GRACEFUL
new_leader_uuid: "61e69695258f48baa984c9d395e8ce3f"
from {username='slave'} at 127.0.0.1:48888
I20260809 15:46:25.186666 25576 raft_consensus.cc:3037] T df20671a7b874dbb98c0ef28371c1308 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:25.188345 23273 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53562 (request call id 1422) took 1666 ms. Trace:
I20260809 15:46:25.188864 23351 leader_election.cc:302] T 297c82ed906d4b48b184985eed3de601 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c
I20260809 15:46:25.188688 23273 rpcz_store.cc:276] 0809 15:46:23.521414 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:25.170337 (+1648923us) service_pool.cc:227] Handling call
0809 15:46:25.187954 (+ 17617us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:25.188101 (+ 147us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:46:25.188122 (+ 21us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 15:46:25.188155 (+ 33us) raft_consensus.cc:1631] Marking committed up to 1
0809 15:46:25.188181 (+ 26us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:25.188223 (+ 42us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:25.188301 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:46:25.174683 25477 raft_consensus.cc:482] T 953eea268e17491ca76f3a555860a512 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:25.191555 24463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3e94b7e50754a81b3d72a07476cb7b9" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f"
I20260809 15:46:25.190110 23270 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53562 (request call id 1415) took 1670 ms. Trace:
I20260809 15:46:25.193143 23270 rpcz_store.cc:276] 0809 15:46:23.519597 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:25.152025 (+1632428us) service_pool.cc:227] Handling call
0809 15:46:25.187868 (+ 35843us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:25.188000 (+ 132us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:46:25.188022 (+ 22us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 15:46:25.188057 (+ 35us) raft_consensus.cc:1631] Marking committed up to 1
0809 15:46:25.188083 (+ 26us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:25.188129 (+ 46us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:25.190059 (+ 1930us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:46:25.093936 25614 raft_consensus.cc:2726] T 30c1740ef6874df0a332fbd6fd4b03ce P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 15:46:25.198596 24410 leader_election.cc:302] T 3d880de657e544468c82157769622a26 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:25.185195 23286 rpcz_store.cc:276] 0809 15:46:23.826360 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:25.182655 (+1356295us) service_pool.cc:227] Handling call
0809 15:46:25.183067 (+ 412us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:25.183222 (+ 155us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:46:25.183318 (+ 96us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 15:46:25.183362 (+ 44us) raft_consensus.cc:1631] Marking committed up to 5
0809 15:46:25.183392 (+ 30us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:25.183451 (+ 59us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:25.183562 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:46:25.199905 23392 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55320 (request call id 971) took 2002 ms. Trace:
I20260809 15:46:25.200230 24410 leader_election.cc:302] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c; no voters:
I20260809 15:46:25.200541 23392 rpcz_store.cc:276] 0809 15:46:23.197095 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:24.861762 (+1664667us) service_pool.cc:227] Handling call
0809 15:46:25.016995 (+155233us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:25.051414 (+ 34419us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:25.199688 (+148274us) spinlock_profiling.cc:243] Waited 154 ms on lock 0x7b5400140ad0. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e921fa09 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:46:25.199819 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":355268864}
I20260809 15:46:25.172662 25616 raft_consensus.cc:2726] T 4bcac4b24e7a4448abdc934bba5a624e P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 15:46:25.191303 23283 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55434 (request call id 70) took 1368 ms. Trace:
I20260809 15:46:25.230101 23283 rpcz_store.cc:276] 0809 15:46:23.823205 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:25.178078 (+1354873us) service_pool.cc:227] Handling call
0809 15:46:25.190875 (+ 12797us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:25.191014 (+ 139us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:46:25.191035 (+ 21us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 15:46:25.191079 (+ 44us) raft_consensus.cc:1631] Marking committed up to 5
0809 15:46:25.191107 (+ 28us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:25.191154 (+ 47us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:25.191241 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:46:25.174806 25593 raft_consensus.cc:2781] T 4564dfc6d4094c1b9240b367ccecf3b1 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260809 15:46:25.196873 23399 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55320 (request call id 976) took 1970 ms. Trace:
I20260809 15:46:25.190222 25477 raft_consensus.cc:3037] T 953eea268e17491ca76f3a555860a512 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:25.232889 24410 leader_election.cc:302] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:25.202955 23518 raft_consensus.cc:595] T 7660dfa2ef694927b8eda4100a2bcd9b P 65c36d164b0540f099d592f38307ee6c [term 1 LEADER]: Received request to transfer leadership to TS 61e69695258f48baa984c9d395e8ce3f
I20260809 15:46:25.188078 23228 leader_election.cc:302] T f7715d71dbac447b9351fc5eb36c5e80 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:46:25.094049 25506 consensus_queue.cc:260] T cee02fff2c9c48a19f43eb358a89e7b0 P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:25.083809 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0741ab44d99439abc3aa3bdd27ee160" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:25.231034 23399 rpcz_store.cc:276] 0809 15:46:23.227246 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:25.025078 (+1797832us) service_pool.cc:227] Handling call
0809 15:46:25.195940 (+170862us) spinlock_profiling.cc:243] Waited 15.1 ms on lock 0x7b58001d7c98. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e613cf11 00007f34e62c0851 00007f34e62c0702 00007f34ef1b0b09 00007f34ee0ec730 00007f34ee1b279b 00007f34ee1b2821 00007f34ee1762b0 00007f34ee1617f9 00007f34ee1612a2 00007f34e7847684
0809 15:46:25.196146 (+ 206us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:25.196358 (+ 212us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:46:25.196391 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 15:46:25.196433 (+ 42us) raft_consensus.cc:1631] Marking committed up to 1
0809 15:46:25.196494 (+ 61us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:25.196549 (+ 55us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:25.196809 (+ 260us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":34663168}
I20260809 15:46:25.231043 25593 raft_consensus.cc:482] T 4564dfc6d4094c1b9240b367ccecf3b1 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Starting leader election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:46:25.235030 23277 raft_consensus.cc:1226] T 73f6f44c259547f1b05a3a434616fa54 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 1. Current term is 2. Ops: []
I20260809 15:46:25.229722 25478 raft_consensus.cc:2726] T 3d880de657e544468c82157769622a26 P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 15:46:25.243223 24471 raft_consensus.cc:1226] T cee02fff2c9c48a19f43eb358a89e7b0 P 61e69695258f48baa984c9d395e8ce3f [term 2 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 1. Current term is 2. Ops: []
I20260809 15:46:25.244097 25603 raft_consensus.cc:2781] T 297c82ed906d4b48b184985eed3de601 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260809 15:46:25.234179 25488 raft_consensus.cc:2726] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 15:46:25.319610 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472816994d814cb198b69e5248697440" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
W20260809 15:46:25.237980 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 49330.3us
I20260809 15:46:25.322474 23351 leader_election.cc:302] T 953eea268e17491ca76f3a555860a512 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c
I20260809 15:46:25.323597 23351 leader_election.cc:302] T 763e58d6ba644eea830a6c1664d39c9b P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
W20260809 15:46:25.254549 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 135334us
W20260809 15:46:25.254356 23472 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 193677us
I20260809 15:46:25.307539 25576 raft_consensus.cc:504] T df20671a7b874dbb98c0ef28371c1308 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:25.260350 25465 raft_consensus.cc:2781] T f7715d71dbac447b9351fc5eb36c5e80 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260809 15:46:25.235738 23282 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53562 (request call id 1420) took 1709 ms. Trace:
I20260809 15:46:25.253726 25453 raft_consensus.cc:2781] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260809 15:46:25.237700 23274 raft_consensus.cc:2454] T f9e7cadcd24d45fcbfa84209225e8f2d P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 2.
I20260809 15:46:25.316747 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ae14a74a26f434db08c652e097385c8" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:25.317674 23277 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53554 (request call id 1041) took 1439 ms. Trace:
I20260809 15:46:25.324945 23472 leader_election.cc:302] T 3924b648461e463b9d7925e5671d68ae P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:25.305495 25506 raft_consensus.cc:3037] T cee02fff2c9c48a19f43eb358a89e7b0 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:25.327282 23282 rpcz_store.cc:276] 0809 15:46:23.520960 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:25.162904 (+1641944us) service_pool.cc:227] Handling call
0809 15:46:25.163157 (+ 253us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:25.169174 (+ 6017us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:46:25.169202 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 15:46:25.169236 (+ 34us) raft_consensus.cc:1631] Marking committed up to 4
0809 15:46:25.169266 (+ 30us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:25.230019 (+ 60753us) spinlock_profiling.cc:243] Waited 5.86 ms on lock 0x7b54001554d4. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e9220f51 00007f34e921f99e 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772 00007f34e764a94c 00007f34e7649d86
0809 15:46:25.230058 (+ 39us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:25.230163 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":13469952}
W20260809 15:46:25.275614 24410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 46113.2us
I20260809 15:46:25.371579 25500 raft_consensus.cc:504] T 00a1881267a14ba8b78c81040f17a014 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:25.368250 23522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2424fd9bd7f14e508fd106cbfda786a4" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:25.374434 23394 raft_consensus.cc:1226] T 00a1881267a14ba8b78c81040f17a014 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Rejecting Update request from peer 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 2. Current term is 3. Ops: []
I20260809 15:46:25.332715 23280 raft_consensus.cc:2454] T c0741ab44d99439abc3aa3bdd27ee160 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 2.
I20260809 15:46:25.345278 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7660dfa2ef694927b8eda4100a2bcd9b" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:25.375658 23280 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55434 (request call id 74) took 1343 ms. Trace:
I20260809 15:46:25.376017 23280 rpcz_store.cc:276] 0809 15:46:24.031898 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:25.028343 (+996445us) service_pool.cc:227] Handling call
0809 15:46:25.375559 (+347216us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:46:25.378737 23274 raft_consensus.cc:1226] T f9e7cadcd24d45fcbfa84209225e8f2d P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 1. Current term is 2. Ops: []
I20260809 15:46:25.329319 23517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30c1740ef6874df0a332fbd6fd4b03ce" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:25.317356 25593 raft_consensus.cc:3037] T 4564dfc6d4094c1b9240b367ccecf3b1 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:25.380102 23277 rpcz_store.cc:276] 0809 15:46:23.878202 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:25.184838 (+1306636us) service_pool.cc:227] Handling call
0809 15:46:25.234912 (+ 50074us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:25.317464 (+ 82552us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:25.317624 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 15:46:25.369820 23472 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 44949.2us
W20260809 15:46:25.381259 23470 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 303133us
I20260809 15:46:25.381669 23274 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53554 (request call id 1048) took 1406 ms. Trace:
I20260809 15:46:25.382529 23470 leader_election.cc:302] T 00a1881267a14ba8b78c81040f17a014 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:25.382436 23472 leader_election.cc:302] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d
W20260809 15:46:25.383709 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 51379.8us
I20260809 15:46:25.384137 23470 leader_election.cc:302] T d0721a4403a440f383318a3cea9d1442 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters: c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:25.327251 23528 raft_consensus.cc:2454] T 05ad7d8fdca64e3da527280ca5be9280 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 2.
I20260809 15:46:25.385263 23470 leader_election.cc:302] T 7baa2492df0340c4a5ac241959ce1ac3 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:25.319789 25603 raft_consensus.cc:482] T 297c82ed906d4b48b184985eed3de601 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:46:25.382194 23274 rpcz_store.cc:276] 0809 15:46:23.975222 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:25.378390 (+1403168us) service_pool.cc:227] Handling call
0809 15:46:25.378626 (+ 236us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:25.381478 (+ 2852us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:25.381624 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:46:25.435545 23394 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55332 (request call id 1104) took 1209 ms. Trace:
W20260809 15:46:25.464718 23470 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 79526.9us
I20260809 15:46:25.326975 25465 raft_consensus.cc:482] T f7715d71dbac447b9351fc5eb36c5e80 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:25.386171 23472 leader_election.cc:302] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:25.329591 25497 raft_consensus.cc:2781] T 953eea268e17491ca76f3a555860a512 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260809 15:46:25.386695 23280 raft_consensus.cc:1226] T df20671a7b874dbb98c0ef28371c1308 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 1. Current term is 2. Ops: []
I20260809 15:46:25.479620 23470 leader_election.cc:302] T 294d4bcc1e7a477cb56455581277a60f P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 61e69695258f48baa984c9d395e8ce3f
W20260809 15:46:25.481178 23472 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 95071.9us
I20260809 15:46:25.483497 23280 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53554 (request call id 1046) took 1589 ms. Trace:
I20260809 15:46:25.382872 23286 raft_consensus.cc:1226] T 998c6cee72cf40228c6dba064c0db2da P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 3. Current term is 4. Ops: []
I20260809 15:46:25.385531 25366 raft_consensus.cc:2726] T 00a1881267a14ba8b78c81040f17a014 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 15:46:25.486672 23470 leader_election.cc:302] T 0c48a3049897452383257aa757834fb5 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 61e69695258f48baa984c9d395e8ce3f, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:25.480957 23274 raft_consensus.cc:1226] T c0741ab44d99439abc3aa3bdd27ee160 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 1. Current term is 2. Ops: []
I20260809 15:46:25.479682 25465 raft_consensus.cc:3037] T f7715d71dbac447b9351fc5eb36c5e80 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:25.377719 25453 raft_consensus.cc:482] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Starting leader election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:46:25.397501 25620 raft_consensus.cc:2781] T d0721a4403a440f383318a3cea9d1442 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260809 15:46:25.448815 25519 raft_consensus.cc:504] T 297c82ed906d4b48b184985eed3de601 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:25.397794 25490 raft_consensus.cc:2726] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 15:46:25.418303 25525 raft_consensus.cc:2726] T 763e58d6ba644eea830a6c1664d39c9b P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 15:46:25.399600 23518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9e7cadcd24d45fcbfa84209225e8f2d" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:25.465847 25616 raft_consensus.cc:2726] T 7baa2492df0340c4a5ac241959ce1ac3 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 15:46:25.488329 23472 leader_election.cc:302] T d1a134ba33ea4076bbaac0ae460c17ba P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:25.483887 23280 rpcz_store.cc:276] 0809 15:46:23.894320 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:25.386275 (+1491955us) service_pool.cc:227] Handling call
0809 15:46:25.386550 (+ 275us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:25.483210 (+ 96660us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:25.483444 (+ 234us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:46:25.374944 25590 raft_consensus.cc:2726] T 3924b648461e463b9d7925e5671d68ae P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 15:46:25.481328 25603 raft_consensus.cc:3037] T 297c82ed906d4b48b184985eed3de601 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:25.538890 23280 raft_consensus.cc:1226] T 297c82ed906d4b48b184985eed3de601 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260809 15:46:25.496657 25453 raft_consensus.cc:3037] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:25.546037 25453 raft_consensus.cc:504] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 61e69695258f48baa984c9d395e8ce3f [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:25.479522 23394 rpcz_store.cc:276] 0809 15:46:24.225683 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:25.069260 (+843577us) service_pool.cc:227] Handling call
0809 15:46:25.069524 (+ 264us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:25.382961 (+313437us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:25.435340 (+ 52379us) spinlock_profiling.cc:243] Waited 305 ms on lock 0x7b540025e754. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e9220f51 00007f34e921f99e 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772 00007f34e764a94c 00007f34e7649d86
0809 15:46:25.435484 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":700875648}
I20260809 15:46:25.497651 23274 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53554 (request call id 1065) took 1287 ms. Trace:
I20260809 15:46:25.547298 23274 rpcz_store.cc:276] 0809 15:46:24.209876 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:25.480551 (+1270675us) service_pool.cc:227] Handling call
0809 15:46:25.480808 (+ 257us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:25.489726 (+ 8918us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:25.490592 (+ 866us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:46:25.398617 24468 raft_consensus.cc:2379] T c0741ab44d99439abc3aa3bdd27ee160 P 61e69695258f48baa984c9d395e8ce3f [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 in current term 2: Already voted for candidate 61e69695258f48baa984c9d395e8ce3f in this term.
I20260809 15:46:25.455565 25573 raft_consensus.cc:504] T cef3e7277d7d4760824a3436ed4a847d P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:25.498564 23286 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53554 (request call id 1044) took 1609 ms. Trace:
I20260809 15:46:25.499506 23519 raft_consensus.cc:1226] T 05ad7d8fdca64e3da527280ca5be9280 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Rejecting Update request from peer 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 1. Current term is 2. Ops: []
I20260809 15:46:25.499807 23528 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56766 (request call id 1556) took 1266 ms. Trace:
I20260809 15:46:25.507756 25529 raft_consensus.cc:504] T 0da4d5283c8a46878bd39b85e5c71f66 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:25.486505 25614 raft_consensus.cc:2726] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 15:46:25.520474 25609 raft_consensus.cc:504] T 7660dfa2ef694927b8eda4100a2bcd9b P 61e69695258f48baa984c9d395e8ce3f [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:25.480589 23276 raft_consensus.cc:2454] T cee02fff2c9c48a19f43eb358a89e7b0 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 2.
I20260809 15:46:25.483590 25497 raft_consensus.cc:482] T 953eea268e17491ca76f3a555860a512 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:25.448140 24410 leader_election.cc:302] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c; no voters: c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:25.432458 25526 consensus_queue.cc:1046] T 563daf42d7784db195ea99c3cb21b370 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.056s
I20260809 15:46:25.456066 25583 raft_consensus.cc:504] T 538a228ddfb54bc6a36528d5ff07597b P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:46:25.535390 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 150875us
W20260809 15:46:25.536105 23353 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 212378us
I20260809 15:46:25.550736 25500 leader_election.cc:288] T 00a1881267a14ba8b78c81040f17a014 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:25.514755 25539 raft_consensus.cc:504] T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:25.492447 25366 raft_consensus.cc:482] T 6ae14a74a26f434db08c652e097385c8 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
W20260809 15:46:25.532526 25460 consensus_meta.cc:287] T 2db057fd10e149a9895dff6ef8c9f038 P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent flushing consensus metadata: real 0.510s user 0.011s sys 0.025s
I20260809 15:46:25.417289 25501 consensus_queue.cc:1046] T 05ad7d8fdca64e3da527280ca5be9280 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.100s
W20260809 15:46:25.549104 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 225597us
I20260809 15:46:25.502394 25467 raft_consensus.cc:504] T 0acc47ce21c3422698d1ef7277c3ed63 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:25.497535 25620 raft_consensus.cc:482] T d0721a4403a440f383318a3cea9d1442 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Starting leader election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:46:25.519096 25551 raft_consensus.cc:504] T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:25.601128 23529 raft_consensus.cc:1226] T 538a228ddfb54bc6a36528d5ff07597b P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Rejecting Update request from peer 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 2. Current term is 3. Ops: []
W20260809 15:46:25.559975 25483 consensus_meta.cc:287] T a3622f6e3e3641cea1960e3a31498bb7 P c0f38c1908414566aa9115bb2cbdd85d: Time spent flushing consensus metadata: real 0.503s user 0.008s sys 0.008s
I20260809 15:46:25.561702 23282 raft_consensus.cc:1226] T cee02fff2c9c48a19f43eb358a89e7b0 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 1. Current term is 2. Ops: []
W20260809 15:46:25.590889 25369 consensus_meta.cc:287] T c0741ab44d99439abc3aa3bdd27ee160 P 65c36d164b0540f099d592f38307ee6c: Time spent flushing consensus metadata: real 0.519s user 0.007s sys 0.003s
I20260809 15:46:25.548964 25490 raft_consensus.cc:2726] T 0c48a3049897452383257aa757834fb5 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 15:46:25.601372 25481 consensus_queue.cc:1046] T 00a1881267a14ba8b78c81040f17a014 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.034s
I20260809 15:46:25.498157 25555 raft_consensus.cc:504] T 538a228ddfb54bc6a36528d5ff07597b P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:25.603742 25501 raft_consensus.cc:3032] T 05ad7d8fdca64e3da527280ca5be9280 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 LEADER]: Stepping down as leader of term 1
I20260809 15:46:25.633507 25529 leader_election.cc:288] T 0da4d5283c8a46878bd39b85e5c71f66 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 2 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:25.566676 23285 raft_consensus.cc:2379] T cef3e7277d7d4760824a3436ed4a847d P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d in current term 2: Already voted for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in this term.
I20260809 15:46:25.594102 25478 raft_consensus.cc:2781] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260809 15:46:25.605700 24410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 168952us
I20260809 15:46:25.634270 25478 raft_consensus.cc:482] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Starting leader election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:46:25.634691 25478 raft_consensus.cc:3037] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:25.635144 25539 leader_election.cc:288] T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 election: Requested vote from peers 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
W20260809 15:46:25.555563 25532 consensus_meta.cc:287] T a88421dba80d4d3388aa5eb62e23853b P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent flushing consensus metadata: real 0.575s user 0.015s sys 0.017s
I20260809 15:46:25.550390 25576 leader_election.cc:288] T df20671a7b874dbb98c0ef28371c1308 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 2 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
W20260809 15:46:25.537998 23472 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 49742.9us
I20260809 15:46:25.549154 25621 raft_consensus.cc:2781] T 294d4bcc1e7a477cb56455581277a60f P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260809 15:46:25.588295 23404 raft_consensus.cc:2379] T 0da4d5283c8a46878bd39b85e5c71f66 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c in current term 2: Already voted for candidate c0f38c1908414566aa9115bb2cbdd85d in this term.
I20260809 15:46:25.588923 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00a1881267a14ba8b78c81040f17a014" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:25.589162 24471 raft_consensus.cc:1226] T 7660dfa2ef694927b8eda4100a2bcd9b P 61e69695258f48baa984c9d395e8ce3f [term 2 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 1. Current term is 2. Ops: []
W20260809 15:46:25.591284 23470 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 104645us
I20260809 15:46:25.591353 23394 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ae6d2c08a7a94f58ac8dd698e3fd456c"
dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
mode: GRACEFUL
new_leader_uuid: "65c36d164b0540f099d592f38307ee6c"
from {username='slave'} at 127.0.0.1:51280
I20260809 15:46:25.565310 25497 raft_consensus.cc:3037] T 953eea268e17491ca76f3a555860a512 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Advancing to term 2
W20260809 15:46:25.570839 23314 debug-util.cc:398] Leaking SignalData structure 0x7b08000e2160 after lost signal to thread 21293
I20260809 15:46:25.595077 23353 leader_election.cc:302] T 472816994d814cb198b69e5248697440 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c
I20260809 15:46:25.593358 23280 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53554 (request call id 1068) took 1374 ms. Trace:
I20260809 15:46:25.564164 23519 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56770 (request call id 1069) took 1513 ms. Trace:
W20260809 15:46:25.566991 23276 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55434 (request call id 73) took 1540 ms (client timeout 1973 ms). Trace:
I20260809 15:46:25.605173 25609 leader_election.cc:288] T 7660dfa2ef694927b8eda4100a2bcd9b P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 2 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:25.603832 25483 raft_consensus.cc:504] T a3622f6e3e3641cea1960e3a31498bb7 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:25.553365 23528 rpcz_store.cc:276] 0809 15:46:24.232898 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:24.871315 (+638417us) service_pool.cc:227] Handling call
0809 15:46:25.499137 (+627822us) spinlock_profiling.cc:243] Waited 187 ms on lock 0x7b540016dd54. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e92242ae 00007f34ee161c27 00007f34e7849c04 00007f34e7849b92 00007f34e7849ac1 00007f34e7849a3c 00007f34e7848cf2 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:46:25.499256 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":430899456}
I20260809 15:46:25.551891 23286 rpcz_store.cc:276] 0809 15:46:23.889168 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:25.204807 (+1315639us) service_pool.cc:227] Handling call
0809 15:46:25.321863 (+117056us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:25.489250 (+167387us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:25.498503 (+ 9253us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:46:25.602754 23351 leader_election.cc:302] T 3296ef50be254fc29cfc77d22a8f7a23 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 61e69695258f48baa984c9d395e8ce3f, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:25.549961 25590 raft_consensus.cc:2726] T d1a134ba33ea4076bbaac0ae460c17ba P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 15:46:25.636448 23529 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56770 (request call id 1064) took 1587 ms. Trace:
I20260809 15:46:25.633960 23402 raft_consensus.cc:1226] T 538a228ddfb54bc6a36528d5ff07597b P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Rejecting Update request from peer 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 2. Current term is 3. Ops: []
I20260809 15:46:25.596583 25460 raft_consensus.cc:504] T 2db057fd10e149a9895dff6ef8c9f038 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:25.563688 25571 raft_consensus.cc:504] T ba56725e42f74a1baa4c04d9660727f9 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:25.594084 25366 raft_consensus.cc:504] T 6ae14a74a26f434db08c652e097385c8 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:25.633582 25501 raft_consensus.cc:729] T 05ad7d8fdca64e3da527280ca5be9280 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Running, Role: LEADER
I20260809 15:46:25.648146 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3e94b7e50754a81b3d72a07476cb7b9" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f"
I20260809 15:46:25.648836 25621 raft_consensus.cc:482] T 294d4bcc1e7a477cb56455581277a60f P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Starting leader election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
I20260809 15:46:25.648782 25490 consensus_queue.cc:1046] T 7660dfa2ef694927b8eda4100a2bcd9b P 65c36d164b0540f099d592f38307ee6c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260809 15:46:25.650322 23472 leader_election.cc:302] T a3622f6e3e3641cea1960e3a31498bb7 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:46:25.649649 23394 raft_consensus.cc:595] T ae6d2c08a7a94f58ac8dd698e3fd456c P c0f38c1908414566aa9115bb2cbdd85d [term 1 LEADER]: Received request to transfer leadership to TS 65c36d164b0540f099d592f38307ee6c
I20260809 15:46:25.652011 23470 leader_election.cc:302] T d2d4f749fcfd4fb3bfb8115af3471598 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters: c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:25.651899 23280 rpcz_store.cc:276] 0809 15:46:24.219185 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:25.533878 (+1314693us) service_pool.cc:227] Handling call
0809 15:46:25.534196 (+ 318us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 15:46:25.539334 (+ 5138us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:25.593194 (+ 53860us) spinlock_profiling.cc:243] Waited 4.48 ms on lock 0x7b5400138dd4. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e9220f51 00007f34e921f99e 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772 00007f34e764a94c 00007f34e7649d86
0809 15:46:25.593317 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":10295168}
I20260809 15:46:25.601432 25620 raft_consensus.cc:3037] T d0721a4403a440f383318a3cea9d1442 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Advancing to term 2
W20260809 15:46:25.653362 23353 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 58345.3us
I20260809 15:46:25.652185 23519 rpcz_store.cc:276] 0809 15:46:24.050953 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:25.002695 (+951742us) service_pool.cc:227] Handling call
0809 15:46:25.499339 (+496644us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:25.552021 (+ 52682us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:25.563980 (+ 11959us) spinlock_profiling.cc:243] Waited 496 ms on lock 0x7b540016dd50. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e921fa09 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:46:25.564105 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1141604608}
W20260809 15:46:25.655292 23404 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55320 (request call id 1018) took 1397 ms (client timeout 1800 ms). Trace:
I20260809 15:46:25.655859 23396 raft_consensus.cc:1226] T 0da4d5283c8a46878bd39b85e5c71f66 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Rejecting Update request from peer 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 1. Current term is 2. Ops: []
W20260809 15:46:25.656638 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 64900.2us
I20260809 15:46:25.633996 25555 leader_election.cc:288] T 538a228ddfb54bc6a36528d5ff07597b P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
W20260809 15:46:25.659137 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 56435.2us
I20260809 15:46:25.660828 23529 rpcz_store.cc:276] 0809 15:46:24.049217 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:25.000108 (+950891us) service_pool.cc:227] Handling call
0809 15:46:25.000359 (+ 251us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:25.602081 (+601722us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:25.636255 (+ 34174us) spinlock_profiling.cc:243] Waited 599 ms on lock 0x7b540016e754. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e9220f51 00007f34e921f99e 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772 00007f34e764a94c 00007f34e7649d86
0809 15:46:25.636400 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1377388288}
I20260809 15:46:25.639211 23282 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53554 (request call id 1047) took 1741 ms. Trace:
I20260809 15:46:25.690867 23282 rpcz_store.cc:276] 0809 15:46:23.899200 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:25.384951 (+1485751us) service_pool.cc:227] Handling call
0809 15:46:25.561474 (+176523us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:25.602251 (+ 40777us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:25.636776 (+ 34525us) spinlock_profiling.cc:243] Waited 176 ms on lock 0x7b54002f58d0. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e921fa09 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:46:25.639165 (+ 2389us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":405431552}
I20260809 15:46:25.658424 24410 leader_election.cc:302] T 16ca412a0dae4feebe85dd01bf1a9173 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
W20260809 15:46:25.689029 23404 rpcz_store.cc:269] 0809 15:46:24.257703 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:24.328350 (+ 70647us) service_pool.cc:227] Handling call
0809 15:46:25.655082 (+1326732us) spinlock_profiling.cc:243] Waited 726 ms on lock 0x7b5400193554. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e92242ae 00007f34ee161c27 00007f34e7849c04 00007f34e7849b92 00007f34e7849ac1 00007f34e7849a3c 00007f34e7848cf2 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:46:25.655220 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1670386048}
I20260809 15:46:25.664619 25635 raft_consensus.cc:3032] T 563daf42d7784db195ea99c3cb21b370 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 LEADER]: Stepping down as leader of term 1
I20260809 15:46:25.654408 23393 raft_consensus.cc:1226] T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Rejecting Update request from peer 61e69695258f48baa984c9d395e8ce3f for earlier term 2. Current term is 3. Ops: [2.5-2.5]
I20260809 15:46:25.695348 23353 leader_election.cc:302] T 7660dfa2ef694927b8eda4100a2bcd9b P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c
I20260809 15:46:25.662894 25507 raft_consensus.cc:2726] T 3296ef50be254fc29cfc77d22a8f7a23 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 15:46:25.655721 25457 raft_consensus.cc:2726] T 472816994d814cb198b69e5248697440 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260809 15:46:25.667306 23402 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55332 (request call id 1083) took 1631 ms. Trace:
I20260809 15:46:25.668076 25467 leader_election.cc:288] T 0acc47ce21c3422698d1ef7277c3ed63 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 2 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:25.594473 25587 consensus_queue.cc:1046] T 30c1740ef6874df0a332fbd6fd4b03ce P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.050s
W20260809 15:46:25.659286 25559 consensus_meta.cc:287] T e71a05ba1d4d42bd95c3f85f9313399e P c0f38c1908414566aa9115bb2cbdd85d: Time spent flushing consensus metadata: real 0.626s user 0.015s sys 0.018s
I20260809 15:46:25.652812 25576 raft_consensus.cc:3032] T 00a1881267a14ba8b78c81040f17a014 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 LEADER]: Stepping down as leader of term 2
I20260809 15:46:25.682744 25637 ts_tablet_manager.cc:927] T 0acc47ce21c3422698d1ef7277c3ed63 P 61e69695258f48baa984c9d395e8ce3f: Initiating tablet copy from peer 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459)
I20260809 15:46:25.683530 25571 leader_election.cc:288] T ba56725e42f74a1baa4c04d9660727f9 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 2 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:25.684108 25453 leader_election.cc:288] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 2 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:25.686534 23274 raft_consensus.cc:1226] T 2db057fd10e149a9895dff6ef8c9f038 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 1. Current term is 2. Ops: []
I20260809 15:46:25.725968 23274 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53554 (request call id 1072) took 1502 ms. Trace:
I20260809 15:46:25.726261 23274 rpcz_store.cc:276] 0809 15:46:24.223114 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:25.548042 (+1324928us) service_pool.cc:227] Handling call
0809 15:46:25.548304 (+ 262us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:25.725283 (+176979us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:25.725761 (+ 478us) spinlock_profiling.cc:243] Waited 138 ms on lock 0x7b54001536d4. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e9220f51 00007f34e921f99e 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772 00007f34e764a94c 00007f34e7649d86
0809 15:46:25.725905 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":317501824}
I20260809 15:46:25.641603 25532 raft_consensus.cc:504] T a88421dba80d4d3388aa5eb62e23853b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:25.693920 23351 leader_election.cc:302] T 0a51c0baaa6e432997c9d3552af4f254 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 61e69695258f48baa984c9d395e8ce3f, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:25.665066 25483 leader_election.cc:288] T a3622f6e3e3641cea1960e3a31498bb7 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 2 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
W20260809 15:46:25.696146 23472 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 45889.1us
I20260809 15:46:25.703361 23402 rpcz_store.cc:276] 0809 15:46:24.035730 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:25.065307 (+1029577us) service_pool.cc:227] Handling call
0809 15:46:25.065541 (+ 234us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:25.661075 (+595534us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:25.667126 (+ 6051us) spinlock_profiling.cc:243] Waited 568 ms on lock 0x7b54001cfcd4. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e9220f51 00007f34e921f99e 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772 00007f34e764a94c 00007f34e7649d86
0809 15:46:25.667237 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1306849152}
W20260809 15:46:25.686925 23314 debug-util.cc:398] Leaking SignalData structure 0x7b08000eaa40 after lost signal to thread 23176
I20260809 15:46:25.734506 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "294d4bcc1e7a477cb56455581277a60f" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:25.735008 23351 leader_election.cc:302] T 7baa2492df0340c4a5ac241959ce1ac3 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:25.704704 25614 raft_consensus.cc:2781] T a3622f6e3e3641cea1960e3a31498bb7 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260809 15:46:25.706593 25587 raft_consensus.cc:3032] T 30c1740ef6874df0a332fbd6fd4b03ce P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 LEADER]: Stepping down as leader of term 2
I20260809 15:46:25.704861 25369 raft_consensus.cc:2781] T d2d4f749fcfd4fb3bfb8115af3471598 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260809 15:46:25.653836 23276 rpcz_store.cc:269] 0809 15:46:24.026712 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:25.018984 (+992272us) service_pool.cc:227] Handling call
0809 15:46:25.565567 (+546583us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 15:46:25.696720 23470 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 44818.5us
I20260809 15:46:25.739539 23399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e71a05ba1d4d42bd95c3f85f9313399e" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:25.738128 25587 raft_consensus.cc:729] T 30c1740ef6874df0a332fbd6fd4b03ce P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Running, Role: LEADER
I20260809 15:46:25.739841 23470 leader_election.cc:302] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 61e69695258f48baa984c9d395e8ce3f
I20260809 15:46:25.698326 23396 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55332 (request call id 1082) took 1662 ms. Trace:
I20260809 15:46:25.695111 25609 raft_consensus.cc:2726] T 16ca412a0dae4feebe85dd01bf1a9173 P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 15:46:25.694931 25635 raft_consensus.cc:729] T 563daf42d7784db195ea99c3cb21b370 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Running, Role: LEADER
I20260809 15:46:25.740521 23396 rpcz_store.cc:276] 0809 15:46:24.035480 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:25.065152 (+1029672us) service_pool.cc:227] Handling call
0809 15:46:25.655690 (+590538us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:25.689142 (+ 33452us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:25.698145 (+ 9003us) spinlock_profiling.cc:243] Waited 564 ms on lock 0x7b5400193550. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e921fa09 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:46:25.698272 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1296395648}
I20260809 15:46:25.741207 25635 consensus_queue.cc:260] T 563daf42d7784db195ea99c3cb21b370 P 67b3aeadfa7a432abe8af4d292e04ab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:25.742362 25635 raft_consensus.cc:3037] T 563daf42d7784db195ea99c3cb21b370 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:25.708514 25457 raft_consensus.cc:2726] T 7660dfa2ef694927b8eda4100a2bcd9b P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260809 15:46:25.744275 24406 leader_election.cc:302] T c0741ab44d99439abc3aa3bdd27ee160 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:46:25.710964 25583 leader_election.cc:288] T 538a228ddfb54bc6a36528d5ff07597b P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:25.711186 25639 consensus_queue.cc:1046] T 7660dfa2ef694927b8eda4100a2bcd9b P 65c36d164b0540f099d592f38307ee6c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260809 15:46:25.712682 25590 raft_consensus.cc:3032] T 7660dfa2ef694927b8eda4100a2bcd9b P 65c36d164b0540f099d592f38307ee6c [term 1 LEADER]: Stepping down as leader of term 1
I20260809 15:46:25.713135 25573 leader_election.cc:288] T cef3e7277d7d4760824a3436ed4a847d P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 2 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
W20260809 15:46:25.738144 23314 debug-util.cc:398] Leaking SignalData structure 0x7b08000ea980 after lost signal to thread 23193
I20260809 15:46:25.715279 25366 leader_election.cc:288] T 6ae14a74a26f434db08c652e097385c8 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:25.668088 25593 raft_consensus.cc:504] T 4564dfc6d4094c1b9240b367ccecf3b1 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:25.681751 25519 leader_election.cc:288] T 297c82ed906d4b48b184985eed3de601 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 3 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:25.720078 23519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5c2e425a74b4e64ae70b5def750434c" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:25.756875 23395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cef3e7277d7d4760824a3436ed4a847d" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:25.757248 23519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cef3e7277d7d4760824a3436ed4a847d" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:25.722460 23394 raft_consensus.cc:1261] T 0acc47ce21c3422698d1ef7277c3ed63 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Refusing update from remote peer 67b3aeadfa7a432abe8af4d292e04ab9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 15:46:25.738193 25507 raft_consensus.cc:2726] T 7baa2492df0340c4a5ac241959ce1ac3 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 15:46:25.687475 25621 raft_consensus.cc:3037] T 294d4bcc1e7a477cb56455581277a60f P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:25.737937 25614 raft_consensus.cc:482] T a3622f6e3e3641cea1960e3a31498bb7 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Starting leader election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
W20260809 15:46:25.731889 23223 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.68:34689, user_credentials={real_user=slave}} blocked reactor thread for 66661.6us
I20260809 15:46:25.732393 23397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e4ffcf83c234ae7910ddef3ec0b4bda" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:25.736660 23529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998c6cee72cf40228c6dba064c0db2da" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:25.760864 23397 raft_consensus.cc:2454] T 8e4ffcf83c234ae7910ddef3ec0b4bda P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 2.
I20260809 15:46:25.743166 25369 raft_consensus.cc:482] T d2d4f749fcfd4fb3bfb8115af3471598 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Starting leader election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:46:25.736963 23517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0741ab44d99439abc3aa3bdd27ee160" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:25.736060 23351 leader_election.cc:302] T 907bacfa79d04a55a68c8dbe60056462 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 61e69695258f48baa984c9d395e8ce3f, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:25.755570 23521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df20671a7b874dbb98c0ef28371c1308" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
W20260809 15:46:25.752185 25477 consensus_meta.cc:287] T 953eea268e17491ca76f3a555860a512 P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent flushing consensus metadata: real 0.519s user 0.000s sys 0.019s
I20260809 15:46:25.762701 23351 leader_election.cc:302] T ba56725e42f74a1baa4c04d9660727f9 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c
I20260809 15:46:25.711815 25551 leader_election.cc:288] T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 election: Requested vote from peers 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:25.738634 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73f6f44c259547f1b05a3a434616fa54" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:25.752066 25467 raft_consensus.cc:2781] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260809 15:46:25.743331 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73f6f44c259547f1b05a3a434616fa54" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:25.741147 23404 raft_consensus.cc:1226] T a3622f6e3e3641cea1960e3a31498bb7 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Rejecting Update request from peer 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 1. Current term is 2. Ops: []
I20260809 15:46:25.745003 23273 raft_consensus.cc:1226] T a88421dba80d4d3388aa5eb62e23853b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 1. Current term is 2. Ops: []
I20260809 15:46:25.753010 25590 raft_consensus.cc:729] T 7660dfa2ef694927b8eda4100a2bcd9b P 65c36d164b0540f099d592f38307ee6c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:46:25.713441 25501 consensus_queue.cc:260] T 05ad7d8fdca64e3da527280ca5be9280 P 67b3aeadfa7a432abe8af4d292e04ab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:25.753126 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba56725e42f74a1baa4c04d9660727f9" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:25.754293 23522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "297c82ed906d4b48b184985eed3de601" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:25.737394 25529 raft_consensus.cc:2726] T 0a51c0baaa6e432997c9d3552af4f254 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 15:46:25.723836 23393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998c6cee72cf40228c6dba064c0db2da" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:25.726997 25465 raft_consensus.cc:504] T f7715d71dbac447b9351fc5eb36c5e80 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:46:25.745630 25569 consensus_meta.cc:287] T c5c2e425a74b4e64ae70b5def750434c P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent flushing consensus metadata: real 0.624s user 0.013s sys 0.003s
I20260809 15:46:25.741556 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3924b648461e463b9d7925e5671d68ae" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f"
I20260809 15:46:25.772847 25573 raft_consensus.cc:482] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
I20260809 15:46:25.723832 25603 raft_consensus.cc:504] T 297c82ed906d4b48b184985eed3de601 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:25.772747 25453 raft_consensus.cc:2781] T c0741ab44d99439abc3aa3bdd27ee160 P 61e69695258f48baa984c9d395e8ce3f [term 2 FOLLOWER]: Leader election won for term 2
I20260809 15:46:25.776152 25453 raft_consensus.cc:686] T c0741ab44d99439abc3aa3bdd27ee160 P 61e69695258f48baa984c9d395e8ce3f [term 2 LEADER]: Becoming Leader. State: Replica: 61e69695258f48baa984c9d395e8ce3f, State: Running, Role: LEADER
I20260809 15:46:25.776860 25603 leader_election.cc:288] T 297c82ed906d4b48b184985eed3de601 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:25.775513 25620 raft_consensus.cc:504] T d0721a4403a440f383318a3cea9d1442 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:25.777060 25453 consensus_queue.cc:237] T c0741ab44d99439abc3aa3bdd27ee160 P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:25.740017 25490 consensus_queue.cc:1035] T 297c82ed906d4b48b184985eed3de601 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
W20260809 15:46:25.765244 23353 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 69992.4us
I20260809 15:46:25.717479 25576 raft_consensus.cc:729] T 00a1881267a14ba8b78c81040f17a014 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Running, Role: LEADER
I20260809 15:46:25.753341 23528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5712f6b6fffc4cc5a8ddfbe63e4097df" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:25.761432 25614 raft_consensus.cc:3037] T a3622f6e3e3641cea1960e3a31498bb7 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:25.766278 25539 raft_consensus.cc:2781] T ba56725e42f74a1baa4c04d9660727f9 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260809 15:46:25.822342 25539 raft_consensus.cc:482] T ba56725e42f74a1baa4c04d9660727f9 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:25.822928 25539 raft_consensus.cc:3037] T ba56725e42f74a1baa4c04d9660727f9 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:25.764642 25369 raft_consensus.cc:3037] T d2d4f749fcfd4fb3bfb8115af3471598 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:25.764349 25470 raft_consensus.cc:2726] T 907bacfa79d04a55a68c8dbe60056462 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 15:46:25.762151 23392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "297c82ed906d4b48b184985eed3de601" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:25.776539 25569 raft_consensus.cc:504] T c5c2e425a74b4e64ae70b5def750434c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:25.766465 25467 raft_consensus.cc:482] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Starting leader election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
I20260809 15:46:25.829808 23392 raft_consensus.cc:2379] T 297c82ed906d4b48b184985eed3de601 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 in current term 3: Already voted for candidate c0f38c1908414566aa9115bb2cbdd85d in this term.
I20260809 15:46:25.829991 25467 raft_consensus.cc:3037] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:25.831530 25593 leader_election.cc:288] T 4564dfc6d4094c1b9240b367ccecf3b1 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 2 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:25.763742 23351 leader_election.cc:302] T 73f6f44c259547f1b05a3a434616fa54 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:25.833554 25569 leader_election.cc:288] T c5c2e425a74b4e64ae70b5def750434c P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 2 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
W20260809 15:46:25.833851 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 70181us
I20260809 15:46:25.834591 23351 leader_election.cc:302] T 6ae14a74a26f434db08c652e097385c8 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:25.835546 23351 leader_election.cc:302] T c5c2e425a74b4e64ae70b5def750434c P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c
I20260809 15:46:25.836637 23270 raft_consensus.cc:1226] T c5c2e425a74b4e64ae70b5def750434c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 1. Current term is 2. Ops: []
W20260809 15:46:25.768786 24410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 75580us
I20260809 15:46:25.813355 23472 leader_election.cc:302] T 0da4d5283c8a46878bd39b85e5c71f66 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:25.813534 25658 consensus_queue.cc:1046] T 472816994d814cb198b69e5248697440 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260809 15:46:25.740482 25587 consensus_queue.cc:260] T 30c1740ef6874df0a332fbd6fd4b03ce P 67b3aeadfa7a432abe8af4d292e04ab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:25.754968 25506 consensus_queue.cc:1046] T f9e7cadcd24d45fcbfa84209225e8f2d P 65c36d164b0540f099d592f38307ee6c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.015s
I20260809 15:46:25.816404 23283 raft_consensus.cc:1226] T f7715d71dbac447b9351fc5eb36c5e80 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 1. Current term is 2. Ops: []
I20260809 15:46:25.817879 25620 leader_election.cc:288] T d0721a4403a440f383318a3cea9d1442 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 2 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
W20260809 15:46:25.778493 23314 debug-util.cc:398] Leaking SignalData structure 0x7b080036ef40 after lost signal to thread 23320
I20260809 15:46:25.862648 23393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3e94b7e50754a81b3d72a07476cb7b9" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:25.863337 23400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ae14a74a26f434db08c652e097385c8" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:25.863782 23393 raft_consensus.cc:2379] T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c in current term 3: Already voted for candidate c0f38c1908414566aa9115bb2cbdd85d in this term.
I20260809 15:46:25.864614 23518 raft_consensus.cc:1226] T 0acc47ce21c3422698d1ef7277c3ed63 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Rejecting Update request from peer 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 1. Current term is 2. Ops: []
I20260809 15:46:25.864858 23393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4564dfc6d4094c1b9240b367ccecf3b1" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:25.865114 25576 consensus_queue.cc:260] T 00a1881267a14ba8b78c81040f17a014 P 67b3aeadfa7a432abe8af4d292e04ab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:25.866493 25576 raft_consensus.cc:3037] T 00a1881267a14ba8b78c81040f17a014 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:25.867100 23404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae6d2c08a7a94f58ac8dd698e3fd456c" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:25.868227 23392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "213543cdbe814c89a17a963469922c1b" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:25.826614 23353 leader_election.cc:302] T 05ad7d8fdca64e3da527280ca5be9280 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:46:25.766649 25477 raft_consensus.cc:504] T 953eea268e17491ca76f3a555860a512 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:46:25.870097 23353 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 43574.6us
I20260809 15:46:25.870713 23283 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53554 (request call id 1045) took 1979 ms. Trace:
I20260809 15:46:25.871065 23283 rpcz_store.cc:276] 0809 15:46:23.891393 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:25.231466 (+1340073us) service_pool.cc:227] Handling call
0809 15:46:25.252010 (+ 20544us) spinlock_profiling.cc:243] Waited 3.53 ms on lock 0x7b5800003398. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e613cf11 00007f34e62c0851 00007f34e62c0702 00007f34ef1b0b09 00007f34ee0ec730 00007f34ee1b279b 00007f34ee1b2821 00007f34ee1762b0 00007f34ee1617f9 00007f34ee1612a2 00007f34e7847684
0809 15:46:25.254224 (+ 2214us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:25.860345 (+606121us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:25.870500 (+ 10155us) spinlock_profiling.cc:243] Waited 547 ms on lock 0x7b54002f44d4. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e9220f51 00007f34e921f99e 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772 00007f34e764a94c 00007f34e7649d86
0809 15:46:25.870659 (+ 159us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1266607872}
W20260809 15:46:25.841603 23470 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 101835us
I20260809 15:46:25.839916 25507 raft_consensus.cc:3037] T 73f6f44c259547f1b05a3a434616fa54 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:25.846333 25529 raft_consensus.cc:2726] T 6ae14a74a26f434db08c652e097385c8 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260809 15:46:25.847594 23395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba56725e42f74a1baa4c04d9660727f9" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:25.848856 23273 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53554 (request call id 1042) took 1970 ms. Trace:
I20260809 15:46:25.894232 23273 rpcz_store.cc:276] 0809 15:46:23.878451 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:25.189639 (+1311188us) service_pool.cc:227] Handling call
0809 15:46:25.254658 (+ 65019us) spinlock_profiling.cc:243] Waited 2.76 ms on lock 0x7b5800003398. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e613cf11 00007f34e62c0851 00007f34e62c0702 00007f34ef1b0b09 00007f34ee0ec730 00007f34ee1b279b 00007f34ee1b2821 00007f34ee1762b0 00007f34ee1617f9 00007f34ee1612a2 00007f34e7847684
0809 15:46:25.254823 (+ 165us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:25.767504 (+512681us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:25.848634 (+ 81130us) spinlock_profiling.cc:243] Waited 490 ms on lock 0x7b5400116a54. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e9220f51 00007f34e921f99e 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772 00007f34e764a94c 00007f34e7649d86
0809 15:46:25.848800 (+ 166us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1133279872}
I20260809 15:46:25.849562 23396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df20671a7b874dbb98c0ef28371c1308" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:25.828032 25590 consensus_queue.cc:260] T 7660dfa2ef694927b8eda4100a2bcd9b P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:25.891314 23470 leader_election.cc:302] T 0c29937a80724ddb9d07a27534a146f3 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters: c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:25.851704 23520 raft_consensus.cc:2454] T c0741ab44d99439abc3aa3bdd27ee160 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 2.
I20260809 15:46:25.784513 25460 leader_election.cc:288] T 2db057fd10e149a9895dff6ef8c9f038 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 2 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:25.866328 23393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0721a4403a440f383318a3cea9d1442" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:25.857638 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05ad7d8fdca64e3da527280ca5be9280" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:25.857954 23397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0acc47ce21c3422698d1ef7277c3ed63" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:25.899978 23277 raft_consensus.cc:1226] T 953eea268e17491ca76f3a555860a512 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 1. Current term is 2. Ops: []
I20260809 15:46:25.858256 23394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "538a228ddfb54bc6a36528d5ff07597b" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:25.775614 25573 raft_consensus.cc:504] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
W20260809 15:46:25.859158 24406 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 114952us
I20260809 15:46:25.862120 23270 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53554 (request call id 1043) took 1973 ms. Trace:
I20260809 15:46:25.852757 25646 raft_consensus.cc:2781] T c5c2e425a74b4e64ae70b5def750434c P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260809 15:46:25.890990 23472 leader_election.cc:302] T d3e94b7e50754a81b3d72a07476cb7b9 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:25.874539 23522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3622f6e3e3641cea1960e3a31498bb7" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:25.904100 23400 raft_consensus.cc:2379] T 6ae14a74a26f434db08c652e097385c8 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c in current term 2: Already voted for candidate c0f38c1908414566aa9115bb2cbdd85d in this term.
I20260809 15:46:25.873849 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3e94b7e50754a81b3d72a07476cb7b9" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:25.892724 23528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "538a228ddfb54bc6a36528d5ff07597b" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:25.893580 23519 raft_consensus.cc:1226] T 4564dfc6d4094c1b9240b367ccecf3b1 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Rejecting Update request from peer c0f38c1908414566aa9115bb2cbdd85d for earlier term 1. Current term is 2. Ops: []
I20260809 15:46:25.859946 25366 raft_consensus.cc:3037] T 0da4d5283c8a46878bd39b85e5c71f66 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:25.891868 25559 raft_consensus.cc:2781] T 05ad7d8fdca64e3da527280ca5be9280 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Leader election won for term 2
I20260809 15:46:25.898038 23518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0da4d5283c8a46878bd39b85e5c71f66" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:25.900252 23470 leader_election.cc:302] T 28e17dcb5efe45b8b6a8c42da03b4819 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters: c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:25.900801 23394 raft_consensus.cc:2379] T 538a228ddfb54bc6a36528d5ff07597b P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c in current term 3: Already voted for candidate c0f38c1908414566aa9115bb2cbdd85d in this term.
I20260809 15:46:25.902055 23270 rpcz_store.cc:276] 0809 15:46:23.888890 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:25.194151 (+1305261us) service_pool.cc:227] Handling call
0809 15:46:25.194437 (+ 286us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:25.836999 (+642562us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:25.861944 (+ 24945us) spinlock_profiling.cc:243] Waited 642 ms on lock 0x7b5400044054. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e9220f51 00007f34e921f99e 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772 00007f34e764a94c 00007f34e7649d86
0809 15:46:25.862079 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1476586880}
I20260809 15:46:25.898558 25506 raft_consensus.cc:3032] T f9e7cadcd24d45fcbfa84209225e8f2d P 65c36d164b0540f099d592f38307ee6c [term 1 LEADER]: Stepping down as leader of term 1
I20260809 15:46:25.906369 23528 raft_consensus.cc:2379] T 538a228ddfb54bc6a36528d5ff07597b P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d in current term 3: Already voted for candidate 65c36d164b0540f099d592f38307ee6c in this term.
I20260809 15:46:25.906682 23527 raft_consensus.cc:2379] T d3e94b7e50754a81b3d72a07476cb7b9 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d in current term 3: Already voted for candidate 65c36d164b0540f099d592f38307ee6c in this term.
I20260809 15:46:25.907789 23519 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f9e7cadcd24d45fcbfa84209225e8f2d"
dest_uuid: "65c36d164b0540f099d592f38307ee6c"
mode: GRACEFUL
new_leader_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
from {username='slave'} at 127.0.0.1:48888
I20260809 15:46:25.827455 25497 raft_consensus.cc:504] T 953eea268e17491ca76f3a555860a512 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:25.897050 25587 raft_consensus.cc:3037] T 30c1740ef6874df0a332fbd6fd4b03ce P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:25.915756 23394 raft_consensus.cc:1226] T 297c82ed906d4b48b184985eed3de601 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260809 15:46:25.897177 25488 consensus_queue.cc:1046] T d3e94b7e50754a81b3d72a07476cb7b9 P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.021s
I20260809 15:46:25.921013 25620 raft_consensus.cc:2726] T d3e94b7e50754a81b3d72a07476cb7b9 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260809 15:46:25.921226 25639 raft_consensus.cc:2781] T 28e17dcb5efe45b8b6a8c42da03b4819 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260809 15:46:25.924072 25639 raft_consensus.cc:482] T 28e17dcb5efe45b8b6a8c42da03b4819 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Starting leader election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:46:25.924535 25639 raft_consensus.cc:3037] T 28e17dcb5efe45b8b6a8c42da03b4819 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:25.919034 23470 leader_election.cc:302] T 07fcabb569414bddbd39df6cedd24e64 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters: c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:25.923859 25457 consensus_queue.cc:1046] T 4564dfc6d4094c1b9240b367ccecf3b1 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260809 15:46:25.907629 25658 raft_consensus.cc:2781] T 0c29937a80724ddb9d07a27534a146f3 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260809 15:46:25.925812 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 89406.4us
I20260809 15:46:25.930646 23351 leader_election.cc:302] T 563daf42d7784db195ea99c3cb21b370 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 61e69695258f48baa984c9d395e8ce3f, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:25.928081 23353 leader_election.cc:302] T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c
I20260809 15:46:25.908416 25646 raft_consensus.cc:482] T c5c2e425a74b4e64ae70b5def750434c P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
W20260809 15:46:25.919652 23314 debug-util.cc:398] Leaking SignalData structure 0x7b0800341a40 after lost signal to thread 23443
W20260809 15:46:25.934072 23520 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44018 (request call id 60) took 1601 ms (client timeout 2020 ms). Trace:
I20260809 15:46:25.923930 25638 consensus_queue.cc:1046] T 297c82ed906d4b48b184985eed3de601 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.011s
I20260809 15:46:25.936551 25593 raft_consensus.cc:3032] T 472816994d814cb198b69e5248697440 P 65c36d164b0540f099d592f38307ee6c [term 1 LEADER]: Stepping down as leader of term 1
I20260809 15:46:25.919283 25506 raft_consensus.cc:729] T f9e7cadcd24d45fcbfa84209225e8f2d P 65c36d164b0540f099d592f38307ee6c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:46:25.938663 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "297c82ed906d4b48b184985eed3de601" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:25.932200 23529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3924b648461e463b9d7925e5671d68ae" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:25.922542 25559 raft_consensus.cc:686] T 05ad7d8fdca64e3da527280ca5be9280 P c0f38c1908414566aa9115bb2cbdd85d [term 2 LEADER]: Becoming Leader. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:46:25.944530 23399 raft_consensus.cc:2454] T e71a05ba1d4d42bd95c3f85f9313399e P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 2.
I20260809 15:46:25.944767 25532 leader_election.cc:288] T a88421dba80d4d3388aa5eb62e23853b P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 2 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:25.930058 25658 raft_consensus.cc:482] T 0c29937a80724ddb9d07a27534a146f3 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Starting leader election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:46:25.946233 25483 raft_consensus.cc:2726] T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20260809 15:46:25.952679 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 199746us
I20260809 15:46:25.943938 25649 raft_consensus.cc:3032] T d3e94b7e50754a81b3d72a07476cb7b9 P 61e69695258f48baa984c9d395e8ce3f [term 2 LEADER]: Stepping down as leader of term 2
I20260809 15:46:25.946712 25646 raft_consensus.cc:3037] T c5c2e425a74b4e64ae70b5def750434c P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:25.953341 25593 raft_consensus.cc:729] T 472816994d814cb198b69e5248697440 P 65c36d164b0540f099d592f38307ee6c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:46:25.946244 23277 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53554 (request call id 1049) took 1970 ms. Trace:
I20260809 15:46:25.960041 23351 leader_election.cc:302] T 0acc47ce21c3422698d1ef7277c3ed63 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:25.947577 23470 leader_election.cc:302] T 213543cdbe814c89a17a963469922c1b P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters: c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:25.946555 25628 raft_consensus.cc:2726] T 563daf42d7784db195ea99c3cb21b370 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260809 15:46:25.945749 25650 raft_consensus.cc:2781] T 07fcabb569414bddbd39df6cedd24e64 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260809 15:46:25.968632 23470 leader_election.cc:302] T 16ca412a0dae4feebe85dd01bf1a9173 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 61e69695258f48baa984c9d395e8ce3f, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:25.970490 23470 leader_election.cc:302] T ae6d2c08a7a94f58ac8dd698e3fd456c P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters: c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:25.965696 23353 leader_election.cc:302] T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c
I20260809 15:46:25.967789 23277 rpcz_store.cc:276] 0809 15:46:23.975548 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:25.381700 (+1406152us) service_pool.cc:227] Handling call
0809 15:46:25.429379 (+ 47679us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:25.901278 (+471899us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:25.941616 (+ 40338us) spinlock_profiling.cc:243] Waited 470 ms on lock 0x7b54003a8ed4. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e9220f51 00007f34e921f99e 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772 00007f34e764a94c 00007f34e7649d86
0809 15:46:25.946191 (+ 4575us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1081900288}
I20260809 15:46:25.971873 23470 leader_election.cc:302] T e14d118b2eaa42aa8a7cd4025aba7128 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 61e69695258f48baa984c9d395e8ce3f
W20260809 15:46:25.964920 23228 connection.cc:232] RPC call timeout handler was delayed by -1.3666331339627504s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1845s for a response
W20260809 15:46:25.972544 23228 connection.cc:232] RPC call timeout handler was delayed by -1.3632274158298969s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1836s for a response
I20260809 15:46:25.956115 25465 leader_election.cc:288] T f7715d71dbac447b9351fc5eb36c5e80 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 2 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:25.971192 25457 raft_consensus.cc:2726] T 0acc47ce21c3422698d1ef7277c3ed63 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 15:46:25.975381 23351 leader_election.cc:302] T 2424fd9bd7f14e508fd106cbfda786a4 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:25.960923 25501 raft_consensus.cc:3037] T 05ad7d8fdca64e3da527280ca5be9280 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:25.964004 25658 raft_consensus.cc:3037] T 0c29937a80724ddb9d07a27534a146f3 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Advancing to term 2
W20260809 15:46:25.955420 24410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 56892.3us
W20260809 15:46:25.972792 23228 connection.cc:232] RPC call timeout handler was delayed by -1.3629271178506315s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16040000000000001s for a response
I20260809 15:46:25.946712 25478 raft_consensus.cc:504] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 61e69695258f48baa984c9d395e8ce3f [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
W20260809 15:46:25.980389 23228 connection.cc:232] RPC call timeout handler was delayed by -1.3597885498311371s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15990000000000001s for a response
W20260809 15:46:25.948069 23520 rpcz_store.cc:269] 0809 15:46:24.334773 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:24.984461 (+649688us) service_pool.cc:227] Handling call
0809 15:46:25.933835 (+949374us) spinlock_profiling.cc:243] Waited 652 ms on lock 0x7b540031fe54. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e92242ae 00007f34ee161c27 00007f34e7849c04 00007f34e7849b92 00007f34e7849ac1 00007f34e7849a3c 00007f34e7848cf2 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:46:25.933997 (+ 162us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1500485888}
W20260809 15:46:25.975479 23314 debug-util.cc:398] Leaking SignalData structure 0x7b08002fcf00 after lost signal to thread 23567
I20260809 15:46:25.966297 25649 raft_consensus.cc:729] T d3e94b7e50754a81b3d72a07476cb7b9 P 61e69695258f48baa984c9d395e8ce3f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 61e69695258f48baa984c9d395e8ce3f, State: Running, Role: LEADER
I20260809 15:46:25.966809 25637 tablet_copy_client.cc:316] T 0acc47ce21c3422698d1ef7277c3ed63 P 61e69695258f48baa984c9d395e8ce3f: tablet copy: Beginning tablet copy session from remote peer at address 127.20.202.65:35459
W20260809 15:46:25.985304 23228 connection.cc:232] RPC call timeout handler was delayed by -1.0666475330945104s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1957s for a response
I20260809 15:46:25.971613 25583 raft_consensus.cc:2726] T 16ca412a0dae4feebe85dd01bf1a9173 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260809 15:46:25.971318 25638 raft_consensus.cc:2781] T 213543cdbe814c89a17a963469922c1b P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260809 15:46:25.991194 25621 raft_consensus.cc:504] T 294d4bcc1e7a477cb56455581277a60f P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:25.973290 25650 raft_consensus.cc:482] T 07fcabb569414bddbd39df6cedd24e64 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Starting leader election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:46:25.977720 25529 raft_consensus.cc:2726] T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20260809 15:46:25.994750 23228 connection.cc:232] RPC call timeout handler was delayed by -1.0275300699286163s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1595s for a response
I20260809 15:46:25.979807 25506 consensus_queue.cc:260] T f9e7cadcd24d45fcbfa84209225e8f2d P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:25.995396 25559 consensus_queue.cc:237] T 05ad7d8fdca64e3da527280ca5be9280 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:25.981319 25490 raft_consensus.cc:2781] T ae6d2c08a7a94f58ac8dd698e3fd456c P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260809 15:46:25.981793 25457 raft_consensus.cc:2726] T 2424fd9bd7f14e508fd106cbfda786a4 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20260809 15:46:25.986747 23314 debug-util.cc:398] Leaking SignalData structure 0x7b0800329e00 after lost signal to thread 23568
I20260809 15:46:25.992513 25369 raft_consensus.cc:504] T d2d4f749fcfd4fb3bfb8115af3471598 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:46:26.023746 23228 connection.cc:232] RPC call timeout handler was delayed by -1.0180076758842915s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16010000000000002s for a response
W20260809 15:46:26.025328 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 50017.8us
I20260809 15:46:25.997692 24410 leader_election.cc:302] T 4bcac4b24e7a4448abdc934bba5a624e P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c; no voters:
I20260809 15:46:26.028218 23351 leader_election.cc:302] T 30c1740ef6874df0a332fbd6fd4b03ce P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:26.002156 24452 raft_consensus.cc:1226] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 61e69695258f48baa984c9d395e8ce3f [term 2 FOLLOWER]: Rejecting Update request from peer c0f38c1908414566aa9115bb2cbdd85d for earlier term 1. Current term is 2. Ops: []
I20260809 15:46:26.034291 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "907bacfa79d04a55a68c8dbe60056462" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:25.998657 25477 leader_election.cc:288] T 953eea268e17491ca76f3a555860a512 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 2 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:26.001819 25650 raft_consensus.cc:3037] T 07fcabb569414bddbd39df6cedd24e64 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:25.999082 25638 raft_consensus.cc:482] T 213543cdbe814c89a17a963469922c1b P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Starting leader election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:46:25.992820 25539 raft_consensus.cc:504] T ba56725e42f74a1baa4c04d9660727f9 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:26.023168 25583 raft_consensus.cc:2781] T e14d118b2eaa42aa8a7cd4025aba7128 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260809 15:46:26.026468 25490 raft_consensus.cc:482] T ae6d2c08a7a94f58ac8dd698e3fd456c P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Starting leader election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:46:26.046528 23529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "953eea268e17491ca76f3a555860a512" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:26.048460 23273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f9d6565aa4c45a3b3f01210d0eb71f7" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:26.045257 25497 leader_election.cc:288] T 953eea268e17491ca76f3a555860a512 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 2 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:26.051855 25478 leader_election.cc:288] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 2 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:26.045519 25583 raft_consensus.cc:482] T e14d118b2eaa42aa8a7cd4025aba7128 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Starting leader election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
I20260809 15:46:26.052701 25583 raft_consensus.cc:3037] T e14d118b2eaa42aa8a7cd4025aba7128 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:26.044602 25663 raft_consensus.cc:2781] T 4bcac4b24e7a4448abdc934bba5a624e P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260809 15:46:26.048695 23351 leader_election.cc:302] T f9e7cadcd24d45fcbfa84209225e8f2d P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c
I20260809 15:46:26.054760 23351 leader_election.cc:302] T cef3e7277d7d4760824a3436ed4a847d P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:26.053498 25663 raft_consensus.cc:482] T 4bcac4b24e7a4448abdc934bba5a624e P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Starting leader election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:46:26.055963 25663 raft_consensus.cc:3037] T 4bcac4b24e7a4448abdc934bba5a624e P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:26.035416 25573 leader_election.cc:288] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:26.059433 25497 raft_consensus.cc:3037] T cef3e7277d7d4760824a3436ed4a847d P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:26.063524 23273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e71a05ba1d4d42bd95c3f85f9313399e" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:26.064538 23277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3be1e356b254d22b995be32f1c12f6f" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:26.065922 23274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00a1881267a14ba8b78c81040f17a014" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:26.067584 23280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e4ffcf83c234ae7910ddef3ec0b4bda" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:26.068739 23280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3622f6e3e3641cea1960e3a31498bb7" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:26.069852 23273 raft_consensus.cc:2379] T e71a05ba1d4d42bd95c3f85f9313399e P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 61e69695258f48baa984c9d395e8ce3f in current term 2: Already voted for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in this term.
I20260809 15:46:26.069787 23280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "953eea268e17491ca76f3a555860a512" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:26.070508 23280 raft_consensus.cc:2379] T 953eea268e17491ca76f3a555860a512 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d in current term 2: Already voted for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in this term.
I20260809 15:46:26.071581 23283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3622f6e3e3641cea1960e3a31498bb7" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:26.042734 25638 raft_consensus.cc:3037] T 213543cdbe814c89a17a963469922c1b P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:26.073140 23283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "297c82ed906d4b48b184985eed3de601" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:26.073832 23283 raft_consensus.cc:2379] T 297c82ed906d4b48b184985eed3de601 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d in current term 3: Already voted for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in this term.
I20260809 15:46:26.042542 25457 raft_consensus.cc:2726] T 30c1740ef6874df0a332fbd6fd4b03ce P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260809 15:46:26.071183 23280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0da4d5283c8a46878bd39b85e5c71f66" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
W20260809 15:46:26.040843 24410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 52632.6us
I20260809 15:46:26.074635 23285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2db057fd10e149a9895dff6ef8c9f038" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:26.081215 23285 raft_consensus.cc:2379] T 2db057fd10e149a9895dff6ef8c9f038 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d in current term 2: Already voted for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in this term.
I20260809 15:46:26.075335 23286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7715d71dbac447b9351fc5eb36c5e80" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:26.076872 25470 raft_consensus.cc:979] T 07fcabb569414bddbd39df6cedd24e64 P c0f38c1908414566aa9115bb2cbdd85d: : Instructing follower 67b3aeadfa7a432abe8af4d292e04ab9 to start an election
I20260809 15:46:26.077296 25525 raft_consensus.cc:979] T ae6d2c08a7a94f58ac8dd698e3fd456c P c0f38c1908414566aa9115bb2cbdd85d: : Instructing follower 65c36d164b0540f099d592f38307ee6c to start an election
I20260809 15:46:26.049463 25490 raft_consensus.cc:3037] T ae6d2c08a7a94f58ac8dd698e3fd456c P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:26.081039 23400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3be1e356b254d22b995be32f1c12f6f" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:26.087568 23394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2d4f749fcfd4fb3bfb8115af3471598" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:26.074690 23282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a88421dba80d4d3388aa5eb62e23853b" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:26.044910 25506 raft_consensus.cc:3037] T f9e7cadcd24d45fcbfa84209225e8f2d P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:25.999157 25593 consensus_queue.cc:260] T 472816994d814cb198b69e5248697440 P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
W20260809 15:46:26.072060 23314 debug-util.cc:398] Leaking SignalData structure 0x7b080031f8a0 after lost signal to thread 24513
I20260809 15:46:26.084478 23277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998c6cee72cf40228c6dba064c0db2da" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:26.066758 25483 raft_consensus.cc:2781] T f9e7cadcd24d45fcbfa84209225e8f2d P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Leader election won for term 2
I20260809 15:46:26.086809 23273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "538a228ddfb54bc6a36528d5ff07597b" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:26.089360 23282 raft_consensus.cc:2379] T a88421dba80d4d3388aa5eb62e23853b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d in current term 2: Already voted for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in this term.
I20260809 15:46:26.090036 23529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c48a3049897452383257aa757834fb5" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:26.078171 25590 raft_consensus.cc:3037] T 7660dfa2ef694927b8eda4100a2bcd9b P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:26.089990 23399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e71a05ba1d4d42bd95c3f85f9313399e" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:26.091796 23286 raft_consensus.cc:2379] T f7715d71dbac447b9351fc5eb36c5e80 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d in current term 2: Already voted for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in this term.
I20260809 15:46:26.080072 25369 leader_election.cc:288] T d2d4f749fcfd4fb3bfb8115af3471598 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 2 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:26.096964 25483 raft_consensus.cc:686] T f9e7cadcd24d45fcbfa84209225e8f2d P c0f38c1908414566aa9115bb2cbdd85d [term 2 LEADER]: Becoming Leader. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:46:26.098582 23277 raft_consensus.cc:2379] T 998c6cee72cf40228c6dba064c0db2da P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d in current term 4: Already voted for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in this term.
I20260809 15:46:26.097405 23520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba56725e42f74a1baa4c04d9660727f9" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:26.087749 25539 leader_election.cc:288] T ba56725e42f74a1baa4c04d9660727f9 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 2 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:26.108879 23285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee02fff2c9c48a19f43eb358a89e7b0" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:26.110687 23285 raft_consensus.cc:2362] T cee02fff2c9c48a19f43eb358a89e7b0 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 2. Re-sending same reply.
I20260809 15:46:26.088414 23276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df20671a7b874dbb98c0ef28371c1308" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:26.108811 23280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "563daf42d7784db195ea99c3cb21b370" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:26.093573 25527 raft_consensus.cc:1067] T 07fcabb569414bddbd39df6cedd24e64 P c0f38c1908414566aa9115bb2cbdd85d [term 1 LEADER]: Signalling peer 67b3aeadfa7a432abe8af4d292e04ab9 to start an election
I20260809 15:46:26.114552 23529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee02fff2c9c48a19f43eb358a89e7b0" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:26.128281 23529 raft_consensus.cc:2454] T cee02fff2c9c48a19f43eb358a89e7b0 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 2.
I20260809 15:46:26.119150 25366 raft_consensus.cc:2726] T 0da4d5283c8a46878bd39b85e5c71f66 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260809 15:46:26.100970 23314 debug-util.cc:398] Leaking SignalData structure 0x7b08000ffb20 after lost signal to thread 24516
I20260809 15:46:26.098106 25470 raft_consensus.cc:1067] T ae6d2c08a7a94f58ac8dd698e3fd456c P c0f38c1908414566aa9115bb2cbdd85d [term 1 LEADER]: Signalling peer 65c36d164b0540f099d592f38307ee6c to start an election
I20260809 15:46:26.057348 25649 consensus_queue.cc:260] T d3e94b7e50754a81b3d72a07476cb7b9 P 61e69695258f48baa984c9d395e8ce3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:26.143712 25621 leader_election.cc:288] T 294d4bcc1e7a477cb56455581277a60f P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:26.143644 25483 consensus_queue.cc:237] T f9e7cadcd24d45fcbfa84209225e8f2d P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:26.144375 25467 raft_consensus.cc:504] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:26.147215 25507 raft_consensus.cc:2726] T 73f6f44c259547f1b05a3a434616fa54 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 15:46:26.154035 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "294d4bcc1e7a477cb56455581277a60f" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f"
I20260809 15:46:26.144515 23399 raft_consensus.cc:2379] T e71a05ba1d4d42bd95c3f85f9313399e P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 61e69695258f48baa984c9d395e8ce3f in current term 2: Already voted for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in this term.
I20260809 15:46:26.155196 23280 raft_consensus.cc:2454] T 563daf42d7784db195ea99c3cb21b370 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 2.
I20260809 15:46:26.155699 23228 leader_election.cc:302] T 4564dfc6d4094c1b9240b367ccecf3b1 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:26.138873 25593 raft_consensus.cc:3037] T 472816994d814cb198b69e5248697440 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:26.164199 23228 leader_election.cc:302] T d2d4f749fcfd4fb3bfb8115af3471598 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:26.167321 23400 raft_consensus.cc:1226] T ba56725e42f74a1baa4c04d9660727f9 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 1. Current term is 2. Ops: []
I20260809 15:46:26.168295 23394 raft_consensus.cc:1226] T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Rejecting Update request from peer 61e69695258f48baa984c9d395e8ce3f for earlier term 2. Current term is 3. Ops: []
I20260809 15:46:26.162524 23276 raft_consensus.cc:2379] T df20671a7b874dbb98c0ef28371c1308 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d in current term 2: Already voted for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in this term.
I20260809 15:46:26.172003 24471 raft_consensus.cc:1226] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 61e69695258f48baa984c9d395e8ce3f [term 2 FOLLOWER]: Rejecting Update request from peer c0f38c1908414566aa9115bb2cbdd85d for earlier term 1. Current term is 2. Ops: []
I20260809 15:46:26.172791 23228 leader_election.cc:302] T d0721a4403a440f383318a3cea9d1442 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:26.165288 25465 raft_consensus.cc:2726] T 4564dfc6d4094c1b9240b367ccecf3b1 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 15:46:26.166724 25649 raft_consensus.cc:3037] T d3e94b7e50754a81b3d72a07476cb7b9 P 61e69695258f48baa984c9d395e8ce3f [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:26.167394 25500 raft_consensus.cc:3032] T 4564dfc6d4094c1b9240b367ccecf3b1 P c0f38c1908414566aa9115bb2cbdd85d [term 1 LEADER]: Stepping down as leader of term 1
I20260809 15:46:26.176424 23228 leader_election.cc:302] T 4b30f28311964a0b9149a50495e83d8f P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:26.168463 25532 raft_consensus.cc:2726] T d2d4f749fcfd4fb3bfb8115af3471598 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260809 15:46:26.177577 23470 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 205747us
I20260809 15:46:26.170567 23393 raft_consensus.cc:1226] T 73f6f44c259547f1b05a3a434616fa54 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 1. Current term is 2. Ops: []
I20260809 15:46:26.170915 23298 tablet_copy_service.cc:140] P 67b3aeadfa7a432abe8af4d292e04ab9: Received BeginTabletCopySession request for tablet 0acc47ce21c3422698d1ef7277c3ed63 from peer 61e69695258f48baa984c9d395e8ce3f ({username='slave'} at 127.0.0.1:55434)
I20260809 15:46:26.179198 25616 raft_consensus.cc:3032] T 297c82ed906d4b48b184985eed3de601 P 65c36d164b0540f099d592f38307ee6c [term 2 LEADER]: Stepping down as leader of term 2
I20260809 15:46:26.179592 25669 consensus_queue.cc:1046] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260809 15:46:26.182467 23519 raft_consensus.cc:595] T f9e7cadcd24d45fcbfa84209225e8f2d P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Received request to transfer leadership to TS 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:26.192816 25616 raft_consensus.cc:729] T 297c82ed906d4b48b184985eed3de601 P 65c36d164b0540f099d592f38307ee6c [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:46:26.192723 23298 tablet_copy_service.cc:161] P 67b3aeadfa7a432abe8af4d292e04ab9: Beginning new tablet copy session on tablet 0acc47ce21c3422698d1ef7277c3ed63 from peer 61e69695258f48baa984c9d395e8ce3f at {username='slave'} at 127.0.0.1:55434: session id = 61e69695258f48baa984c9d395e8ce3f-0acc47ce21c3422698d1ef7277c3ed63
I20260809 15:46:26.194182 25616 consensus_queue.cc:260] T 297c82ed906d4b48b184985eed3de601 P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:26.195734 25616 raft_consensus.cc:3037] T 297c82ed906d4b48b184985eed3de601 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:26.192991 23519 raft_consensus.cc:603] T f9e7cadcd24d45fcbfa84209225e8f2d P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260809 15:46:26.186411 23228 leader_election.cc:302] T 213543cdbe814c89a17a963469922c1b P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:26.190339 23470 leader_election.cc:302] T 2424fd9bd7f14e508fd106cbfda786a4 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:26.198208 23529 raft_consensus.cc:1226] T d0721a4403a440f383318a3cea9d1442 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Rejecting Update request from peer c0f38c1908414566aa9115bb2cbdd85d for earlier term 1. Current term is 2. Ops: []
I20260809 15:46:26.190501 25646 raft_consensus.cc:504] T c5c2e425a74b4e64ae70b5def750434c P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:26.200440 25627 consensus_queue.cc:1046] T d0721a4403a440f383318a3cea9d1442 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260809 15:46:26.201299 25646 leader_election.cc:288] T c5c2e425a74b4e64ae70b5def750434c P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 2 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:26.173907 25477 raft_consensus.cc:2726] T d0721a4403a440f383318a3cea9d1442 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 15:46:26.170624 25621 consensus_queue.cc:1046] T ba56725e42f74a1baa4c04d9660727f9 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.021s
I20260809 15:46:26.201736 25555 raft_consensus.cc:3032] T d0721a4403a440f383318a3cea9d1442 P c0f38c1908414566aa9115bb2cbdd85d [term 1 LEADER]: Stepping down as leader of term 1
I20260809 15:46:26.178838 23399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7660dfa2ef694927b8eda4100a2bcd9b" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:26.203140 23228 leader_election.cc:302] T 99bdb93503db4de4a1cfb0161ea0c07c P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:26.207698 23528 raft_consensus.cc:1226] T d2d4f749fcfd4fb3bfb8115af3471598 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Rejecting Update request from peer c0f38c1908414566aa9115bb2cbdd85d for earlier term 1. Current term is 2. Ops: []
I20260809 15:46:26.204209 25532 raft_consensus.cc:2726] T 213543cdbe814c89a17a963469922c1b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 15:46:26.208777 25477 raft_consensus.cc:2726] T 99bdb93503db4de4a1cfb0161ea0c07c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 15:46:26.209340 25555 raft_consensus.cc:729] T d0721a4403a440f383318a3cea9d1442 P c0f38c1908414566aa9115bb2cbdd85d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:46:26.210012 23521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7660dfa2ef694927b8eda4100a2bcd9b" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:26.210011 23399 raft_consensus.cc:2379] T 7660dfa2ef694927b8eda4100a2bcd9b P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 61e69695258f48baa984c9d395e8ce3f in current term 2: Already voted for candidate c0f38c1908414566aa9115bb2cbdd85d in this term.
I20260809 15:46:26.191725 23520 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ae6d2c08a7a94f58ac8dd698e3fd456c"
dest_uuid: "65c36d164b0540f099d592f38307ee6c"
from {username='slave'} at 127.0.0.1:56766
I20260809 15:46:26.206593 23228 leader_election.cc:302] T 0c29937a80724ddb9d07a27534a146f3 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:26.185701 25500 raft_consensus.cc:729] T 4564dfc6d4094c1b9240b367ccecf3b1 P c0f38c1908414566aa9115bb2cbdd85d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:46:26.191905 25587 raft_consensus.cc:2726] T 4b30f28311964a0b9149a50495e83d8f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 15:46:26.218171 23228 leader_election.cc:302] T d1a134ba33ea4076bbaac0ae460c17ba P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:26.222476 23529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5c2e425a74b4e64ae70b5def750434c" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:26.216670 25621 raft_consensus.cc:3037] T 2424fd9bd7f14e508fd106cbfda786a4 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:26.219323 25477 raft_consensus.cc:2726] T 0c29937a80724ddb9d07a27534a146f3 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 15:46:26.202570 25639 raft_consensus.cc:504] T 28e17dcb5efe45b8b6a8c42da03b4819 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:26.223912 25506 raft_consensus.cc:3032] T ba56725e42f74a1baa4c04d9660727f9 P 65c36d164b0540f099d592f38307ee6c [term 1 LEADER]: Stepping down as leader of term 1
I20260809 15:46:26.233922 23228 leader_election.cc:302] T ae6d2c08a7a94f58ac8dd698e3fd456c P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:26.224718 23399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad8a3b25d7b49e5a2142d5709edd0c7" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:26.216449 25628 consensus_queue.cc:1046] T d2d4f749fcfd4fb3bfb8115af3471598 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260809 15:46:26.227046 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5712f6b6fffc4cc5a8ddfbe63e4097df" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f"
I20260809 15:46:26.227499 25467 leader_election.cc:288] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:26.228065 23528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad8a3b25d7b49e5a2142d5709edd0c7" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:26.198207 25497 raft_consensus.cc:2726] T cef3e7277d7d4760824a3436ed4a847d P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 15:46:26.242214 23400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a823d35f4f348d1b52ba9e6d80b00d2" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:26.240051 25506 raft_consensus.cc:729] T ba56725e42f74a1baa4c04d9660727f9 P 65c36d164b0540f099d592f38307ee6c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:46:26.234284 25532 raft_consensus.cc:2726] T d1a134ba33ea4076bbaac0ae460c17ba P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 15:46:26.258397 25470 raft_consensus.cc:3032] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P c0f38c1908414566aa9115bb2cbdd85d [term 1 LEADER]: Stepping down as leader of term 1
I20260809 15:46:26.258906 25470 raft_consensus.cc:729] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P c0f38c1908414566aa9115bb2cbdd85d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:46:26.259388 23518 raft_consensus.cc:1226] T 294d4bcc1e7a477cb56455581277a60f P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Rejecting Update request from peer 61e69695258f48baa984c9d395e8ce3f for earlier term 2. Current term is 3. Ops: []
I20260809 15:46:26.260010 25470 consensus_queue.cc:260] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P c0f38c1908414566aa9115bb2cbdd85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:26.259413 25506 consensus_queue.cc:260] T ba56725e42f74a1baa4c04d9660727f9 P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:26.261598 25470 raft_consensus.cc:3037] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:26.255259 23394 raft_consensus.cc:1226] T cef3e7277d7d4760824a3436ed4a847d P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 1. Current term is 2. Ops: []
I20260809 15:46:26.263165 25639 leader_election.cc:288] T 28e17dcb5efe45b8b6a8c42da03b4819 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 2 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:26.263844 25500 consensus_queue.cc:260] T 4564dfc6d4094c1b9240b367ccecf3b1 P c0f38c1908414566aa9115bb2cbdd85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:26.232334 25652 consensus_queue.cc:1046] T 73f6f44c259547f1b05a3a434616fa54 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.019s
I20260809 15:46:26.266598 23228 leader_election.cc:302] T 28e17dcb5efe45b8b6a8c42da03b4819 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters: c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:26.251852 23526 raft_consensus.cc:1226] T 28e17dcb5efe45b8b6a8c42da03b4819 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Rejecting Update request from peer c0f38c1908414566aa9115bb2cbdd85d for earlier term 1. Current term is 2. Ops: []
I20260809 15:46:26.261628 25506 raft_consensus.cc:3037] T ba56725e42f74a1baa4c04d9660727f9 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:26.210496 25555 consensus_queue.cc:260] T d0721a4403a440f383318a3cea9d1442 P c0f38c1908414566aa9115bb2cbdd85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:26.247782 25366 consensus_queue.cc:1046] T df20671a7b874dbb98c0ef28371c1308 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.010s
I20260809 15:46:26.254539 25369 consensus_queue.cc:1046] T 998c6cee72cf40228c6dba064c0db2da P 65c36d164b0540f099d592f38307ee6c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.004s
I20260809 15:46:26.268373 23228 leader_election.cc:302] T 07fcabb569414bddbd39df6cedd24e64 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters: c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:26.268390 23400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e17dcb5efe45b8b6a8c42da03b4819" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:26.217393 25583 raft_consensus.cc:504] T e14d118b2eaa42aa8a7cd4025aba7128 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:26.241597 25587 raft_consensus.cc:2726] T ae6d2c08a7a94f58ac8dd698e3fd456c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 15:46:26.272038 23228 leader_election.cc:302] T 763e58d6ba644eea830a6c1664d39c9b P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:26.270463 25501 raft_consensus.cc:2781] T 28e17dcb5efe45b8b6a8c42da03b4819 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260809 15:46:26.275195 25501 raft_consensus.cc:482] T 28e17dcb5efe45b8b6a8c42da03b4819 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:46:26.276712 23529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a823d35f4f348d1b52ba9e6d80b00d2" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:26.275732 25501 raft_consensus.cc:3037] T 28e17dcb5efe45b8b6a8c42da03b4819 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:26.278061 25573 raft_consensus.cc:2726] T 763e58d6ba644eea830a6c1664d39c9b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 15:46:26.288182 21289 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260809 15:46:26.296501 21289 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 15:46:26.297142 21289 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 15:46:26.297086 25628 consensus_queue.cc:1046] T 28e17dcb5efe45b8b6a8c42da03b4819 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260809 15:46:26.298206 25603 raft_consensus.cc:3032] T 28e17dcb5efe45b8b6a8c42da03b4819 P c0f38c1908414566aa9115bb2cbdd85d [term 1 LEADER]: Stepping down as leader of term 1
I20260809 15:46:26.298559 25603 raft_consensus.cc:729] T 28e17dcb5efe45b8b6a8c42da03b4819 P c0f38c1908414566aa9115bb2cbdd85d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:46:26.289194 25614 raft_consensus.cc:504] T a3622f6e3e3641cea1960e3a31498bb7 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:26.273926 25675 consensus_queue.cc:1046] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.011s
I20260809 15:46:26.289110 23519 raft_consensus.cc:1226] T 4564dfc6d4094c1b9240b367ccecf3b1 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Rejecting Update request from peer c0f38c1908414566aa9115bb2cbdd85d for earlier term 1. Current term is 2. Ops: []
I20260809 15:46:26.314884 25675 raft_consensus.cc:3032] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d [term 1 LEADER]: Stepping down as leader of term 1
I20260809 15:46:26.315322 25675 raft_consensus.cc:729] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:46:26.315997 25555 raft_consensus.cc:3037] T d0721a4403a440f383318a3cea9d1442 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:26.316336 25675 consensus_queue.cc:260] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:26.318145 25675 raft_consensus.cc:3037] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:26.274147 25460 raft_consensus.cc:2781] T 07fcabb569414bddbd39df6cedd24e64 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260809 15:46:26.321041 25460 raft_consensus.cc:482] T 07fcabb569414bddbd39df6cedd24e64 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:46:26.321601 25460 raft_consensus.cc:3037] T 07fcabb569414bddbd39df6cedd24e64 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:26.321841 25650 raft_consensus.cc:504] T 07fcabb569414bddbd39df6cedd24e64 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:26.322294 25652 consensus_queue.cc:1046] T 73f6f44c259547f1b05a3a434616fa54 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 15:46:26.299461 25603 consensus_queue.cc:260] T 28e17dcb5efe45b8b6a8c42da03b4819 P c0f38c1908414566aa9115bb2cbdd85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:26.323132 25639 consensus_queue.cc:1046] T f7715d71dbac447b9351fc5eb36c5e80 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260809 15:46:26.324136 25603 raft_consensus.cc:3037] T 28e17dcb5efe45b8b6a8c42da03b4819 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:26.324818 23518 raft_consensus.cc:1226] T 07fcabb569414bddbd39df6cedd24e64 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Rejecting Update request from peer c0f38c1908414566aa9115bb2cbdd85d for earlier term 1. Current term is 2. Ops: []
I20260809 15:46:26.325016 25650 leader_election.cc:288] T 07fcabb569414bddbd39df6cedd24e64 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 2 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:26.325683 24452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e14d118b2eaa42aa8a7cd4025aba7128" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f"
W20260809 15:46:26.271682 24410 connection.cc:232] RPC call timeout handler was delayed by -1.0008788029663265s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1775s for a response
I20260809 15:46:26.325769 25650 raft_consensus.cc:3032] T f7715d71dbac447b9351fc5eb36c5e80 P 65c36d164b0540f099d592f38307ee6c [term 1 LEADER]: Stepping down as leader of term 1
I20260809 15:46:26.328755 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07fcabb569414bddbd39df6cedd24e64" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:26.291313 23228 leader_election.cc:302] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:26.290154 25467 consensus_queue.cc:1046] T 2db057fd10e149a9895dff6ef8c9f038 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20260809 15:46:26.292923 25677 raft_consensus.cc:3032] T 73f6f44c259547f1b05a3a434616fa54 P 65c36d164b0540f099d592f38307ee6c [term 1 LEADER]: Stepping down as leader of term 1
I20260809 15:46:26.293672 23527 raft_consensus.cc:1226] T e14d118b2eaa42aa8a7cd4025aba7128 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Rejecting Update request from peer 61e69695258f48baa984c9d395e8ce3f for earlier term 2. Current term is 3. Ops: []
I20260809 15:46:26.293193 25583 leader_election.cc:288] T e14d118b2eaa42aa8a7cd4025aba7128 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:26.331718 25590 consensus_queue.cc:1046] T a88421dba80d4d3388aa5eb62e23853b P 65c36d164b0540f099d592f38307ee6c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 15:46:26.333427 25587 consensus_queue.cc:1046] T 16ca412a0dae4feebe85dd01bf1a9173 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260809 15:46:26.334358 25614 leader_election.cc:288] T a3622f6e3e3641cea1960e3a31498bb7 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 2 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:26.324414 25639 raft_consensus.cc:3032] T df20671a7b874dbb98c0ef28371c1308 P 65c36d164b0540f099d592f38307ee6c [term 1 LEADER]: Stepping down as leader of term 1
I20260809 15:46:26.298203 23521 raft_consensus.cc:2454] T 7660dfa2ef694927b8eda4100a2bcd9b P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 2.
I20260809 15:46:26.307636 25366 consensus_queue.cc:1046] T cef3e7277d7d4760824a3436ed4a847d P 65c36d164b0540f099d592f38307ee6c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260809 15:46:26.310315 25671 raft_consensus.cc:3032] T d2d4f749fcfd4fb3bfb8115af3471598 P c0f38c1908414566aa9115bb2cbdd85d [term 1 LEADER]: Stepping down as leader of term 1
I20260809 15:46:26.312863 25500 raft_consensus.cc:3037] T 4564dfc6d4094c1b9240b367ccecf3b1 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:26.336678 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3622f6e3e3641cea1960e3a31498bb7" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
W20260809 15:46:26.348829 24406 leader_election.cc:334] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459): Timed out: RequestConsensusVote RPC to 127.20.202.65:35459 timed out after 1.511s (SENT)
I20260809 15:46:26.350884 23402 raft_consensus.cc:2379] T a3622f6e3e3641cea1960e3a31498bb7 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c in current term 2: Already voted for candidate c0f38c1908414566aa9115bb2cbdd85d in this term.
I20260809 15:46:26.328137 25650 raft_consensus.cc:729] T f7715d71dbac447b9351fc5eb36c5e80 P 65c36d164b0540f099d592f38307ee6c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:46:26.294611 25369 raft_consensus.cc:3032] T 998c6cee72cf40228c6dba064c0db2da P 65c36d164b0540f099d592f38307ee6c [term 3 LEADER]: Stepping down as leader of term 3
I20260809 15:46:26.353703 25369 raft_consensus.cc:729] T 998c6cee72cf40228c6dba064c0db2da P 65c36d164b0540f099d592f38307ee6c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:46:26.349104 25639 raft_consensus.cc:729] T df20671a7b874dbb98c0ef28371c1308 P 65c36d164b0540f099d592f38307ee6c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:46:26.354465 25369 consensus_queue.cc:260] T 998c6cee72cf40228c6dba064c0db2da P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:26.354995 25639 consensus_queue.cc:260] T df20671a7b874dbb98c0ef28371c1308 P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:26.355778 25369 raft_consensus.cc:3037] T 998c6cee72cf40228c6dba064c0db2da P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:46:26.356382 25639 raft_consensus.cc:3037] T df20671a7b874dbb98c0ef28371c1308 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:26.349598 23116 catalog_manager.cc:5495] T 05ad7d8fdca64e3da527280ca5be9280 P c0f38c1908414566aa9115bb2cbdd85d reported cstate change: term changed from 1 to 2, leader changed from 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65) to c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66). New cstate: current_term: 2 leader_uuid: "c0f38c1908414566aa9115bb2cbdd85d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } health_report { overall_health: UNKNOWN } } }
I20260809 15:46:26.359566 25490 raft_consensus.cc:504] T ae6d2c08a7a94f58ac8dd698e3fd456c P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:26.334841 25587 raft_consensus.cc:3032] T 16ca412a0dae4feebe85dd01bf1a9173 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 LEADER]: Stepping down as leader of term 1
I20260809 15:46:26.360913 24410 leader_election.cc:302] T 472816994d814cb198b69e5248697440 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:26.361296 25587 raft_consensus.cc:729] T 16ca412a0dae4feebe85dd01bf1a9173 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Running, Role: LEADER
I20260809 15:46:26.361802 25490 leader_election.cc:288] T ae6d2c08a7a94f58ac8dd698e3fd456c P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 2 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:26.362416 25587 consensus_queue.cc:260] T 16ca412a0dae4feebe85dd01bf1a9173 P 67b3aeadfa7a432abe8af4d292e04ab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:26.363471 25658 raft_consensus.cc:504] T 0c29937a80724ddb9d07a27534a146f3 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:26.364391 25587 raft_consensus.cc:3037] T 16ca412a0dae4feebe85dd01bf1a9173 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:26.353063 23228 leader_election.cc:302] T 294d4bcc1e7a477cb56455581277a60f P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c
I20260809 15:46:26.364722 23517 raft_consensus.cc:1226] T 0c29937a80724ddb9d07a27534a146f3 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Rejecting Update request from peer c0f38c1908414566aa9115bb2cbdd85d for earlier term 1. Current term is 2. Ops: []
I20260809 15:46:26.364836 25501 raft_consensus.cc:504] T 28e17dcb5efe45b8b6a8c42da03b4819 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:26.350243 24406 leader_election.cc:302] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f; no voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:26.366925 25366 raft_consensus.cc:3032] T cef3e7277d7d4760824a3436ed4a847d P 65c36d164b0540f099d592f38307ee6c [term 1 LEADER]: Stepping down as leader of term 1
I20260809 15:46:26.369333 25638 raft_consensus.cc:504] T 213543cdbe814c89a17a963469922c1b P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:26.371382 25639 consensus_queue.cc:1046] T c5c2e425a74b4e64ae70b5def750434c P 65c36d164b0540f099d592f38307ee6c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260809 15:46:26.372637 25614 consensus_queue.cc:1046] T 953eea268e17491ca76f3a555860a512 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 15:46:26.373310 25667 raft_consensus.cc:482] T 3d880de657e544468c82157769622a26 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
I20260809 15:46:26.375979 23270 raft_consensus.cc:1226] T ba56725e42f74a1baa4c04d9660727f9 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 1. Current term is 2. Ops: []
I20260809 15:46:26.376768 23400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c29937a80724ddb9d07a27534a146f3" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:26.377418 23283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4564dfc6d4094c1b9240b367ccecf3b1" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
W20260809 15:46:26.401425 24406 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 52629.6us
I20260809 15:46:26.378850 23274 raft_consensus.cc:2454] T 00a1881267a14ba8b78c81040f17a014 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 3.
I20260809 15:46:26.376528 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae6d2c08a7a94f58ac8dd698e3fd456c" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:26.382639 23277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "538a228ddfb54bc6a36528d5ff07597b" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:26.382843 23286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0acc47ce21c3422698d1ef7277c3ed63" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:26.383477 25477 raft_consensus.cc:2726] T 294d4bcc1e7a477cb56455581277a60f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 15:46:26.383702 25621 raft_consensus.cc:2726] T 2424fd9bd7f14e508fd106cbfda786a4 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 15:46:26.384292 23282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ae14a74a26f434db08c652e097385c8" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:26.385233 25470 consensus_queue.cc:1046] T 07fcabb569414bddbd39df6cedd24e64 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260809 15:46:26.386595 23285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3622f6e3e3641cea1960e3a31498bb7" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:26.379540 23520 raft_consensus.cc:482] T ae6d2c08a7a94f58ac8dd698e3fd456c P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 15:46:26.379413 25671 raft_consensus.cc:729] T d2d4f749fcfd4fb3bfb8115af3471598 P c0f38c1908414566aa9115bb2cbdd85d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:46:26.410588 23520 raft_consensus.cc:3037] T ae6d2c08a7a94f58ac8dd698e3fd456c P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:26.411520 25470 raft_consensus.cc:3032] T 07fcabb569414bddbd39df6cedd24e64 P c0f38c1908414566aa9115bb2cbdd85d [term 1 LEADER]: Stepping down as leader of term 1
I20260809 15:46:26.411861 25470 raft_consensus.cc:729] T 07fcabb569414bddbd39df6cedd24e64 P c0f38c1908414566aa9115bb2cbdd85d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:46:26.411243 25671 consensus_queue.cc:260] T d2d4f749fcfd4fb3bfb8115af3471598 P c0f38c1908414566aa9115bb2cbdd85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:26.412840 25671 raft_consensus.cc:3037] T d2d4f749fcfd4fb3bfb8115af3471598 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:26.388152 25635 consensus_queue.cc:1046] T 538a228ddfb54bc6a36528d5ff07597b P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.008s
I20260809 15:46:26.393213 23228 leader_election.cc:302] T c0741ab44d99439abc3aa3bdd27ee160 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c
I20260809 15:46:26.391589 23273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0721a4403a440f383318a3cea9d1442" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:26.416388 23273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e14d118b2eaa42aa8a7cd4025aba7128" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:26.417455 23273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2d4f749fcfd4fb3bfb8115af3471598" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:26.417759 23351 leader_election.cc:302] T 8e4ffcf83c234ae7910ddef3ec0b4bda P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 61e69695258f48baa984c9d395e8ce3f, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:26.418484 23273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5712f6b6fffc4cc5a8ddfbe63e4097df" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:26.418816 23351 leader_election.cc:302] T a3622f6e3e3641cea1960e3a31498bb7 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:26.385339 23280 raft_consensus.cc:1226] T cef3e7277d7d4760824a3436ed4a847d P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 1. Current term is 2. Ops: []
I20260809 15:46:26.335194 25677 raft_consensus.cc:729] T 73f6f44c259547f1b05a3a434616fa54 P 65c36d164b0540f099d592f38307ee6c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:46:26.419485 25628 raft_consensus.cc:2726] T 8e4ffcf83c234ae7910ddef3ec0b4bda P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 15:46:26.420045 25628 raft_consensus.cc:2726] T a3622f6e3e3641cea1960e3a31498bb7 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260809 15:46:26.420046 25677 consensus_queue.cc:260] T 73f6f44c259547f1b05a3a434616fa54 P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:26.421126 23351 leader_election.cc:302] T 953eea268e17491ca76f3a555860a512 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:26.421790 25628 raft_consensus.cc:2726] T 953eea268e17491ca76f3a555860a512 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260809 15:46:26.422044 25677 raft_consensus.cc:3037] T 73f6f44c259547f1b05a3a434616fa54 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:26.404490 23470 leader_election.cc:302] T 538a228ddfb54bc6a36528d5ff07597b P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:26.406571 25665 consensus_queue.cc:1046] T 0c29937a80724ddb9d07a27534a146f3 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.004s
I20260809 15:46:26.423449 23351 leader_election.cc:302] T 297c82ed906d4b48b184985eed3de601 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:26.424116 25551 raft_consensus.cc:3032] T 0c29937a80724ddb9d07a27534a146f3 P c0f38c1908414566aa9115bb2cbdd85d [term 1 LEADER]: Stepping down as leader of term 1
I20260809 15:46:26.424232 25665 raft_consensus.cc:2726] T 297c82ed906d4b48b184985eed3de601 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260809 15:46:26.424469 25551 raft_consensus.cc:729] T 0c29937a80724ddb9d07a27534a146f3 P c0f38c1908414566aa9115bb2cbdd85d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:46:26.425220 25551 consensus_queue.cc:260] T 0c29937a80724ddb9d07a27534a146f3 P c0f38c1908414566aa9115bb2cbdd85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:26.426306 25551 raft_consensus.cc:3037] T 0c29937a80724ddb9d07a27534a146f3 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:26.427685 23280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba56725e42f74a1baa4c04d9660727f9" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:26.428364 23280 raft_consensus.cc:2379] T ba56725e42f74a1baa4c04d9660727f9 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d in current term 2: Already voted for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in this term.
I20260809 15:46:26.428987 23280 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "07fcabb569414bddbd39df6cedd24e64"
dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
from {username='slave'} at 127.0.0.1:53562
I20260809 15:46:26.431178 23351 leader_election.cc:302] T 0da4d5283c8a46878bd39b85e5c71f66 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:26.399695 25366 raft_consensus.cc:729] T cef3e7277d7d4760824a3436ed4a847d P 65c36d164b0540f099d592f38307ee6c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
W20260809 15:46:26.400156 25679 log.cc:924] Time spent T 05ad7d8fdca64e3da527280ca5be9280 P c0f38c1908414566aa9115bb2cbdd85d: Append to log took a long time: real 0.092s user 0.000s sys 0.001s
I20260809 15:46:26.382023 25658 leader_election.cc:288] T 0c29937a80724ddb9d07a27534a146f3 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 2 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:26.412586 25470 consensus_queue.cc:260] T 07fcabb569414bddbd39df6cedd24e64 P c0f38c1908414566aa9115bb2cbdd85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:26.332811 25573 raft_consensus.cc:2726] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 15:46:26.406335 25650 consensus_queue.cc:260] T f7715d71dbac447b9351fc5eb36c5e80 P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:26.435999 23285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e17dcb5efe45b8b6a8c42da03b4819" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:26.435412 23351 leader_election.cc:302] T 2db057fd10e149a9895dff6ef8c9f038 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:26.436371 23277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07fcabb569414bddbd39df6cedd24e64" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:26.435442 23228 leader_election.cc:302] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c
I20260809 15:46:26.402881 25649 raft_consensus.cc:3037] T 472816994d814cb198b69e5248697440 P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:26.438915 25663 raft_consensus.cc:504] T 4bcac4b24e7a4448abdc934bba5a624e P 61e69695258f48baa984c9d395e8ce3f [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:26.439821 23282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "294d4bcc1e7a477cb56455581277a60f" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:26.442389 23470 leader_election.cc:302] T 4564dfc6d4094c1b9240b367ccecf3b1 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:26.466104 23400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2db057fd10e149a9895dff6ef8c9f038" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:26.467314 25501 leader_election.cc:288] T 28e17dcb5efe45b8b6a8c42da03b4819 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 2 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:26.467559 25532 consensus_queue.cc:1046] T 0acc47ce21c3422698d1ef7277c3ed63 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260809 15:46:26.468322 25620 raft_consensus.cc:3032] T 2db057fd10e149a9895dff6ef8c9f038 P 65c36d164b0540f099d592f38307ee6c [term 1 LEADER]: Stepping down as leader of term 1
I20260809 15:46:26.468815 25620 raft_consensus.cc:729] T 2db057fd10e149a9895dff6ef8c9f038 P 65c36d164b0540f099d592f38307ee6c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:46:26.436877 25477 raft_consensus.cc:2726] T c0741ab44d99439abc3aa3bdd27ee160 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 15:46:26.469509 23518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2db057fd10e149a9895dff6ef8c9f038" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:26.467202 23283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae6d2c08a7a94f58ac8dd698e3fd456c" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:26.441920 25639 raft_consensus.cc:2726] T 538a228ddfb54bc6a36528d5ff07597b P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260809 15:46:26.403347 25478 raft_consensus.cc:2726] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 15:46:26.432921 25490 raft_consensus.cc:3032] T 953eea268e17491ca76f3a555860a512 P 65c36d164b0540f099d592f38307ee6c [term 1 LEADER]: Stepping down as leader of term 1
I20260809 15:46:26.472318 25490 raft_consensus.cc:729] T 953eea268e17491ca76f3a555860a512 P 65c36d164b0540f099d592f38307ee6c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:46:26.468982 25532 raft_consensus.cc:3032] T 0acc47ce21c3422698d1ef7277c3ed63 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 LEADER]: Stepping down as leader of term 1
I20260809 15:46:26.473738 25532 raft_consensus.cc:729] T 0acc47ce21c3422698d1ef7277c3ed63 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Running, Role: LEADER
I20260809 15:46:26.473344 25490 consensus_queue.cc:260] T 953eea268e17491ca76f3a555860a512 P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:26.476009 23470 leader_election.cc:302] T 6ae14a74a26f434db08c652e097385c8 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:26.453250 23274 raft_consensus.cc:1226] T 28e17dcb5efe45b8b6a8c42da03b4819 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Rejecting Update request from peer c0f38c1908414566aa9115bb2cbdd85d for earlier term 1. Current term is 2. Ops: []
I20260809 15:46:26.454160 23270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c29937a80724ddb9d07a27534a146f3" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:26.409595 23276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05ad7d8fdca64e3da527280ca5be9280" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:26.477823 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a88421dba80d4d3388aa5eb62e23853b" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:26.448288 25603 raft_consensus.cc:2726] T 0da4d5283c8a46878bd39b85e5c71f66 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260809 15:46:26.457417 25638 leader_election.cc:288] T 213543cdbe814c89a17a963469922c1b P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 2 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:26.458235 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "213543cdbe814c89a17a963469922c1b" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:26.459753 25470 raft_consensus.cc:3037] T 07fcabb569414bddbd39df6cedd24e64 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:26.474787 25490 raft_consensus.cc:3037] T 953eea268e17491ca76f3a555860a512 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:26.405162 25667 raft_consensus.cc:504] T 3d880de657e544468c82157769622a26 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
W20260809 15:46:26.431335 24410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 48758.8us
I20260809 15:46:26.462700 25652 raft_consensus.cc:3032] T a88421dba80d4d3388aa5eb62e23853b P 65c36d164b0540f099d592f38307ee6c [term 1 LEADER]: Stepping down as leader of term 1
I20260809 15:46:26.433145 25614 raft_consensus.cc:3032] T c5c2e425a74b4e64ae70b5def750434c P 65c36d164b0540f099d592f38307ee6c [term 1 LEADER]: Stepping down as leader of term 1
I20260809 15:46:26.469941 25620 consensus_queue.cc:260] T 2db057fd10e149a9895dff6ef8c9f038 P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:26.476594 23273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5c2e425a74b4e64ae70b5def750434c" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:26.477077 23470 leader_election.cc:302] T a3622f6e3e3641cea1960e3a31498bb7 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:26.442155 25460 raft_consensus.cc:504] T 07fcabb569414bddbd39df6cedd24e64 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:26.496511 23519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7715d71dbac447b9351fc5eb36c5e80" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:26.496776 23273 raft_consensus.cc:2379] T c5c2e425a74b4e64ae70b5def750434c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d in current term 2: Already voted for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in this term.
I20260809 15:46:26.479255 23394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7715d71dbac447b9351fc5eb36c5e80" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:26.481108 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9e7cadcd24d45fcbfa84209225e8f2d" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:26.505214 23527 raft_consensus.cc:2454] T f9e7cadcd24d45fcbfa84209225e8f2d P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 2.
I20260809 15:46:26.455847 23351 leader_election.cc:302] T 538a228ddfb54bc6a36528d5ff07597b P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
W20260809 15:46:26.506947 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 51171.4us
I20260809 15:46:26.507313 25603 raft_consensus.cc:2726] T 538a228ddfb54bc6a36528d5ff07597b P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260809 15:46:26.507757 23351 leader_election.cc:302] T a88421dba80d4d3388aa5eb62e23853b P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:26.508867 23351 leader_election.cc:302] T 998c6cee72cf40228c6dba064c0db2da P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:26.509335 23395 raft_consensus.cc:1226] T 2424fd9bd7f14e508fd106cbfda786a4 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Rejecting Update request from peer 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 2. Current term is 3. Ops: []
I20260809 15:46:26.510025 23351 leader_election.cc:302] T f7715d71dbac447b9351fc5eb36c5e80 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:26.481433 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "953eea268e17491ca76f3a555860a512" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:26.511101 25628 raft_consensus.cc:3037] T a88421dba80d4d3388aa5eb62e23853b P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:26.511303 23351 leader_election.cc:302] T df20671a7b874dbb98c0ef28371c1308 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:26.511610 23527 raft_consensus.cc:1226] T 213543cdbe814c89a17a963469922c1b P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Rejecting Update request from peer c0f38c1908414566aa9115bb2cbdd85d for earlier term 1. Current term is 2. Ops: []
I20260809 15:46:26.511831 25667 leader_election.cc:288] T 3d880de657e544468c82157769622a26 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:26.512217 25681 raft_consensus.cc:3037] T 998c6cee72cf40228c6dba064c0db2da P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:46:26.486939 25465 consensus_queue.cc:1046] T 0da4d5283c8a46878bd39b85e5c71f66 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.026s
I20260809 15:46:26.459874 25650 raft_consensus.cc:3037] T f7715d71dbac447b9351fc5eb36c5e80 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:26.486996 23393 raft_consensus.cc:1226] T 00a1881267a14ba8b78c81040f17a014 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Rejecting Update request from peer 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 2. Current term is 3. Ops: []
I20260809 15:46:26.513622 23404 raft_consensus.cc:1226] T 6ae14a74a26f434db08c652e097385c8 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Rejecting Update request from peer 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 1. Current term is 2. Ops: []
I20260809 15:46:26.488174 23399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a88421dba80d4d3388aa5eb62e23853b" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:26.485507 24471 raft_consensus.cc:1226] T 4bcac4b24e7a4448abdc934bba5a624e P 61e69695258f48baa984c9d395e8ce3f [term 2 FOLLOWER]: Rejecting Update request from peer c0f38c1908414566aa9115bb2cbdd85d for earlier term 1. Current term is 2. Ops: []
I20260809 15:46:26.490360 25652 raft_consensus.cc:729] T a88421dba80d4d3388aa5eb62e23853b P 65c36d164b0540f099d592f38307ee6c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:46:26.515707 25603 raft_consensus.cc:3037] T df20671a7b874dbb98c0ef28371c1308 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:26.516330 25652 consensus_queue.cc:260] T a88421dba80d4d3388aa5eb62e23853b P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:26.517769 25652 raft_consensus.cc:3037] T a88421dba80d4d3388aa5eb62e23853b P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:26.475888 25635 raft_consensus.cc:2726] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 15:46:26.487448 24406 leader_election.cc:302] T cee02fff2c9c48a19f43eb358a89e7b0 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c
I20260809 15:46:26.489750 23397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "953eea268e17491ca76f3a555860a512" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:26.489539 25614 raft_consensus.cc:729] T c5c2e425a74b4e64ae70b5def750434c P 65c36d164b0540f099d592f38307ee6c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:46:26.520340 25614 consensus_queue.cc:260] T c5c2e425a74b4e64ae70b5def750434c P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:26.521595 25609 raft_consensus.cc:2781] T cee02fff2c9c48a19f43eb358a89e7b0 P 61e69695258f48baa984c9d395e8ce3f [term 2 FOLLOWER]: Leader election won for term 2
I20260809 15:46:26.522154 25614 raft_consensus.cc:3037] T c5c2e425a74b4e64ae70b5def750434c P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:26.491684 23529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e14d118b2eaa42aa8a7cd4025aba7128" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:26.494915 23470 leader_election.cc:302] T d0721a4403a440f383318a3cea9d1442 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:26.467204 25366 consensus_queue.cc:260] T cef3e7277d7d4760824a3436ed4a847d P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:26.485281 25477 consensus_queue.cc:1046] T 538a228ddfb54bc6a36528d5ff07597b P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.010s
I20260809 15:46:26.497136 23392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "907bacfa79d04a55a68c8dbe60056462" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:26.482942 25639 raft_consensus.cc:3037] T 6ae14a74a26f434db08c652e097385c8 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:26.538697 25667 raft_consensus.cc:2726] T d0721a4403a440f383318a3cea9d1442 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260809 15:46:26.499353 24410 leader_election.cc:302] T d3be1e356b254d22b995be32f1c12f6f P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f; no voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:26.539784 25483 consensus_queue.cc:1046] T 4bcac4b24e7a4448abdc934bba5a624e P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260809 15:46:26.541111 25483 raft_consensus.cc:3032] T 4bcac4b24e7a4448abdc934bba5a624e P c0f38c1908414566aa9115bb2cbdd85d [term 1 LEADER]: Stepping down as leader of term 1
I20260809 15:46:26.537503 25366 raft_consensus.cc:3037] T cef3e7277d7d4760824a3436ed4a847d P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:26.497582 25460 leader_election.cc:288] T 07fcabb569414bddbd39df6cedd24e64 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 2 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:26.478686 23400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5c2e425a74b4e64ae70b5def750434c" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:26.495961 25620 raft_consensus.cc:3037] T 2db057fd10e149a9895dff6ef8c9f038 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:26.486927 23276 raft_consensus.cc:2454] T 05ad7d8fdca64e3da527280ca5be9280 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 2.
I20260809 15:46:26.499040 23280 raft_consensus.cc:482] T 07fcabb569414bddbd39df6cedd24e64 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 15:46:26.545617 23400 raft_consensus.cc:2379] T c5c2e425a74b4e64ae70b5def750434c P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 in current term 2: Already voted for candidate c0f38c1908414566aa9115bb2cbdd85d in this term.
I20260809 15:46:26.499432 25638 raft_consensus.cc:2726] T a3622f6e3e3641cea1960e3a31498bb7 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260809 15:46:26.483685 23228 leader_election.cc:302] T cef3e7277d7d4760824a3436ed4a847d P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:26.484577 23521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5c2e425a74b4e64ae70b5def750434c" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:26.513130 25665 raft_consensus.cc:3037] T f7715d71dbac447b9351fc5eb36c5e80 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:26.479658 25658 raft_consensus.cc:2726] T 4564dfc6d4094c1b9240b367ccecf3b1 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260809 15:46:26.523398 25555 consensus_queue.cc:1046] T 213543cdbe814c89a17a963469922c1b P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260809 15:46:26.537658 25465 raft_consensus.cc:3032] T 0da4d5283c8a46878bd39b85e5c71f66 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 LEADER]: Stepping down as leader of term 1
I20260809 15:46:26.524405 25681 raft_consensus.cc:2726] T 998c6cee72cf40228c6dba064c0db2da P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260809 15:46:26.474716 25532 consensus_queue.cc:260] T 0acc47ce21c3422698d1ef7277c3ed63 P 67b3aeadfa7a432abe8af4d292e04ab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } attrs { replace: true } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } }
I20260809 15:46:26.525341 23351 leader_election.cc:302] T ba56725e42f74a1baa4c04d9660727f9 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:26.483409 25576 raft_consensus.cc:3032] T 538a228ddfb54bc6a36528d5ff07597b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 LEADER]: Stepping down as leader of term 2
I20260809 15:46:26.560333 23351 leader_election.cc:302] T 00a1881267a14ba8b78c81040f17a014 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c
I20260809 15:46:26.550926 25609 raft_consensus.cc:686] T cee02fff2c9c48a19f43eb358a89e7b0 P 61e69695258f48baa984c9d395e8ce3f [term 2 LEADER]: Becoming Leader. State: Replica: 61e69695258f48baa984c9d395e8ce3f, State: Running, Role: LEADER
I20260809 15:46:26.561236 25686 raft_consensus.cc:2726] T d3be1e356b254d22b995be32f1c12f6f P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 15:46:26.561844 25609 consensus_queue.cc:237] T cee02fff2c9c48a19f43eb358a89e7b0 P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:26.563390 23526 raft_consensus.cc:2454] T 953eea268e17491ca76f3a555860a512 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 2.
I20260809 15:46:26.535688 23529 raft_consensus.cc:2379] T e14d118b2eaa42aa8a7cd4025aba7128 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 in current term 3: Already voted for candidate 65c36d164b0540f099d592f38307ee6c in this term.
I20260809 15:46:26.559841 25576 raft_consensus.cc:729] T 538a228ddfb54bc6a36528d5ff07597b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Running, Role: LEADER
I20260809 15:46:26.536140 23470 leader_election.cc:302] T e14d118b2eaa42aa8a7cd4025aba7128 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 61e69695258f48baa984c9d395e8ce3f, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:26.545688 23280 raft_consensus.cc:3037] T 07fcabb569414bddbd39df6cedd24e64 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Advancing to term 3
W20260809 15:46:26.547981 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 64344.3us
I20260809 15:46:26.565702 23228 leader_election.cc:302] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters: 61e69695258f48baa984c9d395e8ce3f
I20260809 15:46:26.566255 25501 raft_consensus.cc:2781] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260809 15:46:26.566610 23228 leader_election.cc:302] T 73f6f44c259547f1b05a3a434616fa54 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:26.566782 25501 raft_consensus.cc:482] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
I20260809 15:46:26.567533 25501 raft_consensus.cc:3037] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:26.564951 25576 consensus_queue.cc:260] T 538a228ddfb54bc6a36528d5ff07597b P 67b3aeadfa7a432abe8af4d292e04ab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:26.571688 25675 raft_consensus.cc:3032] T 213543cdbe814c89a17a963469922c1b P c0f38c1908414566aa9115bb2cbdd85d [term 1 LEADER]: Stepping down as leader of term 1
I20260809 15:46:26.550043 25573 raft_consensus.cc:2726] T cef3e7277d7d4760824a3436ed4a847d P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260809 15:46:26.572288 25675 raft_consensus.cc:729] T 213543cdbe814c89a17a963469922c1b P c0f38c1908414566aa9115bb2cbdd85d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:46:26.550155 25465 raft_consensus.cc:729] T 0da4d5283c8a46878bd39b85e5c71f66 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Running, Role: LEADER
I20260809 15:46:26.573895 25675 consensus_queue.cc:260] T 213543cdbe814c89a17a963469922c1b P c0f38c1908414566aa9115bb2cbdd85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:26.552688 25663 leader_election.cc:288] T 4bcac4b24e7a4448abdc934bba5a624e P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 2 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:26.541513 25483 raft_consensus.cc:729] T 4bcac4b24e7a4448abdc934bba5a624e P c0f38c1908414566aa9115bb2cbdd85d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:46:26.575352 25675 raft_consensus.cc:3037] T 213543cdbe814c89a17a963469922c1b P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:26.574211 23470 leader_election.cc:302] T d2d4f749fcfd4fb3bfb8115af3471598 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:26.576678 25658 raft_consensus.cc:2726] T e14d118b2eaa42aa8a7cd4025aba7128 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260809 15:46:26.577106 25576 raft_consensus.cc:3037] T 538a228ddfb54bc6a36528d5ff07597b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:26.577497 25658 raft_consensus.cc:2726] T d2d4f749fcfd4fb3bfb8115af3471598 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260809 15:46:26.575136 23280 raft_consensus.cc:504] T 07fcabb569414bddbd39df6cedd24e64 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:26.548086 23402 raft_consensus.cc:1226] T c5c2e425a74b4e64ae70b5def750434c P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 1. Current term is 2. Ops: []
I20260809 15:46:26.551347 24406 leader_election.cc:302] T 563daf42d7784db195ea99c3cb21b370 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 67b3aeadfa7a432abe8af4d292e04ab9; no voters: c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:26.569463 25477 raft_consensus.cc:2726] T 73f6f44c259547f1b05a3a434616fa54 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260809 15:46:26.568157 25532 raft_consensus.cc:3037] T 0acc47ce21c3422698d1ef7277c3ed63 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:26.581409 25663 raft_consensus.cc:2781] T 563daf42d7784db195ea99c3cb21b370 P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260809 15:46:26.582022 25663 raft_consensus.cc:482] T 563daf42d7784db195ea99c3cb21b370 P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Starting leader election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
I20260809 15:46:26.582425 25663 raft_consensus.cc:3037] T 563daf42d7784db195ea99c3cb21b370 P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:26.570040 25635 consensus_queue.cc:1046] T a3622f6e3e3641cea1960e3a31498bb7 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260809 15:46:26.568454 25555 raft_consensus.cc:2726] T ba56725e42f74a1baa4c04d9660727f9 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260809 15:46:26.583644 25635 raft_consensus.cc:3032] T a3622f6e3e3641cea1960e3a31498bb7 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 LEADER]: Stepping down as leader of term 1
I20260809 15:46:26.584177 25635 raft_consensus.cc:729] T a3622f6e3e3641cea1960e3a31498bb7 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Running, Role: LEADER
I20260809 15:46:26.574726 25465 consensus_queue.cc:260] T 0da4d5283c8a46878bd39b85e5c71f66 P 67b3aeadfa7a432abe8af4d292e04ab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:26.585757 25465 raft_consensus.cc:3037] T 0da4d5283c8a46878bd39b85e5c71f66 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:26.585296 25635 consensus_queue.cc:260] T a3622f6e3e3641cea1960e3a31498bb7 P 67b3aeadfa7a432abe8af4d292e04ab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:26.478346 25551 raft_consensus.cc:3037] T 2db057fd10e149a9895dff6ef8c9f038 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:26.576128 25483 consensus_queue.cc:260] T 4bcac4b24e7a4448abdc934bba5a624e P c0f38c1908414566aa9115bb2cbdd85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:26.578702 23470 leader_election.cc:302] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 61e69695258f48baa984c9d395e8ce3f, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:26.567981 23228 leader_election.cc:302] T 998c6cee72cf40228c6dba064c0db2da P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:26.591883 24452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d880de657e544468c82157769622a26" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:26.597618 23228 leader_election.cc:302] T 297c82ed906d4b48b184985eed3de601 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:26.592155 23280 leader_election.cc:288] T 07fcabb569414bddbd39df6cedd24e64 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 3 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:26.592099 23351 leader_election.cc:302] T c5c2e425a74b4e64ae70b5def750434c P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:26.598758 25658 raft_consensus.cc:2726] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260809 15:46:26.599354 25587 raft_consensus.cc:2726] T 998c6cee72cf40228c6dba064c0db2da P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260809 15:46:26.597988 23470 leader_election.cc:302] T 0acc47ce21c3422698d1ef7277c3ed63 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:26.601408 23470 leader_election.cc:302] T 28e17dcb5efe45b8b6a8c42da03b4819 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:26.599385 25477 raft_consensus.cc:2726] T 297c82ed906d4b48b184985eed3de601 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260809 15:46:26.602046 25658 raft_consensus.cc:2726] T 0acc47ce21c3422698d1ef7277c3ed63 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260809 15:46:26.602725 25658 raft_consensus.cc:2726] T 28e17dcb5efe45b8b6a8c42da03b4819 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260809 15:46:26.606010 23519 raft_consensus.cc:2454] T f7715d71dbac447b9351fc5eb36c5e80 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 2.
I20260809 15:46:26.608474 23470 leader_election.cc:302] T 294d4bcc1e7a477cb56455581277a60f P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 61e69695258f48baa984c9d395e8ce3f, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:26.609164 25650 raft_consensus.cc:2726] T 294d4bcc1e7a477cb56455581277a60f P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260809 15:46:26.610463 23470 leader_election.cc:302] T ae6d2c08a7a94f58ac8dd698e3fd456c P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:26.611196 25555 raft_consensus.cc:2726] T c5c2e425a74b4e64ae70b5def750434c P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260809 15:46:26.614260 23276 raft_consensus.cc:1261] T f9e7cadcd24d45fcbfa84209225e8f2d P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 15:46:26.616163 23470 leader_election.cc:302] T 07fcabb569414bddbd39df6cedd24e64 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:26.617208 23298 tablet_copy_source_session.cc:205] T 0acc47ce21c3422698d1ef7277c3ed63 P 67b3aeadfa7a432abe8af4d292e04ab9: Tablet Copy: opened 0 blocks and 1 log segments
I20260809 15:46:26.617547 25555 consensus_queue.cc:1035] T f9e7cadcd24d45fcbfa84209225e8f2d P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 15:46:26.616107 23280 raft_consensus.cc:1261] T 05ad7d8fdca64e3da527280ca5be9280 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 15:46:26.619441 23470 leader_election.cc:302] T 0c29937a80724ddb9d07a27534a146f3 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:26.614200 25663 raft_consensus.cc:504] T 563daf42d7784db195ea99c3cb21b370 P 61e69695258f48baa984c9d395e8ce3f [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:26.620775 25507 consensus_queue.cc:1035] T 05ad7d8fdca64e3da527280ca5be9280 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260809 15:46:26.587152 25635 raft_consensus.cc:3037] T a3622f6e3e3641cea1960e3a31498bb7 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:26.572259 25681 raft_consensus.cc:2781] T 00a1881267a14ba8b78c81040f17a014 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader election won for term 3
I20260809 15:46:26.628805 25685 consensus_queue.cc:1046] T 3924b648461e463b9d7925e5671d68ae P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.003s
I20260809 15:46:26.631042 25681 raft_consensus.cc:686] T 00a1881267a14ba8b78c81040f17a014 P c0f38c1908414566aa9115bb2cbdd85d [term 3 LEADER]: Becoming Leader. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:46:26.623677 25638 raft_consensus.cc:2726] T 0c29937a80724ddb9d07a27534a146f3 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260809 15:46:26.620814 25658 raft_consensus.cc:2726] T 07fcabb569414bddbd39df6cedd24e64 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260809 15:46:26.632642 25663 leader_election.cc:288] T 563daf42d7784db195ea99c3cb21b370 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 2 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:26.632184 25681 consensus_queue.cc:237] T 00a1881267a14ba8b78c81040f17a014 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:26.624352 23470 leader_election.cc:302] T 05ad7d8fdca64e3da527280ca5be9280 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters: c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:26.628988 24410 leader_election.cc:302] T e71a05ba1d4d42bd95c3f85f9313399e P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f; no voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:26.639446 23270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "213543cdbe814c89a17a963469922c1b" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
W20260809 15:46:26.642825 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 44235.2us
I20260809 15:46:26.643546 23228 leader_election.cc:302] T ba56725e42f74a1baa4c04d9660727f9 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:26.643976 23270 raft_consensus.cc:1226] T a88421dba80d4d3388aa5eb62e23853b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 1. Current term is 2. Ops: []
I20260809 15:46:26.644685 23228 leader_election.cc:302] T df20671a7b874dbb98c0ef28371c1308 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:26.645784 23228 leader_election.cc:302] T e71a05ba1d4d42bd95c3f85f9313399e P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters: 61e69695258f48baa984c9d395e8ce3f
I20260809 15:46:26.645953 23270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d880de657e544468c82157769622a26" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:26.649216 23470 leader_election.cc:302] T 213543cdbe814c89a17a963469922c1b P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:26.599411 23520 raft_consensus.cc:504] T ae6d2c08a7a94f58ac8dd698e3fd456c P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:26.657981 23470 leader_election.cc:302] T 3d880de657e544468c82157769622a26 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 61e69695258f48baa984c9d395e8ce3f, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:26.631392 25685 raft_consensus.cc:3032] T 3924b648461e463b9d7925e5671d68ae P 61e69695258f48baa984c9d395e8ce3f [term 2 LEADER]: Stepping down as leader of term 2
I20260809 15:46:26.659114 23273 raft_consensus.cc:1226] T 953eea268e17491ca76f3a555860a512 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 1. Current term is 2. Ops: []
I20260809 15:46:26.640202 25658 raft_consensus.cc:2741] T 05ad7d8fdca64e3da527280ca5be9280 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Leader pre-election decision vote started in defunct term 1: won
I20260809 15:46:26.652540 25483 raft_consensus.cc:3037] T 4bcac4b24e7a4448abdc934bba5a624e P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:26.676740 23517 raft_consensus.cc:1226] T ae6d2c08a7a94f58ac8dd698e3fd456c P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Rejecting Update request from peer c0f38c1908414566aa9115bb2cbdd85d for earlier term 1. Current term is 3. Ops: []
I20260809 15:46:26.652575 25460 raft_consensus.cc:2726] T df20671a7b874dbb98c0ef28371c1308 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260809 15:46:26.655362 25532 raft_consensus.cc:2726] T ba56725e42f74a1baa4c04d9660727f9 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260809 15:46:26.656222 25477 raft_consensus.cc:2781] T e71a05ba1d4d42bd95c3f85f9313399e P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 15:46:26.656025 25638 raft_consensus.cc:2726] T 213543cdbe814c89a17a963469922c1b P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260809 15:46:26.661558 25685 raft_consensus.cc:729] T 3924b648461e463b9d7925e5671d68ae P 61e69695258f48baa984c9d395e8ce3f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 61e69695258f48baa984c9d395e8ce3f, State: Running, Role: LEADER
I20260809 15:46:26.659613 25687 raft_consensus.cc:2726] T e71a05ba1d4d42bd95c3f85f9313399e P 61e69695258f48baa984c9d395e8ce3f [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260809 15:46:26.683691 23521 raft_consensus.cc:2454] T c5c2e425a74b4e64ae70b5def750434c P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 2.
I20260809 15:46:26.665588 25614 raft_consensus.cc:2726] T 3d880de657e544468c82157769622a26 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 15:46:26.679309 25639 raft_consensus.cc:2726] T 6ae14a74a26f434db08c652e097385c8 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 15:46:26.679106 25650 raft_consensus.cc:2726] T ae6d2c08a7a94f58ac8dd698e3fd456c P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260809 15:46:26.694994 25477 raft_consensus.cc:686] T e71a05ba1d4d42bd95c3f85f9313399e P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 LEADER]: Becoming Leader. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Running, Role: LEADER
I20260809 15:46:26.697088 25685 consensus_queue.cc:260] T 3924b648461e463b9d7925e5671d68ae P 61e69695258f48baa984c9d395e8ce3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:26.682878 25573 consensus_queue.cc:1046] T 2424fd9bd7f14e508fd106cbfda786a4 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: INVALID_TERM, Last received: 2.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.015s
I20260809 15:46:26.693315 25460 consensus_queue.cc:1046] T 6ae14a74a26f434db08c652e097385c8 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.006s
I20260809 15:46:26.699388 25603 raft_consensus.cc:2726] T df20671a7b874dbb98c0ef28371c1308 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 15:46:26.700495 24410 leader_election.cc:302] T 0c48a3049897452383257aa757834fb5 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:26.709300 25551 raft_consensus.cc:2726] T 2db057fd10e149a9895dff6ef8c9f038 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 15:46:26.710671 25628 raft_consensus.cc:2726] T a88421dba80d4d3388aa5eb62e23853b P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 15:46:26.712903 23520 leader_election.cc:288] T ae6d2c08a7a94f58ac8dd698e3fd456c P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:26.714007 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae6d2c08a7a94f58ac8dd698e3fd456c" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:26.714206 23273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae6d2c08a7a94f58ac8dd698e3fd456c" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:26.740753 23273 raft_consensus.cc:3037] T ae6d2c08a7a94f58ac8dd698e3fd456c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Advancing to term 3
I20260809 15:46:26.684926 25681 consensus_queue.cc:1046] T ae6d2c08a7a94f58ac8dd698e3fd456c P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.034s
I20260809 15:46:26.744293 23402 raft_consensus.cc:3032] T ae6d2c08a7a94f58ac8dd698e3fd456c P c0f38c1908414566aa9115bb2cbdd85d [term 1 LEADER]: Stepping down as leader of term 1
I20260809 15:46:26.744628 23402 raft_consensus.cc:729] T ae6d2c08a7a94f58ac8dd698e3fd456c P c0f38c1908414566aa9115bb2cbdd85d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:46:26.745353 23402 consensus_queue.cc:260] T ae6d2c08a7a94f58ac8dd698e3fd456c P c0f38c1908414566aa9115bb2cbdd85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:26.746474 23402 raft_consensus.cc:3037] T ae6d2c08a7a94f58ac8dd698e3fd456c P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Advancing to term 3
I20260809 15:46:26.719830 25477 consensus_queue.cc:237] T e71a05ba1d4d42bd95c3f85f9313399e P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:26.720607 25665 raft_consensus.cc:2726] T f7715d71dbac447b9351fc5eb36c5e80 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 15:46:26.721228 25685 raft_consensus.cc:3037] T 3924b648461e463b9d7925e5671d68ae P 61e69695258f48baa984c9d395e8ce3f [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:26.721443 25501 raft_consensus.cc:504] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:26.722828 25573 raft_consensus.cc:3032] T 2424fd9bd7f14e508fd106cbfda786a4 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 LEADER]: Stepping down as leader of term 2
I20260809 15:46:26.754269 25573 raft_consensus.cc:729] T 2424fd9bd7f14e508fd106cbfda786a4 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Running, Role: LEADER
I20260809 15:46:26.755551 25573 consensus_queue.cc:260] T 2424fd9bd7f14e508fd106cbfda786a4 P 67b3aeadfa7a432abe8af4d292e04ab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:26.757225 25573 raft_consensus.cc:3037] T 2424fd9bd7f14e508fd106cbfda786a4 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:26.724069 25688 raft_consensus.cc:3032] T 6ae14a74a26f434db08c652e097385c8 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 LEADER]: Stepping down as leader of term 1
I20260809 15:46:26.759804 25688 raft_consensus.cc:729] T 6ae14a74a26f434db08c652e097385c8 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Running, Role: LEADER
I20260809 15:46:26.725097 23518 raft_consensus.cc:2454] T 2db057fd10e149a9895dff6ef8c9f038 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 2.
I20260809 15:46:26.727999 25649 raft_consensus.cc:2726] T 472816994d814cb198b69e5248697440 P 61e69695258f48baa984c9d395e8ce3f [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 15:46:26.729950 25687 raft_consensus.cc:2726] T 0c48a3049897452383257aa757834fb5 P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 15:46:26.731071 24410 leader_election.cc:302] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:26.762620 23402 raft_consensus.cc:2454] T ae6d2c08a7a94f58ac8dd698e3fd456c P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 3.
I20260809 15:46:26.765794 23472 leader_election.cc:302] T ae6d2c08a7a94f58ac8dd698e3fd456c P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:46:26.766659 25652 raft_consensus.cc:2781] T ae6d2c08a7a94f58ac8dd698e3fd456c P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Leader election won for term 3
W20260809 15:46:26.767387 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 103486us
I20260809 15:46:26.765033 23273 raft_consensus.cc:2454] T ae6d2c08a7a94f58ac8dd698e3fd456c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 3.
I20260809 15:46:26.760118 24452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e4ffcf83c234ae7910ddef3ec0b4bda" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f"
I20260809 15:46:26.770488 23228 leader_election.cc:302] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:26.772845 23270 raft_consensus.cc:1261] T c0741ab44d99439abc3aa3bdd27ee160 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Refusing update from remote peer 61e69695258f48baa984c9d395e8ce3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260809 15:46:26.764025 25501 leader_election.cc:288] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 3 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:26.775559 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e17dcb5efe45b8b6a8c42da03b4819" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:26.773612 25687 raft_consensus.cc:2726] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 61e69695258f48baa984c9d395e8ce3f [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260809 15:46:26.777776 23521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07fcabb569414bddbd39df6cedd24e64" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:26.776970 23400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07fcabb569414bddbd39df6cedd24e64" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:26.780521 23520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e17dcb5efe45b8b6a8c42da03b4819" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:26.782364 23521 raft_consensus.cc:1226] T 2424fd9bd7f14e508fd106cbfda786a4 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Rejecting Update request from peer 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 2. Current term is 3. Ops: []
I20260809 15:46:26.786628 23402 raft_consensus.cc:2454] T 28e17dcb5efe45b8b6a8c42da03b4819 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 2.
I20260809 15:46:26.775045 25460 raft_consensus.cc:2726] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 15:46:26.791123 23400 raft_consensus.cc:2454] T 07fcabb569414bddbd39df6cedd24e64 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 2.
I20260809 15:46:26.794394 23228 leader_election.cc:302] T a88421dba80d4d3388aa5eb62e23853b P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:26.771708 25652 raft_consensus.cc:686] T ae6d2c08a7a94f58ac8dd698e3fd456c P 65c36d164b0540f099d592f38307ee6c [term 3 LEADER]: Becoming Leader. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:46:26.795761 23228 leader_election.cc:302] T f9e7cadcd24d45fcbfa84209225e8f2d P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters: c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:26.796510 25576 raft_consensus.cc:2741] T f9e7cadcd24d45fcbfa84209225e8f2d P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Leader pre-election decision vote started in defunct term 1: won
I20260809 15:46:26.796708 25652 consensus_queue.cc:237] T ae6d2c08a7a94f58ac8dd698e3fd456c P 65c36d164b0540f099d592f38307ee6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:26.797556 23228 leader_election.cc:302] T e14d118b2eaa42aa8a7cd4025aba7128 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c
I20260809 15:46:26.798270 25576 raft_consensus.cc:3037] T e14d118b2eaa42aa8a7cd4025aba7128 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:26.792704 23273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "563daf42d7784db195ea99c3cb21b370" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:26.800737 23228 leader_election.cc:302] T 953eea268e17491ca76f3a555860a512 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters: c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:26.791831 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e4ffcf83c234ae7910ddef3ec0b4bda" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:26.795621 25501 raft_consensus.cc:2726] T a88421dba80d4d3388aa5eb62e23853b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260809 15:46:26.806732 25576 raft_consensus.cc:2726] T e14d118b2eaa42aa8a7cd4025aba7128 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260809 15:46:26.780097 24410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 49182.2us
I20260809 15:46:26.816756 23517 raft_consensus.cc:1261] T 05ad7d8fdca64e3da527280ca5be9280 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 15:46:26.802872 23392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07fcabb569414bddbd39df6cedd24e64" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:26.803676 25637 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0acc47ce21c3422698d1ef7277c3ed63. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:46:26.769807 25688 consensus_queue.cc:260] T 6ae14a74a26f434db08c652e097385c8 P 67b3aeadfa7a432abe8af4d292e04ab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:26.805125 23518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07fcabb569414bddbd39df6cedd24e64" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:26.826589 23228 leader_election.cc:302] T f7715d71dbac447b9351fc5eb36c5e80 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters: c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:26.827369 23392 raft_consensus.cc:3037] T 07fcabb569414bddbd39df6cedd24e64 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:26.829953 23518 raft_consensus.cc:3037] T 07fcabb569414bddbd39df6cedd24e64 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:26.830813 23520 raft_consensus.cc:1261] T f9e7cadcd24d45fcbfa84209225e8f2d P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 15:46:26.833623 25688 raft_consensus.cc:3037] T 6ae14a74a26f434db08c652e097385c8 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:26.835906 23228 leader_election.cc:302] T c5c2e425a74b4e64ae70b5def750434c P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters: c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:26.840715 23273 raft_consensus.cc:2454] T 563daf42d7784db195ea99c3cb21b370 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 2.
I20260809 15:46:26.841599 23228 leader_election.cc:302] T 2db057fd10e149a9895dff6ef8c9f038 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters: c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:26.838778 25483 consensus_queue.cc:1035] T 05ad7d8fdca64e3da527280ca5be9280 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.006s
I20260809 15:46:26.844334 25700 raft_consensus.cc:2781] T 2db057fd10e149a9895dff6ef8c9f038 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 15:46:26.845016 25700 raft_consensus.cc:686] T 2db057fd10e149a9895dff6ef8c9f038 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 LEADER]: Becoming Leader. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Running, Role: LEADER
I20260809 15:46:26.846189 25700 consensus_queue.cc:237] T 2db057fd10e149a9895dff6ef8c9f038 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:26.829365 25687 consensus_queue.cc:1035] T c0741ab44d99439abc3aa3bdd27ee160 P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.019s
I20260809 15:46:26.839010 25649 consensus_queue.cc:1046] T 294d4bcc1e7a477cb56455581277a60f P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.012s
I20260809 15:46:26.850592 25646 consensus_queue.cc:1035] T f9e7cadcd24d45fcbfa84209225e8f2d P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 15:46:26.841270 25573 raft_consensus.cc:2781] T c5c2e425a74b4e64ae70b5def750434c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 15:46:26.852046 25649 raft_consensus.cc:3032] T 294d4bcc1e7a477cb56455581277a60f P 61e69695258f48baa984c9d395e8ce3f [term 2 LEADER]: Stepping down as leader of term 2
I20260809 15:46:26.852165 25573 raft_consensus.cc:686] T c5c2e425a74b4e64ae70b5def750434c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 LEADER]: Becoming Leader. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Running, Role: LEADER
I20260809 15:46:26.852408 25649 raft_consensus.cc:729] T 294d4bcc1e7a477cb56455581277a60f P 61e69695258f48baa984c9d395e8ce3f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 61e69695258f48baa984c9d395e8ce3f, State: Running, Role: LEADER
I20260809 15:46:26.853363 25649 consensus_queue.cc:260] T 294d4bcc1e7a477cb56455581277a60f P 61e69695258f48baa984c9d395e8ce3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:26.855400 24410 leader_election.cc:302] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:26.857179 25687 raft_consensus.cc:2726] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 61e69695258f48baa984c9d395e8ce3f [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260809 15:46:26.857538 24410 leader_election.cc:302] T 7660dfa2ef694927b8eda4100a2bcd9b P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c; no voters: c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:26.860080 23517 raft_consensus.cc:1261] T c0741ab44d99439abc3aa3bdd27ee160 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Refusing update from remote peer 61e69695258f48baa984c9d395e8ce3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260809 15:46:26.853204 25573 consensus_queue.cc:237] T c5c2e425a74b4e64ae70b5def750434c P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:26.862792 25687 raft_consensus.cc:2781] T 7660dfa2ef694927b8eda4100a2bcd9b P 61e69695258f48baa984c9d395e8ce3f [term 2 FOLLOWER]: Leader election won for term 2
I20260809 15:46:26.864123 25687 raft_consensus.cc:686] T 7660dfa2ef694927b8eda4100a2bcd9b P 61e69695258f48baa984c9d395e8ce3f [term 2 LEADER]: Becoming Leader. State: Replica: 61e69695258f48baa984c9d395e8ce3f, State: Running, Role: LEADER
I20260809 15:46:26.857345 25685 consensus_queue.cc:1046] T e14d118b2eaa42aa8a7cd4025aba7128 P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.008s
I20260809 15:46:26.866348 25609 raft_consensus.cc:3032] T e14d118b2eaa42aa8a7cd4025aba7128 P 61e69695258f48baa984c9d395e8ce3f [term 2 LEADER]: Stepping down as leader of term 2
I20260809 15:46:26.866760 25609 raft_consensus.cc:729] T e14d118b2eaa42aa8a7cd4025aba7128 P 61e69695258f48baa984c9d395e8ce3f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 61e69695258f48baa984c9d395e8ce3f, State: Running, Role: LEADER
I20260809 15:46:26.867827 25609 consensus_queue.cc:260] T e14d118b2eaa42aa8a7cd4025aba7128 P 61e69695258f48baa984c9d395e8ce3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:26.869163 25609 raft_consensus.cc:3037] T e14d118b2eaa42aa8a7cd4025aba7128 P 61e69695258f48baa984c9d395e8ce3f [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:26.829255 25501 raft_consensus.cc:2781] T 953eea268e17491ca76f3a555860a512 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 15:46:26.871681 25501 raft_consensus.cc:686] T 953eea268e17491ca76f3a555860a512 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 LEADER]: Becoming Leader. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Running, Role: LEADER
I20260809 15:46:26.865257 25687 consensus_queue.cc:237] T 7660dfa2ef694927b8eda4100a2bcd9b P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:26.854929 25649 raft_consensus.cc:3037] T 294d4bcc1e7a477cb56455581277a60f P 61e69695258f48baa984c9d395e8ce3f [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:26.874806 23115 catalog_manager.cc:5495] T ae6d2c08a7a94f58ac8dd698e3fd456c P 65c36d164b0540f099d592f38307ee6c reported cstate change: term changed from 1 to 3, leader changed from c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66) to 65c36d164b0540f099d592f38307ee6c (127.20.202.67). New cstate: current_term: 3 leader_uuid: "65c36d164b0540f099d592f38307ee6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } health_report { overall_health: HEALTHY } } }
I20260809 15:46:26.879163 24406 leader_election.cc:302] T 563daf42d7784db195ea99c3cb21b370 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:46:26.876163 23392 raft_consensus.cc:2454] T 07fcabb569414bddbd39df6cedd24e64 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 3.
I20260809 15:46:26.881397 23228 leader_election.cc:302] T 28e17dcb5efe45b8b6a8c42da03b4819 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c
I20260809 15:46:26.840333 25460 raft_consensus.cc:2781] T f7715d71dbac447b9351fc5eb36c5e80 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 15:46:26.898192 23118 catalog_manager.cc:5495] T c0741ab44d99439abc3aa3bdd27ee160 P 61e69695258f48baa984c9d395e8ce3f reported cstate change: term changed from 1 to 2, leader changed from 65c36d164b0540f099d592f38307ee6c (127.20.202.67) to 61e69695258f48baa984c9d395e8ce3f (127.20.202.68). New cstate: current_term: 2 leader_uuid: "61e69695258f48baa984c9d395e8ce3f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 15:46:26.872865 25501 consensus_queue.cc:237] T 953eea268e17491ca76f3a555860a512 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:26.909986 25460 raft_consensus.cc:686] T f7715d71dbac447b9351fc5eb36c5e80 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 LEADER]: Becoming Leader. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Running, Role: LEADER
I20260809 15:46:26.911194 25460 consensus_queue.cc:237] T f7715d71dbac447b9351fc5eb36c5e80 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:26.893967 23228 leader_election.cc:302] T 07fcabb569414bddbd39df6cedd24e64 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c
I20260809 15:46:26.914115 25576 raft_consensus.cc:2741] T 07fcabb569414bddbd39df6cedd24e64 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader election decision vote started in defunct term 2: won
I20260809 15:46:26.915000 23228 leader_election.cc:302] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:26.918298 23517 raft_consensus.cc:1226] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Rejecting Update request from peer 61e69695258f48baa984c9d395e8ce3f for earlier term 2. Current term is 3. Ops: []
I20260809 15:46:26.920413 23228 leader_election.cc:302] T 07fcabb569414bddbd39df6cedd24e64 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:46:26.889040 25685 raft_consensus.cc:2781] T 563daf42d7784db195ea99c3cb21b370 P 61e69695258f48baa984c9d395e8ce3f [term 2 FOLLOWER]: Leader election won for term 2
I20260809 15:46:26.924500 23517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bcac4b24e7a4448abdc934bba5a624e" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:26.925966 25663 consensus_queue.cc:1035] T c0741ab44d99439abc3aa3bdd27ee160 P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260809 15:46:26.926403 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "563daf42d7784db195ea99c3cb21b370" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:26.926479 23392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bcac4b24e7a4448abdc934bba5a624e" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:26.928256 23402 raft_consensus.cc:2379] T 563daf42d7784db195ea99c3cb21b370 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 61e69695258f48baa984c9d395e8ce3f in current term 2: Already voted for candidate c0f38c1908414566aa9115bb2cbdd85d in this term.
I20260809 15:46:26.875408 23518 raft_consensus.cc:2454] T 07fcabb569414bddbd39df6cedd24e64 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 3.
I20260809 15:46:26.924131 25685 raft_consensus.cc:686] T 563daf42d7784db195ea99c3cb21b370 P 61e69695258f48baa984c9d395e8ce3f [term 2 LEADER]: Becoming Leader. State: Replica: 61e69695258f48baa984c9d395e8ce3f, State: Running, Role: LEADER
I20260809 15:46:26.897663 25477 raft_consensus.cc:2781] T 28e17dcb5efe45b8b6a8c42da03b4819 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 15:46:26.926615 25478 consensus_queue.cc:1046] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.007s
I20260809 15:46:26.933652 25694 raft_consensus.cc:2781] T 07fcabb569414bddbd39df6cedd24e64 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader election won for term 3
I20260809 15:46:26.928086 25576 raft_consensus.cc:2726] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260809 15:46:26.940056 25694 raft_consensus.cc:686] T 07fcabb569414bddbd39df6cedd24e64 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 LEADER]: Becoming Leader. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Running, Role: LEADER
I20260809 15:46:26.944185 23280 raft_consensus.cc:1226] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Rejecting Update request from peer 61e69695258f48baa984c9d395e8ce3f for earlier term 2. Current term is 3. Ops: []
I20260809 15:46:26.941196 25694 consensus_queue.cc:237] T 07fcabb569414bddbd39df6cedd24e64 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:26.954455 25686 consensus_queue.cc:1046] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260809 15:46:26.955482 25663 raft_consensus.cc:3032] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 61e69695258f48baa984c9d395e8ce3f [term 2 LEADER]: Stepping down as leader of term 2
I20260809 15:46:26.955678 25686 raft_consensus.cc:3032] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 61e69695258f48baa984c9d395e8ce3f [term 2 LEADER]: Stepping down as leader of term 2
I20260809 15:46:26.955852 25663 raft_consensus.cc:729] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 61e69695258f48baa984c9d395e8ce3f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 61e69695258f48baa984c9d395e8ce3f, State: Running, Role: LEADER
I20260809 15:46:26.956163 25686 raft_consensus.cc:729] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 61e69695258f48baa984c9d395e8ce3f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 61e69695258f48baa984c9d395e8ce3f, State: Running, Role: LEADER
I20260809 15:46:26.956240 23118 catalog_manager.cc:5495] T f9e7cadcd24d45fcbfa84209225e8f2d P c0f38c1908414566aa9115bb2cbdd85d reported cstate change: term changed from 1 to 2, leader changed from 65c36d164b0540f099d592f38307ee6c (127.20.202.67) to c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66). New cstate: current_term: 2 leader_uuid: "c0f38c1908414566aa9115bb2cbdd85d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } health_report { overall_health: UNKNOWN } } }
I20260809 15:46:26.966698 23118 catalog_manager.cc:5495] T 00a1881267a14ba8b78c81040f17a014 P c0f38c1908414566aa9115bb2cbdd85d reported cstate change: term changed from 2 to 3, leader changed from 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65) to c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66). New cstate: current_term: 3 leader_uuid: "c0f38c1908414566aa9115bb2cbdd85d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } health_report { overall_health: UNKNOWN } } }
I20260809 15:46:26.947665 25477 raft_consensus.cc:686] T 28e17dcb5efe45b8b6a8c42da03b4819 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 LEADER]: Becoming Leader. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Running, Role: LEADER
I20260809 15:46:26.957326 25686 consensus_queue.cc:260] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 61e69695258f48baa984c9d395e8ce3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:26.980124 25686 raft_consensus.cc:3037] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 61e69695258f48baa984c9d395e8ce3f [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:26.992532 25652 raft_consensus.cc:482] T 30c1740ef6874df0a332fbd6fd4b03ce P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
I20260809 15:46:26.993062 25652 raft_consensus.cc:504] T 30c1740ef6874df0a332fbd6fd4b03ce P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:46:26.996902 25713 log.cc:924] Time spent T 05ad7d8fdca64e3da527280ca5be9280 P 65c36d164b0540f099d592f38307ee6c: Append to log took a long time: real 0.108s user 0.000s sys 0.001s
I20260809 15:46:26.974972 25685 consensus_queue.cc:237] T 563daf42d7784db195ea99c3cb21b370 P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:26.983824 25663 consensus_queue.cc:260] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 61e69695258f48baa984c9d395e8ce3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:27.000391 25663 raft_consensus.cc:3037] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 61e69695258f48baa984c9d395e8ce3f [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:27.003365 25637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.0.0.1:38722 (request call id 290) took 1616 ms. Trace:
I20260809 15:46:27.008553 25652 leader_election.cc:288] T 30c1740ef6874df0a332fbd6fd4b03ce P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:27.010097 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30c1740ef6874df0a332fbd6fd4b03ce" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:27.010637 23402 raft_consensus.cc:2379] T 30c1740ef6874df0a332fbd6fd4b03ce P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c in current term 3: Already voted for candidate c0f38c1908414566aa9115bb2cbdd85d in this term.
I20260809 15:46:27.008103 23392 raft_consensus.cc:2454] T 4bcac4b24e7a4448abdc934bba5a624e P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 2.
I20260809 15:46:27.003772 25637 rpcz_store.cc:276] 0809 15:46:25.387195 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:25.397604 (+ 10409us) service_pool.cc:227] Handling call
0809 15:46:25.728511 (+330907us) ts_tablet_manager.cc:928] T 0acc47ce21c3422698d1ef7277c3ed63 P 61e69695258f48baa984c9d395e8ce3f: Initiating tablet copy from peer 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459)
0809 15:46:26.799908 (+1071397us) tablet_copy_client.cc:361] Tablet copy session begun
0809 15:46:26.906608 (+106700us) tablet_metadata.cc:641] Metadata flushed
0809 15:46:26.906955 (+ 347us) tablet_copy_client.cc:490] Wrote new tablet metadata
0809 15:46:27.002594 (+ 95639us) tablet_copy_client.cc:411] Wrote new consensus metadata
0809 15:46:27.002882 (+ 288us) tablet_replica.cc:205] Creating consensus instance
0809 15:46:27.003197 (+ 315us) ts_tablet_manager.cc:983] Replica registered.
0809 15:46:27.003253 (+ 56us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":48392,"tablet-copy.queue_time_us":201646,"thread_start_us":267449,"threads_started":2}
W20260809 15:46:26.981189 25609 consensus_queue.cc:1159] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 61e69695258f48baa984c9d395e8ce3f [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "c0f38c1908414566aa9115bb2cbdd85d" responder_term: 2 status { last_received { term: 2 index: 5 } last_committed_idx: 5 last_received_current_leader { term: 2 index: 5 } }
I20260809 15:46:27.025884 23280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30c1740ef6874df0a332fbd6fd4b03ce" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:27.026346 23280 raft_consensus.cc:2454] T 30c1740ef6874df0a332fbd6fd4b03ce P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 3.
I20260809 15:46:27.028210 23470 leader_election.cc:302] T 30c1740ef6874df0a332fbd6fd4b03ce P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters: c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:27.025489 25637 tablet_copy_client.cc:791] T 0acc47ce21c3422698d1ef7277c3ed63 P 61e69695258f48baa984c9d395e8ce3f: tablet copy: Starting download of 0 data blocks...
I20260809 15:46:27.030988 25650 raft_consensus.cc:2781] T 30c1740ef6874df0a332fbd6fd4b03ce P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260809 15:46:27.027280 24410 leader_election.cc:302] T 4bcac4b24e7a4448abdc934bba5a624e P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c
I20260809 15:46:27.043256 25637 tablet_copy_client.cc:655] T 0acc47ce21c3422698d1ef7277c3ed63 P 61e69695258f48baa984c9d395e8ce3f: tablet copy: Starting download of 1 WAL segments...
I20260809 15:46:27.049567 25687 raft_consensus.cc:2781] T 4bcac4b24e7a4448abdc934bba5a624e P 61e69695258f48baa984c9d395e8ce3f [term 2 FOLLOWER]: Leader election won for term 2
I20260809 15:46:27.050020 25687 raft_consensus.cc:686] T 4bcac4b24e7a4448abdc934bba5a624e P 61e69695258f48baa984c9d395e8ce3f [term 2 LEADER]: Becoming Leader. State: Replica: 61e69695258f48baa984c9d395e8ce3f, State: Running, Role: LEADER
I20260809 15:46:27.050820 25687 consensus_queue.cc:237] T 4bcac4b24e7a4448abdc934bba5a624e P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:27.023137 25477 consensus_queue.cc:237] T 28e17dcb5efe45b8b6a8c42da03b4819 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:27.042413 25650 raft_consensus.cc:482] T 30c1740ef6874df0a332fbd6fd4b03ce P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Starting leader election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
I20260809 15:46:27.056720 25650 raft_consensus.cc:3037] T 30c1740ef6874df0a332fbd6fd4b03ce P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:27.068184 23117 catalog_manager.cc:5495] T cee02fff2c9c48a19f43eb358a89e7b0 P 61e69695258f48baa984c9d395e8ce3f reported cstate change: term changed from 1 to 2, leader changed from 65c36d164b0540f099d592f38307ee6c (127.20.202.67) to 61e69695258f48baa984c9d395e8ce3f (127.20.202.68). New cstate: current_term: 2 leader_uuid: "61e69695258f48baa984c9d395e8ce3f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 15:46:27.071587 25650 raft_consensus.cc:504] T 30c1740ef6874df0a332fbd6fd4b03ce P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:27.072785 23117 catalog_manager.cc:5495] T 7660dfa2ef694927b8eda4100a2bcd9b P 61e69695258f48baa984c9d395e8ce3f reported cstate change: term changed from 1 to 2, leader changed from 65c36d164b0540f099d592f38307ee6c (127.20.202.67) to 61e69695258f48baa984c9d395e8ce3f (127.20.202.68). New cstate: current_term: 2 leader_uuid: "61e69695258f48baa984c9d395e8ce3f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 15:46:27.074079 23273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30c1740ef6874df0a332fbd6fd4b03ce" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:27.077253 23118 catalog_manager.cc:5495] T 2db057fd10e149a9895dff6ef8c9f038 P 67b3aeadfa7a432abe8af4d292e04ab9 reported cstate change: term changed from 1 to 2, leader changed from 65c36d164b0540f099d592f38307ee6c (127.20.202.67) to 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65). New cstate: current_term: 2 leader_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } health_report { overall_health: UNKNOWN } } }
I20260809 15:46:27.078979 23118 catalog_manager.cc:5495] T c5c2e425a74b4e64ae70b5def750434c P 67b3aeadfa7a432abe8af4d292e04ab9 reported cstate change: term changed from 1 to 2, leader changed from 65c36d164b0540f099d592f38307ee6c (127.20.202.67) to 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65). New cstate: current_term: 2 leader_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } health_report { overall_health: UNKNOWN } } }
I20260809 15:46:27.081760 23273 raft_consensus.cc:2454] T 30c1740ef6874df0a332fbd6fd4b03ce P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 3.
I20260809 15:46:27.081575 23118 catalog_manager.cc:5495] T e71a05ba1d4d42bd95c3f85f9313399e P 67b3aeadfa7a432abe8af4d292e04ab9 reported cstate change: term changed from 1 to 2, leader changed from c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66) to 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65). New cstate: current_term: 2 leader_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260809 15:46:27.082916 23470 leader_election.cc:302] T 30c1740ef6874df0a332fbd6fd4b03ce P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:46:27.090279 23118 catalog_manager.cc:5495] T 953eea268e17491ca76f3a555860a512 P 67b3aeadfa7a432abe8af4d292e04ab9 reported cstate change: term changed from 1 to 2, leader changed from 65c36d164b0540f099d592f38307ee6c (127.20.202.67) to 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65). New cstate: current_term: 2 leader_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } health_report { overall_health: UNKNOWN } } }
I20260809 15:46:27.091359 23118 catalog_manager.cc:5495] T f7715d71dbac447b9351fc5eb36c5e80 P 67b3aeadfa7a432abe8af4d292e04ab9 reported cstate change: term changed from 1 to 2, leader changed from 65c36d164b0540f099d592f38307ee6c (127.20.202.67) to 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65). New cstate: current_term: 2 leader_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } health_report { overall_health: UNKNOWN } } }
I20260809 15:46:27.088935 25652 raft_consensus.cc:2781] T 30c1740ef6874df0a332fbd6fd4b03ce P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Leader election won for term 3
I20260809 15:46:27.105218 25652 raft_consensus.cc:686] T 30c1740ef6874df0a332fbd6fd4b03ce P 65c36d164b0540f099d592f38307ee6c [term 3 LEADER]: Becoming Leader. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:46:27.106278 25652 consensus_queue.cc:237] T 30c1740ef6874df0a332fbd6fd4b03ce P 65c36d164b0540f099d592f38307ee6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:27.107678 23392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30c1740ef6874df0a332fbd6fd4b03ce" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:27.108668 23392 raft_consensus.cc:2379] T 30c1740ef6874df0a332fbd6fd4b03ce P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c in current term 3: Already voted for candidate c0f38c1908414566aa9115bb2cbdd85d in this term.
I20260809 15:46:27.109942 25650 leader_election.cc:288] T 30c1740ef6874df0a332fbd6fd4b03ce P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
W20260809 15:46:27.103363 24410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 50399.1us
I20260809 15:46:27.162184 23517 raft_consensus.cc:3037] T 00a1881267a14ba8b78c81040f17a014 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:27.177804 23273 raft_consensus.cc:1261] T 00a1881267a14ba8b78c81040f17a014 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260809 15:46:27.182013 25507 consensus_queue.cc:1035] T 00a1881267a14ba8b78c81040f17a014 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260809 15:46:27.127380 25637 tablet_copy_client.cc:531] T 0acc47ce21c3422698d1ef7277c3ed63 P 61e69695258f48baa984c9d395e8ce3f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260809 15:46:27.215083 23517 raft_consensus.cc:1261] T 00a1881267a14ba8b78c81040f17a014 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260809 15:46:27.218600 25507 consensus_queue.cc:1035] T 00a1881267a14ba8b78c81040f17a014 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20260809 15:46:27.224114 25739 log.cc:924] Time spent T 4bcac4b24e7a4448abdc934bba5a624e P 61e69695258f48baa984c9d395e8ce3f: Append to log took a long time: real 0.096s user 0.001s sys 0.000s
I20260809 15:46:27.238797 23273 raft_consensus.cc:1261] T ae6d2c08a7a94f58ac8dd698e3fd456c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260809 15:46:27.240633 25652 consensus_queue.cc:1035] T ae6d2c08a7a94f58ac8dd698e3fd456c P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 15:46:27.222807 23117 catalog_manager.cc:5495] T 563daf42d7784db195ea99c3cb21b370 P 61e69695258f48baa984c9d395e8ce3f reported cstate change: term changed from 1 to 2, leader changed from 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65) to 61e69695258f48baa984c9d395e8ce3f (127.20.202.68). New cstate: current_term: 2 leader_uuid: "61e69695258f48baa984c9d395e8ce3f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 15:46:27.243772 23117 catalog_manager.cc:5495] T 4bcac4b24e7a4448abdc934bba5a624e P 61e69695258f48baa984c9d395e8ce3f reported cstate change: term changed from 1 to 2, leader changed from c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66) to 61e69695258f48baa984c9d395e8ce3f (127.20.202.68). New cstate: current_term: 2 leader_uuid: "61e69695258f48baa984c9d395e8ce3f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 15:46:27.248396 23517 raft_consensus.cc:1261] T cee02fff2c9c48a19f43eb358a89e7b0 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Refusing update from remote peer 61e69695258f48baa984c9d395e8ce3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260809 15:46:27.250331 23276 raft_consensus.cc:1261] T cee02fff2c9c48a19f43eb358a89e7b0 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Refusing update from remote peer 61e69695258f48baa984c9d395e8ce3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260809 15:46:27.250771 24452 raft_consensus.cc:1261] T e71a05ba1d4d42bd95c3f85f9313399e P 61e69695258f48baa984c9d395e8ce3f [term 2 FOLLOWER]: Refusing update from remote peer 67b3aeadfa7a432abe8af4d292e04ab9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260809 15:46:27.259227 25477 consensus_queue.cc:1035] T e71a05ba1d4d42bd95c3f85f9313399e P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260809 15:46:27.258406 25731 consensus_queue.cc:1035] T cee02fff2c9c48a19f43eb358a89e7b0 P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.004s
I20260809 15:46:27.264032 25746 consensus_queue.cc:1035] T cee02fff2c9c48a19f43eb358a89e7b0 P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260809 15:46:27.292307 25637 tablet_bootstrap.cc:492] T 0acc47ce21c3422698d1ef7277c3ed63 P 61e69695258f48baa984c9d395e8ce3f: Bootstrap starting.
I20260809 15:46:27.291811 23115 catalog_manager.cc:5495] T 30c1740ef6874df0a332fbd6fd4b03ce P 65c36d164b0540f099d592f38307ee6c reported cstate change: term changed from 2 to 3, leader changed from 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65) to 65c36d164b0540f099d592f38307ee6c (127.20.202.67). New cstate: current_term: 3 leader_uuid: "65c36d164b0540f099d592f38307ee6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } health_report { overall_health: HEALTHY } } }
I20260809 15:46:27.314282 23392 raft_consensus.cc:1261] T e71a05ba1d4d42bd95c3f85f9313399e P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Refusing update from remote peer 67b3aeadfa7a432abe8af4d292e04ab9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260809 15:46:27.315764 25477 consensus_queue.cc:1035] T e71a05ba1d4d42bd95c3f85f9313399e P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260809 15:46:27.323908 25681 raft_consensus.cc:482] T 7660dfa2ef694927b8eda4100a2bcd9b P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:27.324165 23521 raft_consensus.cc:1261] T 953eea268e17491ca76f3a555860a512 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Refusing update from remote peer 67b3aeadfa7a432abe8af4d292e04ab9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 15:46:27.324585 25681 raft_consensus.cc:504] T 7660dfa2ef694927b8eda4100a2bcd9b P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:27.330756 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7660dfa2ef694927b8eda4100a2bcd9b" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:27.332633 25532 consensus_queue.cc:1035] T 953eea268e17491ca76f3a555860a512 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260809 15:46:27.333706 23521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7660dfa2ef694927b8eda4100a2bcd9b" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:27.334636 23521 raft_consensus.cc:2454] T 7660dfa2ef694927b8eda4100a2bcd9b P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 2.
I20260809 15:46:27.327005 25681 leader_election.cc:288] T 7660dfa2ef694927b8eda4100a2bcd9b P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:27.345331 23353 leader_election.cc:302] T 7660dfa2ef694927b8eda4100a2bcd9b P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters: 61e69695258f48baa984c9d395e8ce3f
I20260809 15:46:27.348366 23402 raft_consensus.cc:1261] T ae6d2c08a7a94f58ac8dd698e3fd456c P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260809 15:46:27.347038 25681 raft_consensus.cc:2781] T 7660dfa2ef694927b8eda4100a2bcd9b P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260809 15:46:27.352389 25652 consensus_queue.cc:1035] T ae6d2c08a7a94f58ac8dd698e3fd456c P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260809 15:46:27.351646 25681 raft_consensus.cc:482] T 7660dfa2ef694927b8eda4100a2bcd9b P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:46:27.363098 25681 raft_consensus.cc:3037] T 7660dfa2ef694927b8eda4100a2bcd9b P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:27.366618 25650 raft_consensus.cc:482] T 297c82ed906d4b48b184985eed3de601 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:27.367254 25650 raft_consensus.cc:504] T 297c82ed906d4b48b184985eed3de601 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:27.374615 25650 leader_election.cc:288] T 297c82ed906d4b48b184985eed3de601 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:27.378672 23520 raft_consensus.cc:1261] T f7715d71dbac447b9351fc5eb36c5e80 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Refusing update from remote peer 67b3aeadfa7a432abe8af4d292e04ab9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 15:46:27.382640 25694 consensus_queue.cc:1035] T f7715d71dbac447b9351fc5eb36c5e80 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260809 15:46:27.384680 23400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "297c82ed906d4b48b184985eed3de601" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:27.385519 23400 raft_consensus.cc:2454] T 297c82ed906d4b48b184985eed3de601 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 3.
I20260809 15:46:27.387058 23520 raft_consensus.cc:1261] T 7660dfa2ef694927b8eda4100a2bcd9b P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Refusing update from remote peer 61e69695258f48baa984c9d395e8ce3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260809 15:46:27.386162 23118 catalog_manager.cc:5495] T 28e17dcb5efe45b8b6a8c42da03b4819 P 67b3aeadfa7a432abe8af4d292e04ab9 reported cstate change: term changed from 1 to 2, leader changed from c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66) to 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65). New cstate: current_term: 2 leader_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } health_report { overall_health: UNKNOWN } } }
I20260809 15:46:27.388295 23118 catalog_manager.cc:5495] T 07fcabb569414bddbd39df6cedd24e64 P 67b3aeadfa7a432abe8af4d292e04ab9 reported cstate change: term changed from 1 to 3, leader changed from c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66) to 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65). New cstate: current_term: 3 leader_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } health_report { overall_health: UNKNOWN } } }
I20260809 15:46:27.388831 25731 consensus_queue.cc:1035] T 7660dfa2ef694927b8eda4100a2bcd9b P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260809 15:46:27.389376 25650 raft_consensus.cc:482] T 472816994d814cb198b69e5248697440 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:27.394915 23472 leader_election.cc:302] T 297c82ed906d4b48b184985eed3de601 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:46:27.395807 25652 raft_consensus.cc:2781] T 297c82ed906d4b48b184985eed3de601 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260809 15:46:27.396157 25652 raft_consensus.cc:482] T 297c82ed906d4b48b184985eed3de601 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:46:27.396517 25652 raft_consensus.cc:3037] T 297c82ed906d4b48b184985eed3de601 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:46:27.402791 23397 raft_consensus.cc:1261] T c5c2e425a74b4e64ae70b5def750434c P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Refusing update from remote peer 67b3aeadfa7a432abe8af4d292e04ab9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 15:46:27.404241 25470 raft_consensus.cc:482] T 472816994d814cb198b69e5248697440 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:27.404873 25470 raft_consensus.cc:504] T 472816994d814cb198b69e5248697440 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:27.413584 23397 raft_consensus.cc:1261] T 2db057fd10e149a9895dff6ef8c9f038 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Refusing update from remote peer 67b3aeadfa7a432abe8af4d292e04ab9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 15:46:27.415251 23273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "297c82ed906d4b48b184985eed3de601" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:27.418964 23280 raft_consensus.cc:1261] T 563daf42d7784db195ea99c3cb21b370 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Refusing update from remote peer 61e69695258f48baa984c9d395e8ce3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260809 15:46:27.412406 25650 raft_consensus.cc:504] T 472816994d814cb198b69e5248697440 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:27.421069 25650 leader_election.cc:288] T 472816994d814cb198b69e5248697440 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:27.422451 23520 raft_consensus.cc:1261] T 2db057fd10e149a9895dff6ef8c9f038 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Refusing update from remote peer 67b3aeadfa7a432abe8af4d292e04ab9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 15:46:27.431697 23397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472816994d814cb198b69e5248697440" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:27.440483 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472816994d814cb198b69e5248697440" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:27.441296 24468 raft_consensus.cc:2454] T 472816994d814cb198b69e5248697440 P 61e69695258f48baa984c9d395e8ce3f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 2.
I20260809 15:46:27.431743 25694 consensus_queue.cc:1035] T 2db057fd10e149a9895dff6ef8c9f038 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.010s
I20260809 15:46:27.442827 25477 consensus_queue.cc:1035] T c5c2e425a74b4e64ae70b5def750434c P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 15:46:27.443022 23468 leader_election.cc:302] T 472816994d814cb198b69e5248697440 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c; no voters:
I20260809 15:46:27.430838 25731 consensus_queue.cc:1035] T 563daf42d7784db195ea99c3cb21b370 P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.019s
I20260809 15:46:27.449141 23517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472816994d814cb198b69e5248697440" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:27.449796 23517 raft_consensus.cc:2454] T 472816994d814cb198b69e5248697440 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 2.
I20260809 15:46:27.450531 25507 raft_consensus.cc:482] T d0721a4403a440f383318a3cea9d1442 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:27.447368 23397 raft_consensus.cc:2454] T 472816994d814cb198b69e5248697440 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 2.
I20260809 15:46:27.451164 25507 raft_consensus.cc:504] T d0721a4403a440f383318a3cea9d1442 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:27.453380 25507 leader_election.cc:288] T d0721a4403a440f383318a3cea9d1442 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:27.450651 23273 raft_consensus.cc:2454] T 297c82ed906d4b48b184985eed3de601 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 3.
I20260809 15:46:27.460332 21289 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 15:46:27.464543 23353 leader_election.cc:302] T 472816994d814cb198b69e5248697440 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:46:27.469892 25470 leader_election.cc:288] T 472816994d814cb198b69e5248697440 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:27.473359 23517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0721a4403a440f383318a3cea9d1442" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:27.474942 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472816994d814cb198b69e5248697440" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:27.468973 25507 raft_consensus.cc:2781] T 472816994d814cb198b69e5248697440 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260809 15:46:27.484045 25507 raft_consensus.cc:482] T 472816994d814cb198b69e5248697440 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:46:27.484445 25507 raft_consensus.cc:3037] T 472816994d814cb198b69e5248697440 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:27.485183 25652 raft_consensus.cc:504] T 297c82ed906d4b48b184985eed3de601 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:27.487241 25652 leader_election.cc:288] T 297c82ed906d4b48b184985eed3de601 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 4 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:27.483520 24468 raft_consensus.cc:2454] T 472816994d814cb198b69e5248697440 P 61e69695258f48baa984c9d395e8ce3f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 2.
I20260809 15:46:27.489411 23397 raft_consensus.cc:1261] T 563daf42d7784db195ea99c3cb21b370 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Refusing update from remote peer 61e69695258f48baa984c9d395e8ce3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260809 15:46:27.490829 25772 raft_consensus.cc:2781] T 472816994d814cb198b69e5248697440 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260809 15:46:27.491818 25746 consensus_queue.cc:1035] T 563daf42d7784db195ea99c3cb21b370 P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260809 15:46:27.458051 25681 raft_consensus.cc:504] T 7660dfa2ef694927b8eda4100a2bcd9b P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:27.456936 25477 consensus_queue.cc:1035] T 2db057fd10e149a9895dff6ef8c9f038 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.023s
I20260809 15:46:27.493798 23517 raft_consensus.cc:2454] T d0721a4403a440f383318a3cea9d1442 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 2.
I20260809 15:46:27.502452 23274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0721a4403a440f383318a3cea9d1442" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:27.503794 23392 raft_consensus.cc:1226] T 7660dfa2ef694927b8eda4100a2bcd9b P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Rejecting Update request from peer 61e69695258f48baa984c9d395e8ce3f for earlier term 2. Current term is 3. Ops: []
I20260809 15:46:27.506152 23521 raft_consensus.cc:1261] T 07fcabb569414bddbd39df6cedd24e64 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Refusing update from remote peer 67b3aeadfa7a432abe8af4d292e04ab9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260809 15:46:27.509704 23353 leader_election.cc:302] T d0721a4403a440f383318a3cea9d1442 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:46:27.511500 25639 raft_consensus.cc:482] T 3924b648461e463b9d7925e5671d68ae P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
I20260809 15:46:27.512236 25675 raft_consensus.cc:2781] T d0721a4403a440f383318a3cea9d1442 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260809 15:46:27.512672 25675 raft_consensus.cc:482] T d0721a4403a440f383318a3cea9d1442 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:46:27.513042 25675 raft_consensus.cc:3037] T d0721a4403a440f383318a3cea9d1442 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:27.515321 23404 raft_consensus.cc:1261] T 4bcac4b24e7a4448abdc934bba5a624e P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Refusing update from remote peer 61e69695258f48baa984c9d395e8ce3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260809 15:46:27.516449 23274 raft_consensus.cc:2454] T d0721a4403a440f383318a3cea9d1442 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 1.
I20260809 15:46:27.516487 25695 consensus_queue.cc:1035] T 07fcabb569414bddbd39df6cedd24e64 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260809 15:46:27.516850 25731 consensus_queue.cc:1035] T 4bcac4b24e7a4448abdc934bba5a624e P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260809 15:46:27.508692 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "297c82ed906d4b48b184985eed3de601" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:27.521555 23402 raft_consensus.cc:3037] T 297c82ed906d4b48b184985eed3de601 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:46:27.528920 23393 raft_consensus.cc:1261] T 07fcabb569414bddbd39df6cedd24e64 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Refusing update from remote peer 67b3aeadfa7a432abe8af4d292e04ab9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260809 15:46:27.529934 23399 raft_consensus.cc:1261] T 30c1740ef6874df0a332fbd6fd4b03ce P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260809 15:46:27.531386 23276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "297c82ed906d4b48b184985eed3de601" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:27.537441 23276 raft_consensus.cc:3037] T 297c82ed906d4b48b184985eed3de601 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:46:27.540448 25772 raft_consensus.cc:482] T 472816994d814cb198b69e5248697440 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:46:27.540894 25772 raft_consensus.cc:3037] T 472816994d814cb198b69e5248697440 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:27.547889 25731 consensus_queue.cc:1046] T 7660dfa2ef694927b8eda4100a2bcd9b P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260809 15:46:27.548748 25731 raft_consensus.cc:3032] T 7660dfa2ef694927b8eda4100a2bcd9b P 61e69695258f48baa984c9d395e8ce3f [term 2 LEADER]: Stepping down as leader of term 2
I20260809 15:46:27.549114 25731 raft_consensus.cc:729] T 7660dfa2ef694927b8eda4100a2bcd9b P 61e69695258f48baa984c9d395e8ce3f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 61e69695258f48baa984c9d395e8ce3f, State: Running, Role: LEADER
I20260809 15:46:27.550041 25731 consensus_queue.cc:260] T 7660dfa2ef694927b8eda4100a2bcd9b P 61e69695258f48baa984c9d395e8ce3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:27.557313 25746 raft_consensus.cc:482] T 16ca412a0dae4feebe85dd01bf1a9173 P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
I20260809 15:46:27.557894 25746 raft_consensus.cc:504] T 16ca412a0dae4feebe85dd01bf1a9173 P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:27.512187 25639 raft_consensus.cc:504] T 3924b648461e463b9d7925e5671d68ae P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:27.559966 25746 leader_election.cc:288] T 16ca412a0dae4feebe85dd01bf1a9173 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:27.560243 25639 leader_election.cc:288] T 3924b648461e463b9d7925e5671d68ae P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:27.562120 23529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16ca412a0dae4feebe85dd01bf1a9173" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:27.562467 23274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16ca412a0dae4feebe85dd01bf1a9173" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:27.563200 23274 raft_consensus.cc:2454] T 16ca412a0dae4feebe85dd01bf1a9173 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 2.
I20260809 15:46:27.564109 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3924b648461e463b9d7925e5671d68ae" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:27.564754 24468 raft_consensus.cc:2454] T 3924b648461e463b9d7925e5671d68ae P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 3.
I20260809 15:46:27.564931 24406 leader_election.cc:302] T 16ca412a0dae4feebe85dd01bf1a9173 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:46:27.566740 25478 raft_consensus.cc:2781] T 16ca412a0dae4feebe85dd01bf1a9173 P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260809 15:46:27.567121 25478 raft_consensus.cc:482] T 16ca412a0dae4feebe85dd01bf1a9173 P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Starting leader election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
I20260809 15:46:27.567487 25478 raft_consensus.cc:3037] T 16ca412a0dae4feebe85dd01bf1a9173 P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:27.551213 25731 raft_consensus.cc:3037] T 7660dfa2ef694927b8eda4100a2bcd9b P 61e69695258f48baa984c9d395e8ce3f [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:27.572575 23276 raft_consensus.cc:2454] T 297c82ed906d4b48b184985eed3de601 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 4.
I20260809 15:46:27.573984 23470 leader_election.cc:302] T 297c82ed906d4b48b184985eed3de601 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:46:27.575024 25652 raft_consensus.cc:2781] T 297c82ed906d4b48b184985eed3de601 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Leader election won for term 4
I20260809 15:46:27.575852 25652 raft_consensus.cc:686] T 297c82ed906d4b48b184985eed3de601 P 65c36d164b0540f099d592f38307ee6c [term 4 LEADER]: Becoming Leader. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:46:27.576869 25652 consensus_queue.cc:237] T 297c82ed906d4b48b184985eed3de601 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:27.579828 25650 raft_consensus.cc:482] T 16ca412a0dae4feebe85dd01bf1a9173 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:27.580415 25650 raft_consensus.cc:504] T 16ca412a0dae4feebe85dd01bf1a9173 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:27.582428 25650 leader_election.cc:288] T 16ca412a0dae4feebe85dd01bf1a9173 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:27.584546 23468 leader_election.cc:302] T 3924b648461e463b9d7925e5671d68ae P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c; no voters:
I20260809 15:46:27.587780 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16ca412a0dae4feebe85dd01bf1a9173" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:27.590626 23518 raft_consensus.cc:3037] T 4bcac4b24e7a4448abdc934bba5a624e P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:27.548869 25507 raft_consensus.cc:504] T 472816994d814cb198b69e5248697440 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:27.597175 25507 leader_election.cc:288] T 472816994d814cb198b69e5248697440 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 election: Requested vote from peers 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:27.598691 23520 raft_consensus.cc:1261] T 28e17dcb5efe45b8b6a8c42da03b4819 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Refusing update from remote peer 67b3aeadfa7a432abe8af4d292e04ab9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 15:46:27.599511 25650 raft_consensus.cc:2781] T 3924b648461e463b9d7925e5671d68ae P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260809 15:46:27.599937 25650 raft_consensus.cc:482] T 3924b648461e463b9d7925e5671d68ae P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Starting leader election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
I20260809 15:46:27.600311 25650 raft_consensus.cc:3037] T 3924b648461e463b9d7925e5671d68ae P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:27.602025 23402 raft_consensus.cc:2454] T 297c82ed906d4b48b184985eed3de601 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 4.
I20260809 15:46:27.609108 23529 raft_consensus.cc:2379] T 16ca412a0dae4feebe85dd01bf1a9173 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 61e69695258f48baa984c9d395e8ce3f in current term 2: Already voted for candidate 65c36d164b0540f099d592f38307ee6c in this term.
I20260809 15:46:27.618893 23393 raft_consensus.cc:1261] T f7715d71dbac447b9351fc5eb36c5e80 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Refusing update from remote peer 67b3aeadfa7a432abe8af4d292e04ab9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 15:46:27.618257 25477 consensus_queue.cc:1035] T 28e17dcb5efe45b8b6a8c42da03b4819 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20260809 15:46:27.621069 23395 raft_consensus.cc:1261] T 953eea268e17491ca76f3a555860a512 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Refusing update from remote peer 67b3aeadfa7a432abe8af4d292e04ab9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 15:46:27.619941 23399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3924b648461e463b9d7925e5671d68ae" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:27.622231 23399 raft_consensus.cc:2379] T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c in current term 3: Already voted for candidate c0f38c1908414566aa9115bb2cbdd85d in this term.
I20260809 15:46:27.629444 23517 raft_consensus.cc:1261] T c5c2e425a74b4e64ae70b5def750434c P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Refusing update from remote peer 67b3aeadfa7a432abe8af4d292e04ab9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 15:46:27.631609 23529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472816994d814cb198b69e5248697440" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:27.636432 25681 leader_election.cc:288] T 7660dfa2ef694927b8eda4100a2bcd9b P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 election: Requested vote from peers 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:27.627795 25639 consensus_queue.cc:1035] T 30c1740ef6874df0a332fbd6fd4b03ce P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.009s
I20260809 15:46:27.637535 25470 raft_consensus.cc:482] T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:27.638108 25470 raft_consensus.cc:504] T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:27.619948 24452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472816994d814cb198b69e5248697440" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f"
I20260809 15:46:27.639938 24452 raft_consensus.cc:3037] T 472816994d814cb198b69e5248697440 P 61e69695258f48baa984c9d395e8ce3f [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:27.628290 23276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16ca412a0dae4feebe85dd01bf1a9173" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:27.636461 23395 raft_consensus.cc:1261] T 28e17dcb5efe45b8b6a8c42da03b4819 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Refusing update from remote peer 67b3aeadfa7a432abe8af4d292e04ab9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260809 15:46:27.645797 23472 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 44576.5us
I20260809 15:46:27.668268 25700 consensus_queue.cc:1035] T c5c2e425a74b4e64ae70b5def750434c P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.023s
I20260809 15:46:27.670683 23520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7660dfa2ef694927b8eda4100a2bcd9b" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:27.671945 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7660dfa2ef694927b8eda4100a2bcd9b" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f"
I20260809 15:46:27.672273 25773 raft_consensus.cc:482] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:27.672834 25773 raft_consensus.cc:504] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:27.640384 25675 raft_consensus.cc:504] T d0721a4403a440f383318a3cea9d1442 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:27.678722 25675 leader_election.cc:288] T d0721a4403a440f383318a3cea9d1442 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:27.654356 25694 consensus_queue.cc:1035] T 953eea268e17491ca76f3a555860a512 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.025s
I20260809 15:46:27.679647 23274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0721a4403a440f383318a3cea9d1442" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:27.680178 23274 raft_consensus.cc:3037] T d0721a4403a440f383318a3cea9d1442 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Advancing to term 3
I20260809 15:46:27.682359 23518 raft_consensus.cc:1261] T 4bcac4b24e7a4448abdc934bba5a624e P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Refusing update from remote peer 61e69695258f48baa984c9d395e8ce3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260809 15:46:27.684801 24471 raft_consensus.cc:2396] T 7660dfa2ef694927b8eda4100a2bcd9b P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d for term 3 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260809 15:46:27.685976 23520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0721a4403a440f383318a3cea9d1442" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:27.646922 25477 consensus_queue.cc:1035] T f7715d71dbac447b9351fc5eb36c5e80 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.019s
I20260809 15:46:27.660156 23276 raft_consensus.cc:2454] T 16ca412a0dae4feebe85dd01bf1a9173 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 2.
I20260809 15:46:27.689236 25470 leader_election.cc:288] T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:27.689765 25587 consensus_queue.cc:1035] T 07fcabb569414bddbd39df6cedd24e64 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.079s
I20260809 15:46:27.698868 23518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad8a3b25d7b49e5a2142d5709edd0c7" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:27.702414 23274 raft_consensus.cc:2454] T d0721a4403a440f383318a3cea9d1442 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 3.
I20260809 15:46:27.704123 23351 leader_election.cc:302] T d0721a4403a440f383318a3cea9d1442 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:46:27.704938 25470 raft_consensus.cc:2781] T d0721a4403a440f383318a3cea9d1442 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader election won for term 3
I20260809 15:46:27.705470 25470 raft_consensus.cc:686] T d0721a4403a440f383318a3cea9d1442 P c0f38c1908414566aa9115bb2cbdd85d [term 3 LEADER]: Becoming Leader. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:46:27.706485 25470 consensus_queue.cc:237] T d0721a4403a440f383318a3cea9d1442 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:27.686625 23520 raft_consensus.cc:3037] T d0721a4403a440f383318a3cea9d1442 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:27.694976 24452 raft_consensus.cc:2454] T 472816994d814cb198b69e5248697440 P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 3.
I20260809 15:46:27.717880 25587 consensus_queue.cc:1035] T 28e17dcb5efe45b8b6a8c42da03b4819 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 15:46:27.686015 25772 raft_consensus.cc:504] T 472816994d814cb198b69e5248697440 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:27.718912 23117 catalog_manager.cc:5495] T 297c82ed906d4b48b184985eed3de601 P 65c36d164b0540f099d592f38307ee6c reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "65c36d164b0540f099d592f38307ee6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } health_report { overall_health: HEALTHY } } }
I20260809 15:46:27.728236 25772 leader_election.cc:288] T 472816994d814cb198b69e5248697440 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:27.728878 23529 raft_consensus.cc:2379] T 472816994d814cb198b69e5248697440 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d in current term 3: Already voted for candidate 65c36d164b0540f099d592f38307ee6c in this term.
W20260809 15:46:27.729394 23470 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 47165.2us
I20260809 15:46:27.689287 25639 raft_consensus.cc:482] T cef3e7277d7d4760824a3436ed4a847d P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:27.731822 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3e94b7e50754a81b3d72a07476cb7b9" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:27.734071 23517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3e94b7e50754a81b3d72a07476cb7b9" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:27.735086 25675 raft_consensus.cc:482] T 6ae14a74a26f434db08c652e097385c8 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:27.735368 24448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472816994d814cb198b69e5248697440" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f"
I20260809 15:46:27.736109 24448 raft_consensus.cc:2379] T 472816994d814cb198b69e5248697440 P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c in current term 3: Already voted for candidate c0f38c1908414566aa9115bb2cbdd85d in this term.
I20260809 15:46:27.736277 25675 raft_consensus.cc:504] T 6ae14a74a26f434db08c652e097385c8 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:27.737528 23274 raft_consensus.cc:1261] T 30c1740ef6874df0a332fbd6fd4b03ce P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260809 15:46:27.738313 23470 leader_election.cc:302] T 16ca412a0dae4feebe85dd01bf1a9173 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:46:27.738878 23517 raft_consensus.cc:2454] T d3e94b7e50754a81b3d72a07476cb7b9 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 3.
I20260809 15:46:27.723850 25773 leader_election.cc:288] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:27.684713 25478 raft_consensus.cc:504] T 16ca412a0dae4feebe85dd01bf1a9173 P 61e69695258f48baa984c9d395e8ce3f [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:27.744642 25478 leader_election.cc:288] T 16ca412a0dae4feebe85dd01bf1a9173 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 2 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:27.745054 25675 leader_election.cc:288] T 6ae14a74a26f434db08c652e097385c8 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:27.745195 24468 raft_consensus.cc:2454] T 16ca412a0dae4feebe85dd01bf1a9173 P 61e69695258f48baa984c9d395e8ce3f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 2.
I20260809 15:46:27.746510 23529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ae14a74a26f434db08c652e097385c8" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:27.747154 23529 raft_consensus.cc:2454] T 6ae14a74a26f434db08c652e097385c8 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 2.
I20260809 15:46:27.740825 23345 leader_election.cc:302] T 7660dfa2ef694927b8eda4100a2bcd9b P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c
I20260809 15:46:27.749235 25773 raft_consensus.cc:2726] T 7660dfa2ef694927b8eda4100a2bcd9b P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260809 15:46:27.749938 23274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ae14a74a26f434db08c652e097385c8" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:27.741245 25746 consensus_queue.cc:1035] T 4bcac4b24e7a4448abdc934bba5a624e P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260809 15:46:27.739988 25639 raft_consensus.cc:504] T cef3e7277d7d4760824a3436ed4a847d P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:27.740445 24471 raft_consensus.cc:2396] T d3e94b7e50754a81b3d72a07476cb7b9 P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d for term 4 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260809 15:46:27.747807 23276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16ca412a0dae4feebe85dd01bf1a9173" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:27.750552 23274 raft_consensus.cc:2454] T 6ae14a74a26f434db08c652e097385c8 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 2.
I20260809 15:46:27.753939 24448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad8a3b25d7b49e5a2142d5709edd0c7" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:27.755651 23345 leader_election.cc:302] T 472816994d814cb198b69e5248697440 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c
I20260809 15:46:27.760825 23353 leader_election.cc:302] T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:46:27.759711 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472816994d814cb198b69e5248697440" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:27.761742 23402 raft_consensus.cc:2379] T 472816994d814cb198b69e5248697440 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c in current term 3: Already voted for candidate c0f38c1908414566aa9115bb2cbdd85d in this term.
I20260809 15:46:27.740223 25772 raft_consensus.cc:482] T 998c6cee72cf40228c6dba064c0db2da P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:27.765244 23517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16ca412a0dae4feebe85dd01bf1a9173" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:27.766359 25650 raft_consensus.cc:504] T 3924b648461e463b9d7925e5671d68ae P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:27.769805 25772 raft_consensus.cc:504] T 998c6cee72cf40228c6dba064c0db2da P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:27.770337 25738 raft_consensus.cc:482] T d3e94b7e50754a81b3d72a07476cb7b9 P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:27.770790 25738 raft_consensus.cc:504] T d3e94b7e50754a81b3d72a07476cb7b9 P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:27.772591 25639 leader_election.cc:288] T cef3e7277d7d4760824a3436ed4a847d P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:27.776219 24448 raft_consensus.cc:2454] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 61e69695258f48baa984c9d395e8ce3f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 2.
I20260809 15:46:27.776386 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3924b648461e463b9d7925e5671d68ae" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f"
I20260809 15:46:27.776787 23353 leader_election.cc:302] T 6ae14a74a26f434db08c652e097385c8 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:46:27.777951 25507 raft_consensus.cc:2781] T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260809 15:46:27.778156 25772 leader_election.cc:288] T 998c6cee72cf40228c6dba064c0db2da P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:27.778306 25507 raft_consensus.cc:482] T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:46:27.778716 25507 raft_consensus.cc:3037] T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:46:27.778806 25785 raft_consensus.cc:2781] T 16ca412a0dae4feebe85dd01bf1a9173 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260809 15:46:27.779253 25785 raft_consensus.cc:482] T 16ca412a0dae4feebe85dd01bf1a9173 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:46:27.779623 25785 raft_consensus.cc:3037] T 16ca412a0dae4feebe85dd01bf1a9173 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:27.780553 25483 raft_consensus.cc:2781] T 6ae14a74a26f434db08c652e097385c8 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260809 15:46:27.780930 25776 raft_consensus.cc:2781] T 472816994d814cb198b69e5248697440 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader election won for term 3
I20260809 15:46:27.796227 23472 leader_election.cc:302] T 472816994d814cb198b69e5248697440 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:27.780925 25483 raft_consensus.cc:482] T 6ae14a74a26f434db08c652e097385c8 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:46:27.799427 23280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998c6cee72cf40228c6dba064c0db2da" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:27.787484 25650 leader_election.cc:288] T 3924b648461e463b9d7925e5671d68ae P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:27.811551 23280 raft_consensus.cc:2454] T 998c6cee72cf40228c6dba064c0db2da P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 4.
I20260809 15:46:27.813460 23470 leader_election.cc:302] T 998c6cee72cf40228c6dba064c0db2da P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:46:27.811044 25773 raft_consensus.cc:482] T d2d4f749fcfd4fb3bfb8115af3471598 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:27.816344 25795 consensus_queue.cc:1035] T 30c1740ef6874df0a332fbd6fd4b03ce P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.026s
I20260809 15:46:27.807399 25639 raft_consensus.cc:482] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:46:27.809129 25483 raft_consensus.cc:3037] T 6ae14a74a26f434db08c652e097385c8 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:27.822490 23270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cef3e7277d7d4760824a3436ed4a847d" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:27.823173 23270 raft_consensus.cc:2454] T cef3e7277d7d4760824a3436ed4a847d P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 2.
I20260809 15:46:27.824008 25773 raft_consensus.cc:504] T d2d4f749fcfd4fb3bfb8115af3471598 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:27.825839 25773 leader_election.cc:288] T d2d4f749fcfd4fb3bfb8115af3471598 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:27.826265 25773 raft_consensus.cc:482] T 0c29937a80724ddb9d07a27534a146f3 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:27.826683 25773 raft_consensus.cc:504] T 0c29937a80724ddb9d07a27534a146f3 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:27.817205 23276 raft_consensus.cc:2454] T 16ca412a0dae4feebe85dd01bf1a9173 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 2.
I20260809 15:46:27.828233 24406 leader_election.cc:302] T 16ca412a0dae4feebe85dd01bf1a9173 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:46:27.821730 25639 raft_consensus.cc:504] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:27.829826 23520 raft_consensus.cc:2454] T d0721a4403a440f383318a3cea9d1442 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 3.
I20260809 15:46:27.830863 25639 leader_election.cc:288] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:27.832346 25478 raft_consensus.cc:2781] T 16ca412a0dae4feebe85dd01bf1a9173 P 61e69695258f48baa984c9d395e8ce3f [term 2 FOLLOWER]: Leader election won for term 2
I20260809 15:46:27.817947 25772 raft_consensus.cc:2781] T 998c6cee72cf40228c6dba064c0db2da P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260809 15:46:27.832855 25478 raft_consensus.cc:686] T 16ca412a0dae4feebe85dd01bf1a9173 P 61e69695258f48baa984c9d395e8ce3f [term 2 LEADER]: Becoming Leader. State: Replica: 61e69695258f48baa984c9d395e8ce3f, State: Running, Role: LEADER
I20260809 15:46:27.833009 25772 raft_consensus.cc:482] T 998c6cee72cf40228c6dba064c0db2da P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:46:27.833374 25772 raft_consensus.cc:3037] T 998c6cee72cf40228c6dba064c0db2da P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Advancing to term 5
I20260809 15:46:27.816987 25789 raft_consensus.cc:482] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:46:27.840674 23470 leader_election.cc:302] T cef3e7277d7d4760824a3436ed4a847d P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:46:27.833820 24471 raft_consensus.cc:2454] T 3924b648461e463b9d7925e5671d68ae P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 3.
I20260809 15:46:27.815765 25650 raft_consensus.cc:2726] T 472816994d814cb198b69e5248697440 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260809 15:46:27.817021 25807 raft_consensus.cc:482] T cef3e7277d7d4760824a3436ed4a847d P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:27.851874 23468 leader_election.cc:302] T 3924b648461e463b9d7925e5671d68ae P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c; no voters:
I20260809 15:46:27.851994 23399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3e94b7e50754a81b3d72a07476cb7b9" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:27.853497 23518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3e94b7e50754a81b3d72a07476cb7b9" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:27.833799 25738 leader_election.cc:288] T d3e94b7e50754a81b3d72a07476cb7b9 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:27.854712 25637 tablet_bootstrap.cc:492] T 0acc47ce21c3422698d1ef7277c3ed63 P 61e69695258f48baa984c9d395e8ce3f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20260809 15:46:27.854081 25793 raft_consensus.cc:2781] T cef3e7277d7d4760824a3436ed4a847d P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260809 15:46:27.863824 25650 raft_consensus.cc:2781] T 3924b648461e463b9d7925e5671d68ae P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Leader election won for term 3
I20260809 15:46:27.865181 25650 raft_consensus.cc:686] T 3924b648461e463b9d7925e5671d68ae P 65c36d164b0540f099d592f38307ee6c [term 3 LEADER]: Becoming Leader. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:46:27.857875 23520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2d4f749fcfd4fb3bfb8115af3471598" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:27.861634 25793 raft_consensus.cc:482] T cef3e7277d7d4760824a3436ed4a847d P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:46:27.880296 25807 raft_consensus.cc:504] T cef3e7277d7d4760824a3436ed4a847d P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:27.865911 23276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2d4f749fcfd4fb3bfb8115af3471598" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:27.833779 25478 consensus_queue.cc:237] T 16ca412a0dae4feebe85dd01bf1a9173 P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:27.890125 25807 leader_election.cc:288] T cef3e7277d7d4760824a3436ed4a847d P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:27.864009 23518 raft_consensus.cc:2454] T d3e94b7e50754a81b3d72a07476cb7b9 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 3.
I20260809 15:46:27.879673 25650 consensus_queue.cc:237] T 3924b648461e463b9d7925e5671d68ae P 65c36d164b0540f099d592f38307ee6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:27.895239 23274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cef3e7277d7d4760824a3436ed4a847d" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:27.896948 23274 raft_consensus.cc:2454] T cef3e7277d7d4760824a3436ed4a847d P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 2.
I20260809 15:46:27.886564 25793 raft_consensus.cc:3037] T cef3e7277d7d4760824a3436ed4a847d P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:27.908972 23270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c29937a80724ddb9d07a27534a146f3" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:27.910632 23351 leader_election.cc:302] T cef3e7277d7d4760824a3436ed4a847d P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:46:27.851390 25789 raft_consensus.cc:504] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:27.915318 25637 tablet_bootstrap.cc:492] T 0acc47ce21c3422698d1ef7277c3ed63 P 61e69695258f48baa984c9d395e8ce3f: Bootstrap complete.
I20260809 15:46:27.926195 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cef3e7277d7d4760824a3436ed4a847d" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
W20260809 15:46:27.931631 25802 log.cc:924] Time spent T 07fcabb569414bddbd39df6cedd24e64 P c0f38c1908414566aa9115bb2cbdd85d: Append to log took a long time: real 0.070s user 0.001s sys 0.000s
I20260809 15:46:27.934495 24410 leader_election.cc:302] T d3e94b7e50754a81b3d72a07476cb7b9 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c; no voters:
I20260809 15:46:27.935686 25637 ts_tablet_manager.cc:1397] T 0acc47ce21c3422698d1ef7277c3ed63 P 61e69695258f48baa984c9d395e8ce3f: Time spent bootstrapping tablet: real 0.644s user 0.022s sys 0.007s
I20260809 15:46:27.902995 23520 raft_consensus.cc:2454] T d2d4f749fcfd4fb3bfb8115af3471598 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 2.
I20260809 15:46:27.945472 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad8a3b25d7b49e5a2142d5709edd0c7" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:27.946208 24471 raft_consensus.cc:2379] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 61e69695258f48baa984c9d395e8ce3f [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c in current term 2: Already voted for candidate 61e69695258f48baa984c9d395e8ce3f in this term.
I20260809 15:46:27.918267 25681 raft_consensus.cc:2781] T cef3e7277d7d4760824a3436ed4a847d P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260809 15:46:27.942507 25731 raft_consensus.cc:2781] T d3e94b7e50754a81b3d72a07476cb7b9 P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260809 15:46:27.954144 23353 leader_election.cc:302] T d2d4f749fcfd4fb3bfb8115af3471598 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters: 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:27.945292 25773 leader_election.cc:288] T 0c29937a80724ddb9d07a27534a146f3 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:27.951694 25681 raft_consensus.cc:482] T cef3e7277d7d4760824a3436ed4a847d P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:46:27.960125 25776 raft_consensus.cc:686] T 472816994d814cb198b69e5248697440 P c0f38c1908414566aa9115bb2cbdd85d [term 3 LEADER]: Becoming Leader. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:46:27.955051 25731 raft_consensus.cc:482] T d3e94b7e50754a81b3d72a07476cb7b9 P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:46:27.972851 25573 raft_consensus.cc:482] T 0da4d5283c8a46878bd39b85e5c71f66 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:27.973310 25573 raft_consensus.cc:504] T 0da4d5283c8a46878bd39b85e5c71f66 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:27.965802 23518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c29937a80724ddb9d07a27534a146f3" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:27.976608 23518 raft_consensus.cc:2454] T 0c29937a80724ddb9d07a27534a146f3 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 2.
I20260809 15:46:27.979641 25507 raft_consensus.cc:504] T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:27.980795 23399 raft_consensus.cc:2379] T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 61e69695258f48baa984c9d395e8ce3f in current term 4: Already voted for candidate c0f38c1908414566aa9115bb2cbdd85d in this term.
I20260809 15:46:27.981779 25785 raft_consensus.cc:504] T 16ca412a0dae4feebe85dd01bf1a9173 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:27.982966 23517 raft_consensus.cc:2354] T 16ca412a0dae4feebe85dd01bf1a9173 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 61e69695258f48baa984c9d395e8ce3f for earlier term 2. Current term is 3.
I20260809 15:46:27.983816 23353 leader_election.cc:302] T 0c29937a80724ddb9d07a27534a146f3 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters: 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:27.966459 25807 raft_consensus.cc:482] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:27.966946 25681 raft_consensus.cc:3037] T cef3e7277d7d4760824a3436ed4a847d P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:27.967219 25773 raft_consensus.cc:2781] T d2d4f749fcfd4fb3bfb8115af3471598 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260809 15:46:27.991817 25694 raft_consensus.cc:482] T 0c29937a80724ddb9d07a27534a146f3 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:46:27.993320 25694 raft_consensus.cc:504] T 0c29937a80724ddb9d07a27534a146f3 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:27.995326 25700 raft_consensus.cc:482] T 213543cdbe814c89a17a963469922c1b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:46:27.995857 25700 raft_consensus.cc:504] T 213543cdbe814c89a17a963469922c1b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:27.997577 25483 raft_consensus.cc:504] T 6ae14a74a26f434db08c652e097385c8 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:27.984854 25731 raft_consensus.cc:3037] T d3e94b7e50754a81b3d72a07476cb7b9 P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:46:27.968041 25587 raft_consensus.cc:482] T 4564dfc6d4094c1b9240b367ccecf3b1 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:46:27.985203 25573 leader_election.cc:288] T 0da4d5283c8a46878bd39b85e5c71f66 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:27.993242 25501 raft_consensus.cc:482] T d2d4f749fcfd4fb3bfb8115af3471598 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:46:28.000247 25773 raft_consensus.cc:482] T d2d4f749fcfd4fb3bfb8115af3471598 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:46:27.997577 25700 leader_election.cc:288] T 213543cdbe814c89a17a963469922c1b P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:27.995210 25694 leader_election.cc:288] T 0c29937a80724ddb9d07a27534a146f3 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:28.009037 25507 leader_election.cc:288] T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 4 election: Requested vote from peers 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:28.020387 25507 raft_consensus.cc:482] T 73f6f44c259547f1b05a3a434616fa54 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:28.020959 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16ca412a0dae4feebe85dd01bf1a9173" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f"
I20260809 15:46:28.020823 25507 raft_consensus.cc:504] T 73f6f44c259547f1b05a3a434616fa54 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:28.022264 25776 consensus_queue.cc:237] T 472816994d814cb198b69e5248697440 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:28.022761 25507 leader_election.cc:288] T 73f6f44c259547f1b05a3a434616fa54 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:28.012001 25773 raft_consensus.cc:3037] T d2d4f749fcfd4fb3bfb8115af3471598 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Advancing to term 3
W20260809 15:46:28.045666 23472 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 249502us
I20260809 15:46:28.014760 25785 leader_election.cc:288] T 16ca412a0dae4feebe85dd01bf1a9173 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:28.004657 25587 raft_consensus.cc:504] T 4564dfc6d4094c1b9240b367ccecf3b1 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:28.007114 25807 raft_consensus.cc:504] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:28.012540 25483 leader_election.cc:288] T 6ae14a74a26f434db08c652e097385c8 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:28.050452 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a823d35f4f348d1b52ba9e6d80b00d2" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:28.003283 25637 raft_consensus.cc:348] T 0acc47ce21c3422698d1ef7277c3ed63 P 61e69695258f48baa984c9d395e8ce3f [term 2 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:28.034272 25700 raft_consensus.cc:482] T a3622f6e3e3641cea1960e3a31498bb7 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:28.031255 25789 leader_election.cc:288] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:28.060149 23392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3924b648461e463b9d7925e5671d68ae" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:28.060985 23392 raft_consensus.cc:2379] T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c in current term 3: Already voted for candidate c0f38c1908414566aa9115bb2cbdd85d in this term.
I20260809 15:46:28.060642 25793 raft_consensus.cc:504] T cef3e7277d7d4760824a3436ed4a847d P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:28.062602 25793 leader_election.cc:288] T cef3e7277d7d4760824a3436ed4a847d P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:28.063221 23399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cef3e7277d7d4760824a3436ed4a847d" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:28.063591 23527 raft_consensus.cc:2379] T cef3e7277d7d4760824a3436ed4a847d P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d in current term 3: Already voted for candidate 65c36d164b0540f099d592f38307ee6c in this term.
I20260809 15:46:28.067592 25573 raft_consensus.cc:482] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
I20260809 15:46:28.068130 25573 raft_consensus.cc:504] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:28.070031 25573 leader_election.cc:288] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:28.076400 24471 raft_consensus.cc:2379] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 61e69695258f48baa984c9d395e8ce3f [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c in current term 2: Already voted for candidate 61e69695258f48baa984c9d395e8ce3f in this term.
I20260809 15:46:28.081420 23280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16ca412a0dae4feebe85dd01bf1a9173" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:28.081985 23280 raft_consensus.cc:3037] T 16ca412a0dae4feebe85dd01bf1a9173 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:28.094738 23345 leader_election.cc:302] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c
I20260809 15:46:28.036510 25501 raft_consensus.cc:504] T d2d4f749fcfd4fb3bfb8115af3471598 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:28.098608 23270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cef3e7277d7d4760824a3436ed4a847d" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:28.099220 23270 raft_consensus.cc:3037] T cef3e7277d7d4760824a3436ed4a847d P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:28.103619 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5712f6b6fffc4cc5a8ddfbe63e4097df" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:28.104327 24468 raft_consensus.cc:2454] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 3.
I20260809 15:46:28.104799 25501 leader_election.cc:288] T d2d4f749fcfd4fb3bfb8115af3471598 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:28.111406 23519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "213543cdbe814c89a17a963469922c1b" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:28.098968 23518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0da4d5283c8a46878bd39b85e5c71f66" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:28.112228 23519 raft_consensus.cc:2379] T 213543cdbe814c89a17a963469922c1b P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 in current term 2: Already voted for candidate 65c36d164b0540f099d592f38307ee6c in this term.
I20260809 15:46:28.112576 23518 raft_consensus.cc:2454] T 0da4d5283c8a46878bd39b85e5c71f66 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 2.
I20260809 15:46:28.112890 25831 raft_consensus.cc:2781] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260809 15:46:28.113324 25831 raft_consensus.cc:482] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:46:28.113649 25831 raft_consensus.cc:3037] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:28.114624 23223 leader_election.cc:302] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:46:28.060163 25808 raft_consensus.cc:2781] T 0c29937a80724ddb9d07a27534a146f3 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260809 15:46:28.123540 25477 raft_consensus.cc:482] T 0acc47ce21c3422698d1ef7277c3ed63 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:28.125029 25501 raft_consensus.cc:2781] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260809 15:46:28.095525 25639 raft_consensus.cc:482] T ba56725e42f74a1baa4c04d9660727f9 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:28.099686 25791 raft_consensus.cc:482] T 4bcac4b24e7a4448abdc934bba5a624e P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
I20260809 15:46:28.075096 25700 raft_consensus.cc:504] T a3622f6e3e3641cea1960e3a31498bb7 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:28.099514 25637 raft_consensus.cc:729] T 0acc47ce21c3422698d1ef7277c3ed63 P 61e69695258f48baa984c9d395e8ce3f [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 61e69695258f48baa984c9d395e8ce3f, State: Initialized, Role: LEARNER
I20260809 15:46:28.130371 25808 raft_consensus.cc:482] T 0c29937a80724ddb9d07a27534a146f3 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:46:28.146791 24410 leader_election.cc:398] T 16ca412a0dae4feebe85dd01bf1a9173 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 2 election: Vote denied by peer 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553) with higher term. Message: Invalid argument: T 16ca412a0dae4feebe85dd01bf1a9173 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 61e69695258f48baa984c9d395e8ce3f for earlier term 2. Current term is 3.
I20260809 15:46:28.137498 25501 raft_consensus.cc:482] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
I20260809 15:46:28.147058 25791 raft_consensus.cc:504] T 4bcac4b24e7a4448abdc934bba5a624e P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:28.156971 25700 leader_election.cc:288] T a3622f6e3e3641cea1960e3a31498bb7 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:28.154054 24452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3e94b7e50754a81b3d72a07476cb7b9" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f"
I20260809 15:46:28.147171 23392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998c6cee72cf40228c6dba064c0db2da" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:28.154769 25637 consensus_queue.cc:260] T 0acc47ce21c3422698d1ef7277c3ed63 P 61e69695258f48baa984c9d395e8ce3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } attrs { replace: true } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } }
W20260809 15:46:28.156672 25825 log.cc:924] Time spent T 3924b648461e463b9d7925e5671d68ae P 65c36d164b0540f099d592f38307ee6c: Append to log took a long time: real 0.216s user 0.003s sys 0.000s
I20260809 15:46:28.148429 25650 raft_consensus.cc:482] T a88421dba80d4d3388aa5eb62e23853b P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:28.159925 25700 raft_consensus.cc:482] T 2424fd9bd7f14e508fd106cbfda786a4 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:28.179903 25700 raft_consensus.cc:504] T 2424fd9bd7f14e508fd106cbfda786a4 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:28.166723 25795 raft_consensus.cc:482] T 73f6f44c259547f1b05a3a434616fa54 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:28.181753 25700 leader_election.cc:288] T 2424fd9bd7f14e508fd106cbfda786a4 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:28.182101 25795 raft_consensus.cc:504] T 73f6f44c259547f1b05a3a434616fa54 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:28.157259 25808 raft_consensus.cc:3037] T 0c29937a80724ddb9d07a27534a146f3 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:28.184132 25795 leader_election.cc:288] T 73f6f44c259547f1b05a3a434616fa54 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:28.160864 25587 leader_election.cc:288] T 4564dfc6d4094c1b9240b367ccecf3b1 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
W20260809 15:46:28.161693 24410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 66221.1us
I20260809 15:46:28.166821 23392 raft_consensus.cc:2454] T 998c6cee72cf40228c6dba064c0db2da P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 4.
I20260809 15:46:28.197961 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c29937a80724ddb9d07a27534a146f3" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:28.144855 25675 raft_consensus.cc:482] T 4564dfc6d4094c1b9240b367ccecf3b1 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:28.198769 23526 raft_consensus.cc:2379] T 0c29937a80724ddb9d07a27534a146f3 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 in current term 2: Already voted for candidate 65c36d164b0540f099d592f38307ee6c in this term.
I20260809 15:46:28.146920 25639 raft_consensus.cc:504] T ba56725e42f74a1baa4c04d9660727f9 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:28.199409 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bcac4b24e7a4448abdc934bba5a624e" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:28.200008 25731 raft_consensus.cc:504] T d3e94b7e50754a81b3d72a07476cb7b9 P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:28.200939 25639 leader_election.cc:288] T ba56725e42f74a1baa4c04d9660727f9 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:28.172751 25637 ts_tablet_manager.cc:1428] T 0acc47ce21c3422698d1ef7277c3ed63 P 61e69695258f48baa984c9d395e8ce3f: Time spent starting tablet: real 0.237s user 0.011s sys 0.000s
I20260809 15:46:28.201653 24452 raft_consensus.cc:2379] T d3e94b7e50754a81b3d72a07476cb7b9 P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d in current term 4: Already voted for candidate 61e69695258f48baa984c9d395e8ce3f in this term.
I20260809 15:46:28.201942 25731 leader_election.cc:288] T d3e94b7e50754a81b3d72a07476cb7b9 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 4 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:28.203900 25822 raft_consensus.cc:482] T 0acc47ce21c3422698d1ef7277c3ed63 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
I20260809 15:46:28.159420 25501 raft_consensus.cc:3037] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:28.186043 23270 raft_consensus.cc:2454] T cef3e7277d7d4760824a3436ed4a847d P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 3.
I20260809 15:46:28.188294 23276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73f6f44c259547f1b05a3a434616fa54" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:28.188899 25785 raft_consensus.cc:482] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:28.189839 25807 leader_election.cc:288] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:28.190472 25816 raft_consensus.cc:482] T 538a228ddfb54bc6a36528d5ff07597b P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:28.192040 23273 raft_consensus.cc:1261] T 297c82ed906d4b48b184985eed3de601 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260809 15:46:28.195751 25650 raft_consensus.cc:504] T a88421dba80d4d3388aa5eb62e23853b P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:28.165453 25738 raft_consensus.cc:482] T e14d118b2eaa42aa8a7cd4025aba7128 P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:28.195009 25791 leader_election.cc:288] T 4bcac4b24e7a4448abdc934bba5a624e P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:28.232580 25650 leader_election.cc:288] T a88421dba80d4d3388aa5eb62e23853b P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:28.217365 25773 raft_consensus.cc:504] T d2d4f749fcfd4fb3bfb8115af3471598 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:28.211776 23280 raft_consensus.cc:2454] T 16ca412a0dae4feebe85dd01bf1a9173 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 3.
I20260809 15:46:28.222435 23274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba56725e42f74a1baa4c04d9660727f9" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:28.145874 25477 raft_consensus.cc:504] T 0acc47ce21c3422698d1ef7277c3ed63 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } attrs { replace: true } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } }
I20260809 15:46:28.230664 23276 raft_consensus.cc:2454] T 73f6f44c259547f1b05a3a434616fa54 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 2.
I20260809 15:46:28.230540 25738 raft_consensus.cc:504] T e14d118b2eaa42aa8a7cd4025aba7128 P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:28.231223 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5712f6b6fffc4cc5a8ddfbe63e4097df" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:28.233651 23115 catalog_manager.cc:5495] T 3924b648461e463b9d7925e5671d68ae P 65c36d164b0540f099d592f38307ee6c reported cstate change: term changed from 2 to 3, leader changed from 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) to 65c36d164b0540f099d592f38307ee6c (127.20.202.67). New cstate: current_term: 3 leader_uuid: "65c36d164b0540f099d592f38307ee6c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260809 15:46:28.212245 25822 raft_consensus.cc:504] T 0acc47ce21c3422698d1ef7277c3ed63 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:28.204274 25681 raft_consensus.cc:504] T cef3e7277d7d4760824a3436ed4a847d P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:28.243672 23274 raft_consensus.cc:2454] T ba56725e42f74a1baa4c04d9660727f9 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 2.
I20260809 15:46:28.245759 23521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2d4f749fcfd4fb3bfb8115af3471598" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:28.259666 23521 raft_consensus.cc:2379] T d2d4f749fcfd4fb3bfb8115af3471598 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 in current term 2: Already voted for candidate 65c36d164b0540f099d592f38307ee6c in this term.
I20260809 15:46:28.220464 25639 raft_consensus.cc:482] T a3622f6e3e3641cea1960e3a31498bb7 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:28.260098 23395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c29937a80724ddb9d07a27534a146f3" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:28.198786 25675 raft_consensus.cc:504] T 4564dfc6d4094c1b9240b367ccecf3b1 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:28.231283 25470 raft_consensus.cc:482] T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:28.262096 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0da4d5283c8a46878bd39b85e5c71f66" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:28.262987 23402 raft_consensus.cc:2454] T 0da4d5283c8a46878bd39b85e5c71f66 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 2.
I20260809 15:46:28.264106 23521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3e94b7e50754a81b3d72a07476cb7b9" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:28.264653 23521 raft_consensus.cc:3037] T d3e94b7e50754a81b3d72a07476cb7b9 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:46:28.268780 23400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "213543cdbe814c89a17a963469922c1b" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:28.269387 23400 raft_consensus.cc:2454] T 213543cdbe814c89a17a963469922c1b P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 2.
I20260809 15:46:28.273164 23470 leader_election.cc:302] T cef3e7277d7d4760824a3436ed4a847d P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:46:28.225485 25785 raft_consensus.cc:504] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:28.283113 23274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ae14a74a26f434db08c652e097385c8" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:28.291973 23273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73f6f44c259547f1b05a3a434616fa54" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:28.306885 23274 raft_consensus.cc:3037] T 6ae14a74a26f434db08c652e097385c8 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:28.284132 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2d4f749fcfd4fb3bfb8115af3471598" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:28.263587 25791 raft_consensus.cc:482] T df20671a7b874dbb98c0ef28371c1308 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:28.309003 23470 leader_election.cc:302] T 16ca412a0dae4feebe85dd01bf1a9173 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 61e69695258f48baa984c9d395e8ce3f
I20260809 15:46:28.309511 23402 raft_consensus.cc:2354] T d2d4f749fcfd4fb3bfb8115af3471598 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 2. Current term is 3.
I20260809 15:46:28.267706 23280 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "28e17dcb5efe45b8b6a8c42da03b4819"
dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
mode: GRACEFUL
new_leader_uuid: "65c36d164b0540f099d592f38307ee6c"
from {username='slave'} at 127.0.0.1:33468
I20260809 15:46:28.314669 23280 raft_consensus.cc:595] T 28e17dcb5efe45b8b6a8c42da03b4819 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 LEADER]: Received request to transfer leadership to TS 65c36d164b0540f099d592f38307ee6c
I20260809 15:46:28.315886 23526 raft_consensus.cc:2379] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 in current term 3: Already voted for candidate 65c36d164b0540f099d592f38307ee6c in this term.
I20260809 15:46:28.244282 25831 raft_consensus.cc:504] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:28.315729 23399 raft_consensus.cc:2379] T cef3e7277d7d4760824a3436ed4a847d P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c in current term 3: Already voted for candidate c0f38c1908414566aa9115bb2cbdd85d in this term.
I20260809 15:46:28.281275 25776 raft_consensus.cc:482] T 2424fd9bd7f14e508fd106cbfda786a4 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:28.306496 23273 raft_consensus.cc:2454] T 73f6f44c259547f1b05a3a434616fa54 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 2.
I20260809 15:46:28.324187 23397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad8a3b25d7b49e5a2142d5709edd0c7" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:28.327559 25685 raft_consensus.cc:482] T 294d4bcc1e7a477cb56455581277a60f P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:28.327893 23400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a823d35f4f348d1b52ba9e6d80b00d2" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:28.328073 25685 raft_consensus.cc:504] T 294d4bcc1e7a477cb56455581277a60f P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:28.329835 23285 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6ae14a74a26f434db08c652e097385c8"
dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
mode: GRACEFUL
new_leader_uuid: "65c36d164b0540f099d592f38307ee6c"
from {username='slave'} at 127.0.0.1:33468
I20260809 15:46:28.330029 25685 leader_election.cc:288] T 294d4bcc1e7a477cb56455581277a60f P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:28.328485 23400 raft_consensus.cc:2454] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 2.
I20260809 15:46:28.238494 25816 raft_consensus.cc:504] T 538a228ddfb54bc6a36528d5ff07597b P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:28.331537 25773 leader_election.cc:288] T d2d4f749fcfd4fb3bfb8115af3471598 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:28.332765 25816 leader_election.cc:288] T 538a228ddfb54bc6a36528d5ff07597b P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:28.333346 25675 leader_election.cc:288] T 4564dfc6d4094c1b9240b367ccecf3b1 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:28.313369 25587 raft_consensus.cc:482] T 294d4bcc1e7a477cb56455581277a60f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
I20260809 15:46:28.335654 25587 raft_consensus.cc:504] T 294d4bcc1e7a477cb56455581277a60f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:28.338070 25587 leader_election.cc:288] T 294d4bcc1e7a477cb56455581277a60f P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:28.340627 25738 leader_election.cc:288] T e14d118b2eaa42aa8a7cd4025aba7128 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:28.343770 25831 leader_election.cc:288] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 election: Requested vote from peers 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:28.344784 25477 leader_election.cc:288] T 0acc47ce21c3422698d1ef7277c3ed63 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:28.345815 23397 raft_consensus.cc:2354] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c for earlier term 2. Current term is 3.
I20260809 15:46:28.346714 25808 raft_consensus.cc:504] T 0c29937a80724ddb9d07a27534a146f3 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:28.348706 25808 leader_election.cc:288] T 0c29937a80724ddb9d07a27534a146f3 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:28.351161 25681 leader_election.cc:288] T cef3e7277d7d4760824a3436ed4a847d P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:28.351765 23395 raft_consensus.cc:2354] T 0c29937a80724ddb9d07a27534a146f3 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 2. Current term is 3.
I20260809 15:46:28.317183 23276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a88421dba80d4d3388aa5eb62e23853b" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:28.352938 23276 raft_consensus.cc:2454] T a88421dba80d4d3388aa5eb62e23853b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 2.
I20260809 15:46:28.354271 25793 consensus_queue.cc:1035] T 297c82ed906d4b48b184985eed3de601 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 15:46:28.355677 25793 raft_consensus.cc:2781] T 16ca412a0dae4feebe85dd01bf1a9173 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Leader election won for term 3
W20260809 15:46:28.339017 23353 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 355250us
I20260809 15:46:28.282342 25639 raft_consensus.cc:504] T a3622f6e3e3641cea1960e3a31498bb7 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:28.361065 23529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "294d4bcc1e7a477cb56455581277a60f" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:28.356184 25793 raft_consensus.cc:686] T 16ca412a0dae4feebe85dd01bf1a9173 P 65c36d164b0540f099d592f38307ee6c [term 3 LEADER]: Becoming Leader. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:46:28.304289 25772 raft_consensus.cc:504] T 998c6cee72cf40228c6dba064c0db2da P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:28.280514 25470 raft_consensus.cc:504] T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:28.333375 25816 raft_consensus.cc:2781] T cef3e7277d7d4760824a3436ed4a847d P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Leader election won for term 3
I20260809 15:46:28.355638 25785 leader_election.cc:288] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:28.380359 23529 raft_consensus.cc:2454] T 294d4bcc1e7a477cb56455581277a60f P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 3.
I20260809 15:46:28.383049 23472 leader_election.cc:302] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters: 61e69695258f48baa984c9d395e8ce3f
I20260809 15:46:28.321646 25791 raft_consensus.cc:504] T df20671a7b874dbb98c0ef28371c1308 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:28.386159 23397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cef3e7277d7d4760824a3436ed4a847d" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:28.387862 23472 leader_election.cc:398] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 2 pre-election: Vote denied by peer c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263) with higher term. Message: Invalid argument: T 5ad8a3b25d7b49e5a2142d5709edd0c7 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c for earlier term 2. Current term is 3.
I20260809 15:46:28.326825 25776 raft_consensus.cc:504] T 2424fd9bd7f14e508fd106cbfda786a4 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:28.388522 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ae14a74a26f434db08c652e097385c8" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:28.548570 23526 raft_consensus.cc:3037] T 6ae14a74a26f434db08c652e097385c8 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:28.386770 23519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73f6f44c259547f1b05a3a434616fa54" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
W20260809 15:46:28.549126 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 235732us
I20260809 15:46:28.389307 23517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e14d118b2eaa42aa8a7cd4025aba7128" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:28.478763 23392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73f6f44c259547f1b05a3a434616fa54" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:28.441977 24452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "294d4bcc1e7a477cb56455581277a60f" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:28.386678 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5712f6b6fffc4cc5a8ddfbe63e4097df" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:28.523136 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba56725e42f74a1baa4c04d9660727f9" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
W20260809 15:46:28.543371 24410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 101334us
I20260809 15:46:28.387967 23397 raft_consensus.cc:2379] T cef3e7277d7d4760824a3436ed4a847d P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c in current term 3: Already voted for candidate c0f38c1908414566aa9115bb2cbdd85d in this term.
I20260809 15:46:28.545485 23472 leader_election.cc:401] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20260809 15:46:28.551653 23280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4564dfc6d4094c1b9240b367ccecf3b1" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:28.549527 23519 raft_consensus.cc:2454] T 73f6f44c259547f1b05a3a434616fa54 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 2.
I20260809 15:46:28.555531 23529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a823d35f4f348d1b52ba9e6d80b00d2" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:28.488273 23518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3e94b7e50754a81b3d72a07476cb7b9" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:28.449579 23395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3e94b7e50754a81b3d72a07476cb7b9" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:28.558040 23400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bcac4b24e7a4448abdc934bba5a624e" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
W20260809 15:46:28.545192 25843 log.cc:924] Time spent T 472816994d814cb198b69e5248697440 P c0f38c1908414566aa9115bb2cbdd85d: Append to log took a long time: real 0.056s user 0.000s sys 0.001s
I20260809 15:46:28.563804 23273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2d4f749fcfd4fb3bfb8115af3471598" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:28.570092 23273 raft_consensus.cc:3037] T d2d4f749fcfd4fb3bfb8115af3471598 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Advancing to term 3
I20260809 15:46:28.564884 23402 raft_consensus.cc:2454] T ba56725e42f74a1baa4c04d9660727f9 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 2.
I20260809 15:46:28.559162 23280 raft_consensus.cc:2454] T 4564dfc6d4094c1b9240b367ccecf3b1 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 1.
I20260809 15:46:28.566139 23517 raft_consensus.cc:2454] T e14d118b2eaa42aa8a7cd4025aba7128 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 3.
I20260809 15:46:28.544580 25851 raft_consensus.cc:482] T 294d4bcc1e7a477cb56455581277a60f P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:28.556742 23274 raft_consensus.cc:2454] T 6ae14a74a26f434db08c652e097385c8 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 3.
I20260809 15:46:28.573982 23285 raft_consensus.cc:595] T 6ae14a74a26f434db08c652e097385c8 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Received request to transfer leadership to TS 65c36d164b0540f099d592f38307ee6c
I20260809 15:46:28.574340 23285 raft_consensus.cc:603] T 6ae14a74a26f434db08c652e097385c8 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260809 15:46:28.567191 23270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cef3e7277d7d4760824a3436ed4a847d" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:28.562980 23529 raft_consensus.cc:2454] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 1.
I20260809 15:46:28.562454 24410 leader_election.cc:302] T 294d4bcc1e7a477cb56455581277a60f P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c; no voters:
I20260809 15:46:28.575238 23270 raft_consensus.cc:2379] T cef3e7277d7d4760824a3436ed4a847d P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d in current term 3: Already voted for candidate 65c36d164b0540f099d592f38307ee6c in this term.
W20260809 15:46:28.580528 23472 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 194882us
I20260809 15:46:28.549203 25695 raft_consensus.cc:482] T 538a228ddfb54bc6a36528d5ff07597b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:28.563107 23394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a88421dba80d4d3388aa5eb62e23853b" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:28.554490 25816 raft_consensus.cc:686] T cef3e7277d7d4760824a3436ed4a847d P 65c36d164b0540f099d592f38307ee6c [term 3 LEADER]: Becoming Leader. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
W20260809 15:46:28.674759 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 232226us
I20260809 15:46:28.733289 25849 raft_consensus.cc:2781] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260809 15:46:28.739423 25695 raft_consensus.cc:504] T 538a228ddfb54bc6a36528d5ff07597b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:28.781711 25857 raft_consensus.cc:482] T df20671a7b874dbb98c0ef28371c1308 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:28.781888 25840 raft_consensus.cc:482] T a3622f6e3e3641cea1960e3a31498bb7 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:28.628310 25851 raft_consensus.cc:504] T 294d4bcc1e7a477cb56455581277a60f P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:28.754508 23396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "538a228ddfb54bc6a36528d5ff07597b" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:28.703227 23395 raft_consensus.cc:2379] T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 61e69695258f48baa984c9d395e8ce3f in current term 4: Already voted for candidate c0f38c1908414566aa9115bb2cbdd85d in this term.
I20260809 15:46:28.564262 23392 raft_consensus.cc:2454] T 73f6f44c259547f1b05a3a434616fa54 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 2.
I20260809 15:46:28.559283 25793 consensus_queue.cc:237] T 16ca412a0dae4feebe85dd01bf1a9173 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:28.561065 25785 raft_consensus.cc:482] T 0acc47ce21c3422698d1ef7277c3ed63 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:28.583235 25675 raft_consensus.cc:482] T 8e4ffcf83c234ae7910ddef3ec0b4bda P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
I20260809 15:46:28.564532 24452 raft_consensus.cc:2454] T 294d4bcc1e7a477cb56455581277a60f P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 3.
I20260809 15:46:28.570286 24468 raft_consensus.cc:2454] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 3.
W20260809 15:46:28.776079 23470 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 470899us
I20260809 15:46:28.862411 23470 leader_election.cc:302] T ba56725e42f74a1baa4c04d9660727f9 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:46:28.777408 23397 raft_consensus.cc:1261] T 297c82ed906d4b48b184985eed3de601 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260809 15:46:28.779742 23351 leader_election.cc:302] T 73f6f44c259547f1b05a3a434616fa54 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters:
W20260809 15:46:28.863792 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 84250.3us
W20260809 15:46:28.864464 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 87655.4us
I20260809 15:46:28.864886 23470 leader_election.cc:302] T 73f6f44c259547f1b05a3a434616fa54 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:46:28.781025 25685 raft_consensus.cc:482] T 7660dfa2ef694927b8eda4100a2bcd9b P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:28.869150 23351 leader_election.cc:302] T 4564dfc6d4094c1b9240b367ccecf3b1 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:46:28.777407 23394 raft_consensus.cc:2454] T a88421dba80d4d3388aa5eb62e23853b P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 2.
W20260809 15:46:28.790369 25501 consensus_meta.cc:287] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent flushing consensus metadata: real 0.569s user 0.001s sys 0.014s
I20260809 15:46:28.791644 23273 raft_consensus.cc:2454] T d2d4f749fcfd4fb3bfb8115af3471598 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 3.
I20260809 15:46:28.791831 23526 raft_consensus.cc:2454] T 6ae14a74a26f434db08c652e097385c8 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 3.
I20260809 15:46:28.792111 25857 raft_consensus.cc:504] T df20671a7b874dbb98c0ef28371c1308 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:46:28.791725 23521 consensus_meta.cc:287] T d3e94b7e50754a81b3d72a07476cb7b9 P 65c36d164b0540f099d592f38307ee6c: Time spent flushing consensus metadata: real 0.524s user 0.007s sys 0.001s
I20260809 15:46:28.879293 23521 raft_consensus.cc:2454] T d3e94b7e50754a81b3d72a07476cb7b9 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 4.
I20260809 15:46:28.832077 25695 leader_election.cc:288] T 538a228ddfb54bc6a36528d5ff07597b P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:28.793313 23396 raft_consensus.cc:2454] T 538a228ddfb54bc6a36528d5ff07597b P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 3.
I20260809 15:46:28.833071 25816 consensus_queue.cc:237] T cef3e7277d7d4760824a3436ed4a847d P 65c36d164b0540f099d592f38307ee6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:28.861742 25650 raft_consensus.cc:3037] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Advancing to term 3
I20260809 15:46:28.838078 25822 leader_election.cc:288] T 0acc47ce21c3422698d1ef7277c3ed63 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:28.831156 25851 leader_election.cc:288] T 294d4bcc1e7a477cb56455581277a60f P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:28.834645 25776 leader_election.cc:288] T 2424fd9bd7f14e508fd106cbfda786a4 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:28.827724 25573 raft_consensus.cc:482] T df20671a7b874dbb98c0ef28371c1308 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:28.793211 25840 raft_consensus.cc:504] T a3622f6e3e3641cea1960e3a31498bb7 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:28.927047 24448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad8a3b25d7b49e5a2142d5709edd0c7" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f"
I20260809 15:46:28.927717 25840 leader_election.cc:288] T a3622f6e3e3641cea1960e3a31498bb7 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:28.828311 25639 leader_election.cc:288] T a3622f6e3e3641cea1960e3a31498bb7 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:28.928227 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a823d35f4f348d1b52ba9e6d80b00d2" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:28.873328 23351 leader_election.cc:302] T 6ae14a74a26f434db08c652e097385c8 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters:
W20260809 15:46:28.875861 23115 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.595s user 0.022s sys 0.067s
I20260809 15:46:28.927728 24448 raft_consensus.cc:3037] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 61e69695258f48baa984c9d395e8ce3f [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:28.865412 25746 raft_consensus.cc:2781] T 294d4bcc1e7a477cb56455581277a60f P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260809 15:46:28.931126 25746 raft_consensus.cc:482] T 294d4bcc1e7a477cb56455581277a60f P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:46:28.931530 25746 raft_consensus.cc:3037] T 294d4bcc1e7a477cb56455581277a60f P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:46:28.875169 25831 raft_consensus.cc:2781] T 73f6f44c259547f1b05a3a434616fa54 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260809 15:46:28.937698 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 64446.7us
I20260809 15:46:28.867700 25772 leader_election.cc:288] T 998c6cee72cf40228c6dba064c0db2da P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 5 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:28.928830 24471 raft_consensus.cc:2454] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 61e69695258f48baa984c9d395e8ce3f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 2.
W20260809 15:46:28.929469 23115 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.649s user 0.022s sys 0.067s
I20260809 15:46:28.935128 23521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2d4f749fcfd4fb3bfb8115af3471598" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:28.938663 23351 leader_election.cc:302] T d2d4f749fcfd4fb3bfb8115af3471598 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:46:28.920399 25795 raft_consensus.cc:482] T 4564dfc6d4094c1b9240b367ccecf3b1 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:28.941150 23521 raft_consensus.cc:3037] T d2d4f749fcfd4fb3bfb8115af3471598 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:28.943933 23345 leader_election.cc:302] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:46:28.928978 25639 raft_consensus.cc:2781] T ba56725e42f74a1baa4c04d9660727f9 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260809 15:46:28.877503 25675 raft_consensus.cc:504] T 8e4ffcf83c234ae7910ddef3ec0b4bda P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:28.934754 25822 raft_consensus.cc:2781] T 4564dfc6d4094c1b9240b367ccecf3b1 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260809 15:46:28.879341 25501 raft_consensus.cc:504] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:28.881242 25685 raft_consensus.cc:504] T 7660dfa2ef694927b8eda4100a2bcd9b P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:28.883754 25785 raft_consensus.cc:504] T 0acc47ce21c3422698d1ef7277c3ed63 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:28.940276 25831 raft_consensus.cc:482] T 73f6f44c259547f1b05a3a434616fa54 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:46:28.973658 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3924b648461e463b9d7925e5671d68ae" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:28.974929 23273 raft_consensus.cc:1261] T d0721a4403a440f383318a3cea9d1442 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260809 15:46:28.942020 25773 raft_consensus.cc:2781] T 6ae14a74a26f434db08c652e097385c8 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader election won for term 3
I20260809 15:46:28.940264 25470 leader_election.cc:288] T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:28.953688 25776 raft_consensus.cc:2781] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260809 15:46:28.955904 25483 raft_consensus.cc:2781] T d2d4f749fcfd4fb3bfb8115af3471598 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader election won for term 3
I20260809 15:46:28.989293 23223 leader_election.cc:302] T 294d4bcc1e7a477cb56455581277a60f P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:46:28.995365 23273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2424fd9bd7f14e508fd106cbfda786a4" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:28.996910 23273 raft_consensus.cc:2454] T 2424fd9bd7f14e508fd106cbfda786a4 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 3.
I20260809 15:46:28.985183 24471 raft_consensus.cc:2454] T 3924b648461e463b9d7925e5671d68ae P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 3.
I20260809 15:46:28.936918 25573 raft_consensus.cc:504] T df20671a7b874dbb98c0ef28371c1308 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:28.975529 25822 raft_consensus.cc:482] T 4564dfc6d4094c1b9240b367ccecf3b1 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:46:28.953756 25639 raft_consensus.cc:482] T ba56725e42f74a1baa4c04d9660727f9 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:46:28.992808 25831 raft_consensus.cc:3037] T 73f6f44c259547f1b05a3a434616fa54 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:29.006026 23273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0acc47ce21c3422698d1ef7277c3ed63" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:29.006897 23273 raft_consensus.cc:2396] T 0acc47ce21c3422698d1ef7277c3ed63 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d for term 2 because replica has last-logged OpId of term: 1 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260809 15:46:28.991330 25776 raft_consensus.cc:482] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:46:28.957845 25795 raft_consensus.cc:504] T 4564dfc6d4094c1b9240b367ccecf3b1 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:28.973244 25791 leader_election.cc:288] T df20671a7b874dbb98c0ef28371c1308 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:29.002445 25822 raft_consensus.cc:3037] T 4564dfc6d4094c1b9240b367ccecf3b1 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:29.003377 25483 raft_consensus.cc:686] T d2d4f749fcfd4fb3bfb8115af3471598 P c0f38c1908414566aa9115bb2cbdd85d [term 3 LEADER]: Becoming Leader. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:46:29.020869 23521 raft_consensus.cc:2454] T d2d4f749fcfd4fb3bfb8115af3471598 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 3.
I20260809 15:46:29.004699 25639 raft_consensus.cc:3037] T ba56725e42f74a1baa4c04d9660727f9 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:29.011633 25532 raft_consensus.cc:2781] T 294d4bcc1e7a477cb56455581277a60f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260809 15:46:29.003767 25773 raft_consensus.cc:686] T 6ae14a74a26f434db08c652e097385c8 P c0f38c1908414566aa9115bb2cbdd85d [term 3 LEADER]: Becoming Leader. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
W20260809 15:46:29.026196 23470 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 161384us
I20260809 15:46:29.032089 23470 leader_election.cc:302] T a88421dba80d4d3388aa5eb62e23853b P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:46:29.016491 25776 raft_consensus.cc:3037] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Advancing to term 3
W20260809 15:46:29.020426 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 141598us
I20260809 15:46:29.016266 23520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4564dfc6d4094c1b9240b367ccecf3b1" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:29.041451 23517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad8a3b25d7b49e5a2142d5709edd0c7" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:29.056991 24448 raft_consensus.cc:2454] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 3.
I20260809 15:46:29.059147 25694 raft_consensus.cc:482] T e14d118b2eaa42aa8a7cd4025aba7128 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:29.059690 25694 raft_consensus.cc:504] T e14d118b2eaa42aa8a7cd4025aba7128 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:29.061707 25859 raft_consensus.cc:2781] T 73f6f44c259547f1b05a3a434616fa54 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260809 15:46:29.062157 25859 raft_consensus.cc:482] T 73f6f44c259547f1b05a3a434616fa54 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:46:29.062469 25859 raft_consensus.cc:3037] T 73f6f44c259547f1b05a3a434616fa54 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:29.066241 25694 leader_election.cc:288] T e14d118b2eaa42aa8a7cd4025aba7128 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:29.072420 23273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3622f6e3e3641cea1960e3a31498bb7" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:29.076390 23273 raft_consensus.cc:2454] T a3622f6e3e3641cea1960e3a31498bb7 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 2.
I20260809 15:46:29.035480 25532 raft_consensus.cc:482] T 294d4bcc1e7a477cb56455581277a60f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
I20260809 15:46:29.088586 23472 leader_election.cc:302] T 4bcac4b24e7a4448abdc934bba5a624e P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:29.054248 25791 raft_consensus.cc:2781] T a88421dba80d4d3388aa5eb62e23853b P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260809 15:46:29.093488 23345 leader_election.cc:302] T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:46:29.059557 25675 leader_election.cc:288] T 8e4ffcf83c234ae7910ddef3ec0b4bda P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:29.096203 23526 raft_consensus.cc:1226] T 16ca412a0dae4feebe85dd01bf1a9173 P 65c36d164b0540f099d592f38307ee6c [term 3 LEADER]: Rejecting Update request from peer 61e69695258f48baa984c9d395e8ce3f for earlier term 2. Current term is 3. Ops: []
I20260809 15:46:29.053712 25862 raft_consensus.cc:482] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:29.100139 23345 leader_election.cc:302] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:46:29.100908 23270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5712f6b6fffc4cc5a8ddfbe63e4097df" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:29.101843 23273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "538a228ddfb54bc6a36528d5ff07597b" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:29.090452 25532 raft_consensus.cc:3037] T 294d4bcc1e7a477cb56455581277a60f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:29.051970 23520 raft_consensus.cc:2454] T 4564dfc6d4094c1b9240b367ccecf3b1 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 2.
I20260809 15:46:29.096930 23521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c29937a80724ddb9d07a27534a146f3" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:29.091235 25652 raft_consensus.cc:482] T 2424fd9bd7f14e508fd106cbfda786a4 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:29.107944 23351 leader_election.cc:302] T 2424fd9bd7f14e508fd106cbfda786a4 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:46:29.108637 23280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998c6cee72cf40228c6dba064c0db2da" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:29.115311 23280 raft_consensus.cc:3037] T 998c6cee72cf40228c6dba064c0db2da P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Advancing to term 5
I20260809 15:46:29.108757 23273 raft_consensus.cc:2454] T 538a228ddfb54bc6a36528d5ff07597b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 3.
I20260809 15:46:29.109210 23270 raft_consensus.cc:2454] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 3.
I20260809 15:46:29.109863 23472 leader_election.cc:302] T 538a228ddfb54bc6a36528d5ff07597b P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:46:29.098538 24448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e14d118b2eaa42aa8a7cd4025aba7128" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:29.119902 24448 raft_consensus.cc:2454] T e14d118b2eaa42aa8a7cd4025aba7128 P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 3.
I20260809 15:46:29.116729 25863 raft_consensus.cc:2781] T 538a228ddfb54bc6a36528d5ff07597b P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260809 15:46:29.111699 25840 raft_consensus.cc:2781] T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260809 15:46:29.120311 23277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "294d4bcc1e7a477cb56455581277a60f" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:29.120736 25840 raft_consensus.cc:482] T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:46:29.121104 25840 raft_consensus.cc:3037] T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:46:29.121196 23396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3622f6e3e3641cea1960e3a31498bb7" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:29.121788 23396 raft_consensus.cc:2454] T a3622f6e3e3641cea1960e3a31498bb7 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 2.
I20260809 15:46:29.112931 23521 raft_consensus.cc:3037] T 0c29937a80724ddb9d07a27534a146f3 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:29.094704 25791 raft_consensus.cc:482] T a88421dba80d4d3388aa5eb62e23853b P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:46:29.123716 25864 consensus_queue.cc:1035] T d0721a4403a440f383318a3cea9d1442 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.009s
I20260809 15:46:29.124091 25791 raft_consensus.cc:3037] T a88421dba80d4d3388aa5eb62e23853b P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Advancing to term 3
W20260809 15:46:29.125484 24410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 108924us
I20260809 15:46:29.125525 24471 raft_consensus.cc:1261] T 472816994d814cb198b69e5248697440 P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260809 15:46:29.112157 23282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c29937a80724ddb9d07a27534a146f3" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:29.127949 23282 raft_consensus.cc:3037] T 0c29937a80724ddb9d07a27534a146f3 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Advancing to term 3
I20260809 15:46:29.129449 23223 leader_election.cc:302] T e14d118b2eaa42aa8a7cd4025aba7128 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:46:29.112797 25483 consensus_queue.cc:237] T d2d4f749fcfd4fb3bfb8115af3471598 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:29.141703 24410 leader_election.cc:302] T e14d118b2eaa42aa8a7cd4025aba7128 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c; no voters:
I20260809 15:46:29.142232 25868 raft_consensus.cc:2726] T 4bcac4b24e7a4448abdc934bba5a624e P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 15:46:29.125205 25685 leader_election.cc:288] T 7660dfa2ef694927b8eda4100a2bcd9b P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:29.142566 25731 raft_consensus.cc:2781] T e14d118b2eaa42aa8a7cd4025aba7128 P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260809 15:46:29.125627 25863 raft_consensus.cc:482] T 538a228ddfb54bc6a36528d5ff07597b P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:46:29.113256 25675 raft_consensus.cc:2781] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader election won for term 3
I20260809 15:46:29.143077 25731 raft_consensus.cc:482] T e14d118b2eaa42aa8a7cd4025aba7128 P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:46:29.143517 24410 leader_election.cc:302] T d3e94b7e50754a81b3d72a07476cb7b9 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c; no voters: c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:29.160082 25731 raft_consensus.cc:3037] T e14d118b2eaa42aa8a7cd4025aba7128 P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:46:29.162045 25865 raft_consensus.cc:2781] T d3e94b7e50754a81b3d72a07476cb7b9 P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Leader election won for term 4
I20260809 15:46:29.162530 25865 raft_consensus.cc:686] T d3e94b7e50754a81b3d72a07476cb7b9 P 61e69695258f48baa984c9d395e8ce3f [term 4 LEADER]: Becoming Leader. State: Replica: 61e69695258f48baa984c9d395e8ce3f, State: Running, Role: LEADER
I20260809 15:46:29.155949 23283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3622f6e3e3641cea1960e3a31498bb7" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:29.156770 25861 raft_consensus.cc:482] T 7660dfa2ef694927b8eda4100a2bcd9b P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
I20260809 15:46:29.163403 25865 consensus_queue.cc:237] T d3e94b7e50754a81b3d72a07476cb7b9 P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:29.163843 25861 raft_consensus.cc:504] T 7660dfa2ef694927b8eda4100a2bcd9b P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:29.156023 25863 raft_consensus.cc:3037] T 538a228ddfb54bc6a36528d5ff07597b P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:46:29.163811 23283 raft_consensus.cc:2454] T a3622f6e3e3641cea1960e3a31498bb7 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 2.
I20260809 15:46:29.158360 25795 leader_election.cc:288] T 4564dfc6d4094c1b9240b367ccecf3b1 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:29.168081 25675 raft_consensus.cc:686] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P c0f38c1908414566aa9115bb2cbdd85d [term 3 LEADER]: Becoming Leader. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:46:29.168550 23396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998c6cee72cf40228c6dba064c0db2da" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:29.169080 25785 leader_election.cc:288] T 0acc47ce21c3422698d1ef7277c3ed63 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:29.166975 23397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7660dfa2ef694927b8eda4100a2bcd9b" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:29.170311 23397 raft_consensus.cc:2454] T 7660dfa2ef694927b8eda4100a2bcd9b P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 3.
I20260809 15:46:29.173833 25652 raft_consensus.cc:504] T 2424fd9bd7f14e508fd106cbfda786a4 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:29.125689 24452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e4ffcf83c234ae7910ddef3ec0b4bda" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:29.117852 25501 leader_election.cc:288] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 3 election: Requested vote from peers 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:29.177778 24410 leader_election.cc:302] T 7660dfa2ef694927b8eda4100a2bcd9b P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:46:29.169106 23396 raft_consensus.cc:3037] T 998c6cee72cf40228c6dba064c0db2da P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Advancing to term 5
I20260809 15:46:29.142446 25873 raft_consensus.cc:2781] T e14d118b2eaa42aa8a7cd4025aba7128 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260809 15:46:29.190717 25478 raft_consensus.cc:2781] T 7660dfa2ef694927b8eda4100a2bcd9b P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260809 15:46:29.176283 23518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cef3e7277d7d4760824a3436ed4a847d" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:29.118686 25857 leader_election.cc:288] T df20671a7b874dbb98c0ef28371c1308 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
W20260809 15:46:29.159051 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 124648us
I20260809 15:46:29.206724 23228 leader_election.cc:302] T 213543cdbe814c89a17a963469922c1b P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:46:29.204489 23353 leader_election.cc:302] T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c
I20260809 15:46:29.211208 23472 leader_election.cc:302] T a3622f6e3e3641cea1960e3a31498bb7 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:46:29.201656 25873 raft_consensus.cc:482] T e14d118b2eaa42aa8a7cd4025aba7128 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:46:29.212337 25873 raft_consensus.cc:3037] T e14d118b2eaa42aa8a7cd4025aba7128 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:46:29.213424 23228 leader_election.cc:302] T 0da4d5283c8a46878bd39b85e5c71f66 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:46:29.116122 25793 consensus_queue.cc:1035] T 297c82ed906d4b48b184985eed3de601 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.085s
I20260809 15:46:29.216229 23468 leader_election.cc:302] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c; no voters:
W20260809 15:46:29.221275 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 113408us
I20260809 15:46:29.221730 23529 raft_consensus.cc:1261] T d0721a4403a440f383318a3cea9d1442 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260809 15:46:29.224531 23283 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a3622f6e3e3641cea1960e3a31498bb7"
dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
mode: GRACEFUL
new_leader_uuid: "65c36d164b0540f099d592f38307ee6c"
from {username='slave'} at 127.0.0.1:33468
I20260809 15:46:29.225059 23283 raft_consensus.cc:595] T a3622f6e3e3641cea1960e3a31498bb7 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Received request to transfer leadership to TS 65c36d164b0540f099d592f38307ee6c
I20260809 15:46:29.225378 23283 raft_consensus.cc:603] T a3622f6e3e3641cea1960e3a31498bb7 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260809 15:46:29.201937 25478 raft_consensus.cc:482] T 7660dfa2ef694927b8eda4100a2bcd9b P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:46:29.233739 23397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df20671a7b874dbb98c0ef28371c1308" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:29.234339 23397 raft_consensus.cc:2454] T df20671a7b874dbb98c0ef28371c1308 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 2.
I20260809 15:46:29.234620 23280 raft_consensus.cc:2454] T 998c6cee72cf40228c6dba064c0db2da P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 5.
I20260809 15:46:29.241041 23228 leader_election.cc:398] T d2d4f749fcfd4fb3bfb8115af3471598 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 2 pre-election: Vote denied by peer c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263) with higher term. Message: Invalid argument: T d2d4f749fcfd4fb3bfb8115af3471598 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 2. Current term is 3.
I20260809 15:46:29.241364 23228 leader_election.cc:401] T d2d4f749fcfd4fb3bfb8115af3471598 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20260809 15:46:29.242702 23529 raft_consensus.cc:1261] T 472816994d814cb198b69e5248697440 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260809 15:46:29.212355 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "294d4bcc1e7a477cb56455581277a60f" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:29.220752 25501 raft_consensus.cc:2781] T 213543cdbe814c89a17a963469922c1b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260809 15:46:29.244300 23285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df20671a7b874dbb98c0ef28371c1308" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:29.280411 24448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5712f6b6fffc4cc5a8ddfbe63e4097df" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f"
I20260809 15:46:29.127242 25773 consensus_queue.cc:237] T 6ae14a74a26f434db08c652e097385c8 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:29.220861 25857 raft_consensus.cc:2726] T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260809 15:46:29.284289 23519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0acc47ce21c3422698d1ef7277c3ed63" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:29.282595 23285 raft_consensus.cc:2454] T df20671a7b874dbb98c0ef28371c1308 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 2.
I20260809 15:46:29.220662 25573 leader_election.cc:288] T df20671a7b874dbb98c0ef28371c1308 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:29.111081 25862 raft_consensus.cc:504] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:29.286547 23283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0acc47ce21c3422698d1ef7277c3ed63" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
W20260809 15:46:29.282594 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 58064.2us
I20260809 15:46:29.291986 23521 raft_consensus.cc:2454] T 0c29937a80724ddb9d07a27534a146f3 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 3.
I20260809 15:46:29.292456 25852 raft_consensus.cc:482] T 213543cdbe814c89a17a963469922c1b P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:29.292770 25875 raft_consensus.cc:2781] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260809 15:46:29.294450 23519 raft_consensus.cc:2379] T 0acc47ce21c3422698d1ef7277c3ed63 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d in current term 2: Already voted for candidate 65c36d164b0540f099d592f38307ee6c in this term.
I20260809 15:46:29.235328 25478 raft_consensus.cc:3037] T 7660dfa2ef694927b8eda4100a2bcd9b P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:46:29.295046 23285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4564dfc6d4094c1b9240b367ccecf3b1" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:29.243932 25868 raft_consensus.cc:2781] T a3622f6e3e3641cea1960e3a31498bb7 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260809 15:46:29.297015 23283 raft_consensus.cc:2396] T 0acc47ce21c3422698d1ef7277c3ed63 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c for term 3 because replica has last-logged OpId of term: 1 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260809 15:46:29.242115 25789 raft_consensus.cc:482] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:46:29.300292 23228 leader_election.cc:398] T 0c29937a80724ddb9d07a27534a146f3 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 2 pre-election: Vote denied by peer c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263) with higher term. Message: Invalid argument: T 0c29937a80724ddb9d07a27534a146f3 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 2. Current term is 3.
I20260809 15:46:29.278249 25795 raft_consensus.cc:482] T 28e17dcb5efe45b8b6a8c42da03b4819 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
I20260809 15:46:29.284885 25501 raft_consensus.cc:482] T 213543cdbe814c89a17a963469922c1b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:46:29.303369 23228 leader_election.cc:401] T 0c29937a80724ddb9d07a27534a146f3 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20260809 15:46:29.303480 23280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2424fd9bd7f14e508fd106cbfda786a4" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:29.293470 23522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3622f6e3e3641cea1960e3a31498bb7" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:29.302847 23285 raft_consensus.cc:2454] T 4564dfc6d4094c1b9240b367ccecf3b1 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 1.
I20260809 15:46:29.287106 23520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7660dfa2ef694927b8eda4100a2bcd9b" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
W20260809 15:46:29.310879 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 86970.3us
I20260809 15:46:29.345587 24471 raft_consensus.cc:1261] T 3924b648461e463b9d7925e5671d68ae P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260809 15:46:29.348577 23280 raft_consensus.cc:2454] T 2424fd9bd7f14e508fd106cbfda786a4 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 3.
I20260809 15:46:29.354606 23351 leader_election.cc:302] T a3622f6e3e3641cea1960e3a31498bb7 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:46:29.294117 25785 raft_consensus.cc:482] T 7baa2492df0340c4a5ac241959ce1ac3 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
W20260809 15:46:29.346621 23353 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 112942us
I20260809 15:46:29.302058 25868 raft_consensus.cc:482] T a3622f6e3e3641cea1960e3a31498bb7 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:46:29.360276 25861 leader_election.cc:288] T 7660dfa2ef694927b8eda4100a2bcd9b P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:29.360436 25868 raft_consensus.cc:3037] T a3622f6e3e3641cea1960e3a31498bb7 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Advancing to term 3
W20260809 15:46:29.362922 25842 log.cc:924] Time spent T 30c1740ef6874df0a332fbd6fd4b03ce P c0f38c1908414566aa9115bb2cbdd85d: Append to log took a long time: real 0.536s user 0.002s sys 0.007s
I20260809 15:46:29.364140 23404 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55320 (request call id 1058) took 1613 ms. Trace:
I20260809 15:46:29.364442 23404 rpcz_store.cc:276] 0809 15:46:27.750829 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:27.751474 (+ 645us) service_pool.cc:227] Handling call
0809 15:46:27.754545 (+ 3071us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 15:46:27.768687 (+ 14142us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:46:27.768715 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 15:46:27.769951 (+ 1236us) log.cc:842] Serialized 26 byte log entry
0809 15:46:27.999672 (+229721us) spinlock_profiling.cc:243] Waited 11.5 ms on lock 0x7b18003bbd80. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34ef1978b1 00007f34ef1a4661 00007f34e62da345 00007f34ef111a39 00007f34e62f78bb 00007f34e62f5cbf 00007f34e62f7e22 00007f34e8ea15bb 00007f34e8ea7063 00007f34e8ea7411 00007f34e91f45cf
0809 15:46:28.371028 (+371356us) raft_consensus.cc:1631] Marking committed up to 2
0809 15:46:28.371069 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:28.371146 (+ 77us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 15:46:29.362561 (+991415us) spinlock_profiling.cc:243] Waited 1.42 ms on lock 0x7b240014b3e8. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e613cf11 00007f34e6150cf4 00007f34e614edfb 00007f34e61dd559 00007f34e61dd1e5 00007f34e61dc82d 00007f34e8e8b46e 00007f34e8ea2c0c 00007f34e8ea1f36 00007f34e8ea1b58 00007f34e8eab551
0809 15:46:29.363926 (+ 1365us) raft_consensus.cc:1671] finished
0809 15:46:29.363945 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:29.364066 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2609,"raft.queue_time_us":8591,"raft.run_cpu_time_us":89,"raft.run_wall_time_us":89,"spinlock_wait_cycles":29655808,"thread_start_us":371209,"threads_started":1,"wal-append.queue_time_us":858433}
I20260809 15:46:29.357468 24452 raft_consensus.cc:2454] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 3.
I20260809 15:46:29.366628 25831 raft_consensus.cc:504] T 73f6f44c259547f1b05a3a434616fa54 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:29.298888 25849 raft_consensus.cc:482] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Starting leader election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
W20260809 15:46:29.376338 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 76174.5us
I20260809 15:46:29.367363 25507 raft_consensus.cc:2781] T a3622f6e3e3641cea1960e3a31498bb7 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260809 15:46:29.367693 25841 raft_consensus.cc:2781] T 2424fd9bd7f14e508fd106cbfda786a4 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260809 15:46:29.368019 25869 raft_consensus.cc:2781] T 0da4d5283c8a46878bd39b85e5c71f66 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260809 15:46:29.367895 25868 raft_consensus.cc:504] T a3622f6e3e3641cea1960e3a31498bb7 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:29.353447 25501 raft_consensus.cc:3037] T 213543cdbe814c89a17a963469922c1b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:29.372419 25859 raft_consensus.cc:504] T 73f6f44c259547f1b05a3a434616fa54 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:29.373675 25776 raft_consensus.cc:504] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:29.374539 25785 raft_consensus.cc:504] T 7baa2492df0340c4a5ac241959ce1ac3 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:29.307034 25807 consensus_queue.cc:1035] T 472816994d814cb198b69e5248697440 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.068s
I20260809 15:46:29.292765 23528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3924b648461e463b9d7925e5671d68ae" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:29.378547 25507 raft_consensus.cc:482] T a3622f6e3e3641cea1960e3a31498bb7 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:46:29.299576 25857 raft_consensus.cc:482] T f7715d71dbac447b9351fc5eb36c5e80 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
I20260809 15:46:29.298815 25851 raft_consensus.cc:482] T c5c2e425a74b4e64ae70b5def750434c P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
I20260809 15:46:29.346077 25652 leader_election.cc:288] T 2424fd9bd7f14e508fd106cbfda786a4 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:29.304469 25848 raft_consensus.cc:2726] T d2d4f749fcfd4fb3bfb8115af3471598 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263) with higher term. Message: Invalid argument: T d2d4f749fcfd4fb3bfb8115af3471598 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 2. Current term is 3.
I20260809 15:46:29.355284 25793 raft_consensus.cc:482] T 2db057fd10e149a9895dff6ef8c9f038 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
I20260809 15:46:29.391153 23521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2424fd9bd7f14e508fd106cbfda786a4" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:29.429467 23521 raft_consensus.cc:2454] T 2424fd9bd7f14e508fd106cbfda786a4 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 3.
W20260809 15:46:29.377022 24410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 173039us
I20260809 15:46:29.382674 25816 raft_consensus.cc:482] T 07fcabb569414bddbd39df6cedd24e64 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
I20260809 15:46:29.432981 25816 raft_consensus.cc:504] T 07fcabb569414bddbd39df6cedd24e64 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:29.434890 25816 leader_election.cc:288] T 07fcabb569414bddbd39df6cedd24e64 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:29.435971 25882 consensus_queue.cc:1046] T 16ca412a0dae4feebe85dd01bf1a9173 P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260809 15:46:29.436362 25785 leader_election.cc:288] T 7baa2492df0340c4a5ac241959ce1ac3 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:29.436672 25882 consensus_queue.cc:1225] T 16ca412a0dae4feebe85dd01bf1a9173 P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Peer 65c36d164b0540f099d592f38307ee6c log is divergent from this leader: its last log entry 3.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260809 15:46:29.437170 25882 raft_consensus.cc:482] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:46:29.437744 25864 consensus_queue.cc:1035] T 472816994d814cb198b69e5248697440 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260809 15:46:29.438318 25738 raft_consensus.cc:3032] T 16ca412a0dae4feebe85dd01bf1a9173 P 61e69695258f48baa984c9d395e8ce3f [term 2 LEADER]: Stepping down as leader of term 2
I20260809 15:46:29.438673 25738 raft_consensus.cc:729] T 16ca412a0dae4feebe85dd01bf1a9173 P 61e69695258f48baa984c9d395e8ce3f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 61e69695258f48baa984c9d395e8ce3f, State: Running, Role: LEADER
I20260809 15:46:29.438961 25864 raft_consensus.cc:482] T c5c2e425a74b4e64ae70b5def750434c P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
I20260809 15:46:29.439419 25864 raft_consensus.cc:504] T c5c2e425a74b4e64ae70b5def750434c P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:29.439468 25738 consensus_queue.cc:260] T 16ca412a0dae4feebe85dd01bf1a9173 P 61e69695258f48baa984c9d395e8ce3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:29.224947 25675 consensus_queue.cc:237] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:29.440675 25738 raft_consensus.cc:3037] T 16ca412a0dae4feebe85dd01bf1a9173 P 61e69695258f48baa984c9d395e8ce3f [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:29.360539 25876 raft_consensus.cc:482] T e71a05ba1d4d42bd95c3f85f9313399e P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
I20260809 15:46:29.395674 25587 raft_consensus.cc:482] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
I20260809 15:46:29.392643 23353 leader_election.cc:302] T cef3e7277d7d4760824a3436ed4a847d P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:29.398532 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2424fd9bd7f14e508fd106cbfda786a4" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:29.302364 25875 raft_consensus.cc:482] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:46:29.377987 25849 raft_consensus.cc:3037] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:29.376596 23520 raft_consensus.cc:2454] T 7660dfa2ef694927b8eda4100a2bcd9b P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 2.
I20260809 15:46:29.459337 25876 raft_consensus.cc:504] T e71a05ba1d4d42bd95c3f85f9313399e P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:29.378751 25808 raft_consensus.cc:482] T 28e17dcb5efe45b8b6a8c42da03b4819 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
I20260809 15:46:29.383467 25841 raft_consensus.cc:482] T 2424fd9bd7f14e508fd106cbfda786a4 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:46:29.454278 23527 raft_consensus.cc:2454] T 2424fd9bd7f14e508fd106cbfda786a4 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 3.
I20260809 15:46:29.386065 25869 raft_consensus.cc:482] T 0da4d5283c8a46878bd39b85e5c71f66 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:46:29.345168 25795 raft_consensus.cc:504] T 28e17dcb5efe45b8b6a8c42da03b4819 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:29.410670 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3622f6e3e3641cea1960e3a31498bb7" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:29.412217 23522 raft_consensus.cc:2379] T a3622f6e3e3641cea1960e3a31498bb7 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d in current term 3: Already voted for candidate 65c36d164b0540f099d592f38307ee6c in this term.
I20260809 15:46:29.412249 23519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4564dfc6d4094c1b9240b367ccecf3b1" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:29.415460 25831 leader_election.cc:288] T 73f6f44c259547f1b05a3a434616fa54 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:29.416594 23518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "294d4bcc1e7a477cb56455581277a60f" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:29.353178 25789 raft_consensus.cc:504] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:29.424749 25900 consensus_queue.cc:1035] T d0721a4403a440f383318a3cea9d1442 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.004s
I20260809 15:46:29.426190 25532 raft_consensus.cc:504] T 294d4bcc1e7a477cb56455581277a60f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:29.427341 24448 raft_consensus.cc:2454] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 3.
I20260809 15:46:29.437572 25882 raft_consensus.cc:504] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:29.395218 23283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e4ffcf83c234ae7910ddef3ec0b4bda" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:29.411464 25507 raft_consensus.cc:3037] T a3622f6e3e3641cea1960e3a31498bb7 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:29.456035 23528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "538a228ddfb54bc6a36528d5ff07597b" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:29.461663 23521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0acc47ce21c3422698d1ef7277c3ed63" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:29.465597 23519 raft_consensus.cc:2379] T 4564dfc6d4094c1b9240b367ccecf3b1 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 in current term 2: Already voted for candidate 65c36d164b0540f099d592f38307ee6c in this term.
W20260809 15:46:29.434967 25746 consensus_meta.cc:287] T 294d4bcc1e7a477cb56455581277a60f P 61e69695258f48baa984c9d395e8ce3f: Time spent flushing consensus metadata: real 0.503s user 0.005s sys 0.007s
I20260809 15:46:29.304754 25852 raft_consensus.cc:504] T 213543cdbe814c89a17a963469922c1b P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:46:29.469566 23345 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.68:34689, user_credentials={real_user=slave}} blocked reactor thread for 244064us
I20260809 15:46:29.424110 25851 raft_consensus.cc:504] T c5c2e425a74b4e64ae70b5def750434c P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:29.430248 25857 raft_consensus.cc:504] T f7715d71dbac447b9351fc5eb36c5e80 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:29.452421 25793 raft_consensus.cc:504] T 2db057fd10e149a9895dff6ef8c9f038 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:29.666392 23529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df20671a7b874dbb98c0ef28371c1308" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
W20260809 15:46:29.692627 23353 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 300053us
I20260809 15:46:29.801287 23353 leader_election.cc:302] T 0c29937a80724ddb9d07a27534a146f3 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:46:29.462095 25869 raft_consensus.cc:3037] T 0da4d5283c8a46878bd39b85e5c71f66 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:29.709316 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e14d118b2eaa42aa8a7cd4025aba7128" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:29.709715 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5712f6b6fffc4cc5a8ddfbe63e4097df" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:29.801895 23529 raft_consensus.cc:2454] T df20671a7b874dbb98c0ef28371c1308 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 2.
I20260809 15:46:29.461964 25841 raft_consensus.cc:3037] T 2424fd9bd7f14e508fd106cbfda786a4 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:46:29.752120 23280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df20671a7b874dbb98c0ef28371c1308" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:29.627763 23277 raft_consensus.cc:2454] T 294d4bcc1e7a477cb56455581277a60f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 3.
I20260809 15:46:29.461442 25876 leader_election.cc:288] T e71a05ba1d4d42bd95c3f85f9313399e P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:29.463070 25862 leader_election.cc:288] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
W20260809 15:46:29.659469 25840 consensus_meta.cc:287] T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d: Time spent flushing consensus metadata: real 0.538s user 0.006s sys 0.009s
I20260809 15:46:29.806797 23526 raft_consensus.cc:2454] T e14d118b2eaa42aa8a7cd4025aba7128 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 3.
I20260809 15:46:29.643575 23518 raft_consensus.cc:2379] T 294d4bcc1e7a477cb56455581277a60f P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 in current term 3: Already voted for candidate 65c36d164b0540f099d592f38307ee6c in this term.
I20260809 15:46:29.811477 23280 raft_consensus.cc:2454] T df20671a7b874dbb98c0ef28371c1308 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 2.
W20260809 15:46:29.739245 25890 log.cc:924] Time spent T 297c82ed906d4b48b184985eed3de601 P 67b3aeadfa7a432abe8af4d292e04ab9: Append to log took a long time: real 0.054s user 0.001s sys 0.000s
W20260809 15:46:29.708282 25908 log.cc:924] Time spent T 28e17dcb5efe45b8b6a8c42da03b4819 P 67b3aeadfa7a432abe8af4d292e04ab9: Append to log took a long time: real 0.060s user 0.000s sys 0.000s
W20260809 15:46:29.657171 25895 log.cc:924] Time spent T cef3e7277d7d4760824a3436ed4a847d P 65c36d164b0540f099d592f38307ee6c: Append to log took a long time: real 0.130s user 0.000s sys 0.000s
I20260809 15:46:29.463744 25875 raft_consensus.cc:3037] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:46:29.692845 25795 leader_election.cc:288] T 28e17dcb5efe45b8b6a8c42da03b4819 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:29.819703 23283 raft_consensus.cc:2379] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d in current term 3: Already voted for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in this term.
I20260809 15:46:29.786093 25776 leader_election.cc:288] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 election: Requested vote from peers 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:29.708648 25816 raft_consensus.cc:482] T 472816994d814cb198b69e5248697440 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:29.706334 25532 leader_election.cc:288] T 294d4bcc1e7a477cb56455581277a60f P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 3 election: Requested vote from peers 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:29.695369 25868 leader_election.cc:288] T a3622f6e3e3641cea1960e3a31498bb7 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:29.627822 25861 raft_consensus.cc:482] T 3d880de657e544468c82157769622a26 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
I20260809 15:46:29.791244 25859 leader_election.cc:288] T 73f6f44c259547f1b05a3a434616fa54 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:29.710788 25772 raft_consensus.cc:482] T 953eea268e17491ca76f3a555860a512 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
I20260809 15:46:29.802312 23521 raft_consensus.cc:2454] T 0acc47ce21c3422698d1ef7277c3ed63 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 2.
I20260809 15:46:29.700273 23392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e4ffcf83c234ae7910ddef3ec0b4bda" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:29.696307 24463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7660dfa2ef694927b8eda4100a2bcd9b" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:29.832585 23223 leader_election.cc:302] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:46:29.461790 25587 raft_consensus.cc:504] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:29.461783 25808 raft_consensus.cc:504] T 28e17dcb5efe45b8b6a8c42da03b4819 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:29.511026 25731 raft_consensus.cc:504] T e14d118b2eaa42aa8a7cd4025aba7128 P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:29.818631 25840 raft_consensus.cc:504] T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:29.461046 25873 raft_consensus.cc:504] T e14d118b2eaa42aa8a7cd4025aba7128 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:29.845710 23353 leader_election.cc:302] T 0acc47ce21c3422698d1ef7277c3ed63 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
W20260809 15:46:29.848042 23282 consensus_meta.cc:287] T 0c29937a80724ddb9d07a27534a146f3 P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent flushing consensus metadata: real 0.700s user 0.010s sys 0.002s
I20260809 15:46:29.849323 25927 consensus_queue.cc:1035] T 3924b648461e463b9d7925e5671d68ae P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.015s
I20260809 15:46:29.850111 25921 raft_consensus.cc:482] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:46:29.855719 25900 raft_consensus.cc:2726] T cef3e7277d7d4760824a3436ed4a847d P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260809 15:46:29.840090 23520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df20671a7b874dbb98c0ef28371c1308" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:29.856678 25861 raft_consensus.cc:504] T 3d880de657e544468c82157769622a26 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:29.857522 25681 raft_consensus.cc:2781] T 0c29937a80724ddb9d07a27534a146f3 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader election won for term 3
I20260809 15:46:29.845719 25795 raft_consensus.cc:482] T f7715d71dbac447b9351fc5eb36c5e80 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
W20260809 15:46:29.858558 25639 consensus_meta.cc:287] T ba56725e42f74a1baa4c04d9660727f9 P 65c36d164b0540f099d592f38307ee6c: Time spent flushing consensus metadata: real 0.815s user 0.003s sys 0.011s
I20260809 15:46:29.859242 25573 raft_consensus.cc:482] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:29.810509 25746 raft_consensus.cc:504] T 294d4bcc1e7a477cb56455581277a60f P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:29.860765 23345 leader_election.cc:302] T 8e4ffcf83c234ae7910ddef3ec0b4bda P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d; no voters:
W20260809 15:46:29.862342 25650 consensus_meta.cc:287] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 65c36d164b0540f099d592f38307ee6c: Time spent flushing consensus metadata: real 0.971s user 0.014s sys 0.000s
I20260809 15:46:29.862672 23280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73f6f44c259547f1b05a3a434616fa54" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:29.864337 23285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e71a05ba1d4d42bd95c3f85f9313399e" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:29.864383 23273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5c2e425a74b4e64ae70b5def750434c" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:29.838977 24452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "294d4bcc1e7a477cb56455581277a60f" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f"
I20260809 15:46:29.855875 25885 raft_consensus.cc:482] T c0741ab44d99439abc3aa3bdd27ee160 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
I20260809 15:46:29.883665 23353 leader_election.cc:302] T df20671a7b874dbb98c0ef28371c1308 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
W20260809 15:46:29.907043 23223 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.68:34689, user_credentials={real_user=slave}} blocked reactor thread for 74544.7us
W20260809 15:46:30.002024 23345 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.68:34689, user_credentials={real_user=slave}} blocked reactor thread for 141395us
W20260809 15:46:29.946761 25791 consensus_meta.cc:287] T a88421dba80d4d3388aa5eb62e23853b P 65c36d164b0540f099d592f38307ee6c: Time spent flushing consensus metadata: real 0.822s user 0.015s sys 0.000s
I20260809 15:46:29.880472 23282 raft_consensus.cc:2454] T 0c29937a80724ddb9d07a27534a146f3 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 3.
I20260809 15:46:29.865450 25772 raft_consensus.cc:504] T 953eea268e17491ca76f3a555860a512 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:29.875540 25883 raft_consensus.cc:482] T 563daf42d7784db195ea99c3cb21b370 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
I20260809 15:46:29.930763 23394 raft_consensus.cc:1226] T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 3. Current term is 4. Ops: []
W20260809 15:46:29.952419 23396 consensus_meta.cc:287] T 998c6cee72cf40228c6dba064c0db2da P c0f38c1908414566aa9115bb2cbdd85d: Time spent flushing consensus metadata: real 0.750s user 0.007s sys 0.001s
W20260809 15:46:29.991482 24410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 333501us
I20260809 15:46:29.993397 23280 raft_consensus.cc:3037] T 73f6f44c259547f1b05a3a434616fa54 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Advancing to term 3
W20260809 15:46:30.015117 23353 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 131511us
W20260809 15:46:29.869297 25822 consensus_meta.cc:287] T 4564dfc6d4094c1b9240b367ccecf3b1 P c0f38c1908414566aa9115bb2cbdd85d: Time spent flushing consensus metadata: real 0.836s user 0.008s sys 0.004s
I20260809 15:46:29.875427 25857 leader_election.cc:288] T f7715d71dbac447b9351fc5eb36c5e80 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:29.867494 25816 raft_consensus.cc:504] T 472816994d814cb198b69e5248697440 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:29.944494 24448 raft_consensus.cc:1261] T 16ca412a0dae4feebe85dd01bf1a9173 P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 5. (term mismatch)
I20260809 15:46:30.019537 23397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0acc47ce21c3422698d1ef7277c3ed63" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:30.019855 23404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4564dfc6d4094c1b9240b367ccecf3b1" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:30.017889 25695 raft_consensus.cc:2726] T 0c29937a80724ddb9d07a27534a146f3 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263) with higher term. Message: Invalid argument: T 0c29937a80724ddb9d07a27534a146f3 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 2. Current term is 3.
I20260809 15:46:30.015479 25785 raft_consensus.cc:482] T d0721a4403a440f383318a3cea9d1442 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:29.911697 25700 raft_consensus.cc:2781] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader election won for term 3
I20260809 15:46:30.020690 24468 raft_consensus.cc:2379] T 294d4bcc1e7a477cb56455581277a60f P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c in current term 4: Already voted for candidate 61e69695258f48baa984c9d395e8ce3f in this term.
I20260809 15:46:30.004557 25639 raft_consensus.cc:504] T ba56725e42f74a1baa4c04d9660727f9 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:30.002197 23520 raft_consensus.cc:2454] T df20671a7b874dbb98c0ef28371c1308 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 2.
I20260809 15:46:29.869686 23392 raft_consensus.cc:2454] T 8e4ffcf83c234ae7910ddef3ec0b4bda P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 2.
I20260809 15:46:29.942061 25913 raft_consensus.cc:3037] T 0acc47ce21c3422698d1ef7277c3ed63 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:29.943132 25793 leader_election.cc:288] T 2db057fd10e149a9895dff6ef8c9f038 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:29.909140 25789 leader_election.cc:288] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:29.865842 25501 raft_consensus.cc:504] T 213543cdbe814c89a17a963469922c1b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:29.950944 25873 leader_election.cc:288] T e14d118b2eaa42aa8a7cd4025aba7128 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 election: Requested vote from peers 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:30.021510 25885 raft_consensus.cc:504] T c0741ab44d99439abc3aa3bdd27ee160 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
W20260809 15:46:30.025887 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 202937us
I20260809 15:46:30.023985 23282 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53562 (request call id 1522) took 1462 ms. Trace:
I20260809 15:46:30.288476 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07fcabb569414bddbd39df6cedd24e64" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:30.297712 23285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7715d71dbac447b9351fc5eb36c5e80" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:30.395234 23396 raft_consensus.cc:2454] T 998c6cee72cf40228c6dba064c0db2da P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 5.
I20260809 15:46:30.327703 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e71a05ba1d4d42bd95c3f85f9313399e" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
W20260809 15:46:30.295800 23470 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 47415us
W20260809 15:46:30.020184 25917 log.cc:924] Time spent T 00000000000000000000000000000000 P a86a2513b8024a869d80303dcbf44163: Append to log took a long time: real 0.174s user 0.000s sys 0.000s
I20260809 15:46:30.395720 23402 raft_consensus.cc:2454] T 07fcabb569414bddbd39df6cedd24e64 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 3.
I20260809 15:46:30.394281 23282 rpcz_store.cc:276] 0809 15:46:28.554127 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:28.554289 (+ 162us) service_pool.cc:227] Handling call
0809 15:46:29.074130 (+519841us) spinlock_profiling.cc:243] Waited 8.36 ms on lock 0x7b5800003398. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e613cf11 00007f34e62c0851 00007f34e62c0702 00007f34ef1b0b09 00007f34ee0ec730 00007f34ee1b279b 00007f34ee1b2821 00007f34ee1621b0 00007f34ee161a94 00007f34e7849c04 00007f34e7849b92
0809 15:46:30.016276 (+942146us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":19226880}
I20260809 15:46:30.002923 25650 raft_consensus.cc:2726] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263) with higher term. Message: Invalid argument: T 5ad8a3b25d7b49e5a2142d5709edd0c7 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c for earlier term 2. Current term is 3.
I20260809 15:46:30.291108 23472 leader_election.cc:302] T df20671a7b874dbb98c0ef28371c1308 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:46:29.951813 25921 raft_consensus.cc:504] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:30.001940 25681 raft_consensus.cc:686] T 0c29937a80724ddb9d07a27534a146f3 P c0f38c1908414566aa9115bb2cbdd85d [term 3 LEADER]: Becoming Leader. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:46:30.407021 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ced718c1a5c4da79cb8c1ebd1435b87" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
W20260809 15:46:30.031653 25856 log.cc:924] Time spent T 30c1740ef6874df0a332fbd6fd4b03ce P 65c36d164b0540f099d592f38307ee6c: Append to log took a long time: real 1.026s user 0.004s sys 0.001s
I20260809 15:46:30.394989 24448 pending_rounds.cc:76] T 16ca412a0dae4feebe85dd01bf1a9173 P 61e69695258f48baa984c9d395e8ce3f: Aborting all ops after (but not including) 4
I20260809 15:46:30.409642 25907 raft_consensus.cc:482] T 213543cdbe814c89a17a963469922c1b P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:30.381780 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a823d35f4f348d1b52ba9e6d80b00d2" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f"
W20260809 15:46:30.380964 25875 consensus_meta.cc:287] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 65c36d164b0540f099d592f38307ee6c: Time spent flushing consensus metadata: real 0.517s user 0.006s sys 0.000s
I20260809 15:46:30.382344 25868 raft_consensus.cc:482] T 6ae14a74a26f434db08c652e097385c8 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:30.082301 25532 raft_consensus.cc:482] T a88421dba80d4d3388aa5eb62e23853b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:30.022714 25694 raft_consensus.cc:482] T 16ca412a0dae4feebe85dd01bf1a9173 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:30.330680 25927 raft_consensus.cc:482] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:46:30.282050 25899 raft_consensus.cc:482] T 763e58d6ba644eea830a6c1664d39c9b P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:30.403398 25773 raft_consensus.cc:482] T 3296ef50be254fc29cfc77d22a8f7a23 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
I20260809 15:46:30.346474 25864 leader_election.cc:288] T c5c2e425a74b4e64ae70b5def750434c P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:30.302621 25731 leader_election.cc:288] T e14d118b2eaa42aa8a7cd4025aba7128 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 4 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:30.301169 25851 leader_election.cc:288] T c5c2e425a74b4e64ae70b5def750434c P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:30.020185 25852 leader_election.cc:288] T 213543cdbe814c89a17a963469922c1b P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:30.291705 23393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7baa2492df0340c4a5ac241959ce1ac3" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:29.996083 25795 raft_consensus.cc:504] T f7715d71dbac447b9351fc5eb36c5e80 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:30.393621 25848 raft_consensus.cc:482] T d0721a4403a440f383318a3cea9d1442 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:46:30.441933 25587 leader_election.cc:288] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:30.440568 23519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73f6f44c259547f1b05a3a434616fa54" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:30.503157 23519 raft_consensus.cc:2379] T 73f6f44c259547f1b05a3a434616fa54 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d in current term 3: Already voted for candidate 65c36d164b0540f099d592f38307ee6c in this term.
I20260809 15:46:30.418273 23522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5c2e425a74b4e64ae70b5def750434c" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:30.440117 23521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a823d35f4f348d1b52ba9e6d80b00d2" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:30.450030 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73f6f44c259547f1b05a3a434616fa54" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:30.043320 25652 raft_consensus.cc:482] T 4b30f28311964a0b9149a50495e83d8f P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:46:30.504426 23402 raft_consensus.cc:2379] T 73f6f44c259547f1b05a3a434616fa54 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c in current term 3: Already voted for candidate c0f38c1908414566aa9115bb2cbdd85d in this term.
I20260809 15:46:30.423785 24472 raft_consensus.cc:3037] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 61e69695258f48baa984c9d395e8ce3f [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:30.282070 23399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7660dfa2ef694927b8eda4100a2bcd9b" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:30.292317 24455 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e14d118b2eaa42aa8a7cd4025aba7128" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f"
I20260809 15:46:30.425967 23517 raft_consensus.cc:2454] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 3.
I20260809 15:46:30.414139 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e14d118b2eaa42aa8a7cd4025aba7128" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:30.540997 23526 raft_consensus.cc:3037] T e14d118b2eaa42aa8a7cd4025aba7128 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:46:30.455466 23280 raft_consensus.cc:2454] T 73f6f44c259547f1b05a3a434616fa54 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 3.
I20260809 15:46:30.402348 25900 raft_consensus.cc:2781] T 8e4ffcf83c234ae7910ddef3ec0b4bda P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260809 15:46:30.003702 25573 raft_consensus.cc:504] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
W20260809 15:46:30.457273 23470 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 59155.5us
I20260809 15:46:30.545071 23470 leader_election.cc:302] T 998c6cee72cf40228c6dba064c0db2da P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:46:30.031875 25931 raft_consensus.cc:2781] T df20671a7b874dbb98c0ef28371c1308 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260809 15:46:30.403172 25840 leader_election.cc:288] T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 4 election: Requested vote from peers 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
W20260809 15:46:30.417943 25914 log.cc:924] Time spent T 30c1740ef6874df0a332fbd6fd4b03ce P 67b3aeadfa7a432abe8af4d292e04ab9: Append to log took a long time: real 0.424s user 0.000s sys 0.000s
W20260809 15:46:30.420603 25507 consensus_meta.cc:287] T a3622f6e3e3641cea1960e3a31498bb7 P c0f38c1908414566aa9115bb2cbdd85d: Time spent flushing consensus metadata: real 0.609s user 0.008s sys 0.000s
I20260809 15:46:30.478852 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "907bacfa79d04a55a68c8dbe60056462" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:30.017570 25791 raft_consensus.cc:504] T a88421dba80d4d3388aa5eb62e23853b P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:46:30.419404 25841 consensus_meta.cc:287] T 2424fd9bd7f14e508fd106cbfda786a4 P c0f38c1908414566aa9115bb2cbdd85d: Time spent flushing consensus metadata: real 0.585s user 0.008s sys 0.000s
I20260809 15:46:30.410749 23395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2424fd9bd7f14e508fd106cbfda786a4" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:30.406607 25808 leader_election.cc:288] T 28e17dcb5efe45b8b6a8c42da03b4819 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:30.412870 25700 raft_consensus.cc:686] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 LEADER]: Becoming Leader. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Running, Role: LEADER
I20260809 15:46:30.413341 25789 raft_consensus.cc:2781] T df20671a7b874dbb98c0ef28371c1308 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260809 15:46:30.450789 24448 pending_rounds.cc:97] T 16ca412a0dae4feebe85dd01bf1a9173 P 61e69695258f48baa984c9d395e8ce3f: Aborting uncommitted NO_OP operation due to leader change: 2.5
I20260809 15:46:30.487129 25905 raft_consensus.cc:482] T 2db057fd10e149a9895dff6ef8c9f038 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
I20260809 15:46:30.419714 25746 leader_election.cc:288] T 294d4bcc1e7a477cb56455581277a60f P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 4 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:30.487550 25885 leader_election.cc:288] T c0741ab44d99439abc3aa3bdd27ee160 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:30.398128 25822 raft_consensus.cc:504] T 4564dfc6d4094c1b9240b367ccecf3b1 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:30.487743 23520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7715d71dbac447b9351fc5eb36c5e80" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
W20260809 15:46:30.407444 23396 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55320 (request call id 1075) took 1267 ms (client timeout 1672 ms). Trace:
I20260809 15:46:30.474352 25857 raft_consensus.cc:482] T 07fcabb569414bddbd39df6cedd24e64 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
I20260809 15:46:30.534502 23285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07fcabb569414bddbd39df6cedd24e64" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:30.566898 23520 raft_consensus.cc:2454] T f7715d71dbac447b9351fc5eb36c5e80 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 2.
W20260809 15:46:30.568243 25934 log.cc:924] Time spent T 5ad8a3b25d7b49e5a2142d5709edd0c7 P c0f38c1908414566aa9115bb2cbdd85d: Append to log took a long time: real 0.063s user 0.000s sys 0.000s
W20260809 15:46:30.568526 23396 rpcz_store.cc:269] 0809 15:46:29.132641 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:29.159598 (+ 26957us) service_pool.cc:227] Handling call
0809 15:46:30.399784 (+1240186us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:46:30.569563 23396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3aa6a9a52804df0ba9b0ca3c82f38c3" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:30.504961 25899 raft_consensus.cc:504] T 763e58d6ba644eea830a6c1664d39c9b P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:30.570680 23396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3622f6e3e3641cea1960e3a31498bb7" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:30.538231 25652 raft_consensus.cc:504] T 4b30f28311964a0b9149a50495e83d8f P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:30.554955 23399 raft_consensus.cc:2379] T 7660dfa2ef694927b8eda4100a2bcd9b P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c in current term 3: Already voted for candidate c0f38c1908414566aa9115bb2cbdd85d in this term.
I20260809 15:46:30.572536 25857 raft_consensus.cc:504] T 07fcabb569414bddbd39df6cedd24e64 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:30.404358 25785 raft_consensus.cc:504] T d0721a4403a440f383318a3cea9d1442 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:46:30.488468 25869 consensus_meta.cc:287] T 0da4d5283c8a46878bd39b85e5c71f66 P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent flushing consensus metadata: real 0.670s user 0.008s sys 0.000s
I20260809 15:46:30.479064 25882 leader_election.cc:288] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:30.490306 25927 raft_consensus.cc:504] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:30.538396 23282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7baa2492df0340c4a5ac241959ce1ac3" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:30.539001 23273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3622f6e3e3641cea1960e3a31498bb7" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:30.503702 23522 raft_consensus.cc:2454] T c5c2e425a74b4e64ae70b5def750434c P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 2.
I20260809 15:46:30.242576 25883 raft_consensus.cc:504] T 563daf42d7784db195ea99c3cb21b370 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
W20260809 15:46:30.506597 23472 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 215562us
W20260809 15:46:30.485719 24406 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.20.202.65:35459 timed out after 2.053s (SENT) [suppressed 27 similar messages]
I20260809 15:46:30.540272 23274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73f6f44c259547f1b05a3a434616fa54" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:30.546947 23470 leader_election.cc:302] T 4564dfc6d4094c1b9240b367ccecf3b1 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:46:30.504778 25861 leader_election.cc:288] T 3d880de657e544468c82157769622a26 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:30.476854 25868 raft_consensus.cc:504] T 6ae14a74a26f434db08c652e097385c8 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:46:30.550704 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 148476us
I20260809 15:46:30.549316 23519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3924b648461e463b9d7925e5671d68ae" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:30.479167 25694 raft_consensus.cc:504] T 16ca412a0dae4feebe85dd01bf1a9173 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:30.554404 25929 raft_consensus.cc:482] T f9e7cadcd24d45fcbfa84209225e8f2d P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:46:30.554826 23280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ced718c1a5c4da79cb8c1ebd1435b87" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:30.555490 23277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e17dcb5efe45b8b6a8c42da03b4819" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:30.458609 25875 raft_consensus.cc:504] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:30.481701 25532 raft_consensus.cc:504] T a88421dba80d4d3388aa5eb62e23853b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:30.539193 25772 leader_election.cc:288] T 953eea268e17491ca76f3a555860a512 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:30.482780 25773 raft_consensus.cc:504] T 3296ef50be254fc29cfc77d22a8f7a23 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:30.517091 23393 raft_consensus.cc:2454] T 7baa2492df0340c4a5ac241959ce1ac3 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 1.
I20260809 15:46:30.542083 25933 raft_consensus.cc:482] T cee02fff2c9c48a19f43eb358a89e7b0 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
I20260809 15:46:30.545693 23392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2db057fd10e149a9895dff6ef8c9f038" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:30.533131 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e17dcb5efe45b8b6a8c42da03b4819" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:30.486488 25907 raft_consensus.cc:504] T 213543cdbe814c89a17a963469922c1b P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:30.506798 25848 raft_consensus.cc:504] T d0721a4403a440f383318a3cea9d1442 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:30.545426 25681 consensus_queue.cc:237] T 0c29937a80724ddb9d07a27534a146f3 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:30.572937 23399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "213543cdbe814c89a17a963469922c1b" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:30.909369 24472 raft_consensus.cc:2454] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 3.
I20260809 15:46:30.557057 25841 raft_consensus.cc:504] T 2424fd9bd7f14e508fd106cbfda786a4 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:30.556910 25851 raft_consensus.cc:2781] T 998c6cee72cf40228c6dba064c0db2da P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Leader election won for term 5
W20260809 15:46:30.665406 24410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 237646us
I20260809 15:46:30.911885 23274 raft_consensus.cc:2379] T 73f6f44c259547f1b05a3a434616fa54 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c in current term 3: Already voted for candidate c0f38c1908414566aa9115bb2cbdd85d in this term.
I20260809 15:46:30.572278 23270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e17dcb5efe45b8b6a8c42da03b4819" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:30.924177 23270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5c2e425a74b4e64ae70b5def750434c" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:30.915625 23402 raft_consensus.cc:2454] T 28e17dcb5efe45b8b6a8c42da03b4819 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 2.
I20260809 15:46:30.917615 23273 raft_consensus.cc:3037] T a3622f6e3e3641cea1960e3a31498bb7 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:30.675024 23526 raft_consensus.cc:2454] T e14d118b2eaa42aa8a7cd4025aba7128 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 4.
I20260809 15:46:30.874575 23353 leader_election.cc:302] T f7715d71dbac447b9351fc5eb36c5e80 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
W20260809 15:46:30.919822 23470 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 372914us
I20260809 15:46:30.928928 23470 leader_election.cc:302] T 2424fd9bd7f14e508fd106cbfda786a4 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:46:30.929081 23274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "213543cdbe814c89a17a963469922c1b" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:30.931456 23274 raft_consensus.cc:2454] T 213543cdbe814c89a17a963469922c1b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 2.
I20260809 15:46:30.931208 23228 leader_election.cc:302] T 2424fd9bd7f14e508fd106cbfda786a4 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:46:30.553377 25840 raft_consensus.cc:482] T 953eea268e17491ca76f3a555860a512 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
I20260809 15:46:30.909227 25921 leader_election.cc:288] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:30.934929 25861 raft_consensus.cc:2781] T 4564dfc6d4094c1b9240b367ccecf3b1 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260809 15:46:30.935456 25861 raft_consensus.cc:482] T 4564dfc6d4094c1b9240b367ccecf3b1 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:46:30.935905 25861 raft_consensus.cc:3037] T 4564dfc6d4094c1b9240b367ccecf3b1 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:30.936749 23393 raft_consensus.cc:1261] T cef3e7277d7d4760824a3436ed4a847d P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260809 15:46:30.887137 25639 leader_election.cc:288] T ba56725e42f74a1baa4c04d9660727f9 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:30.939730 23393 raft_consensus.cc:1261] T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Refusing update from remote peer 61e69695258f48baa984c9d395e8ce3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260809 15:46:30.940886 23276 raft_consensus.cc:1261] T 16ca412a0dae4feebe85dd01bf1a9173 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260809 15:46:30.941818 25694 leader_election.cc:288] T 16ca412a0dae4feebe85dd01bf1a9173 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:30.942636 23519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "294d4bcc1e7a477cb56455581277a60f" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:30.944077 23519 raft_consensus.cc:3037] T 294d4bcc1e7a477cb56455581277a60f P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:46:30.587647 24455 raft_consensus.cc:2379] T e14d118b2eaa42aa8a7cd4025aba7128 P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 in current term 4: Already voted for candidate 61e69695258f48baa984c9d395e8ce3f in this term.
I20260809 15:46:30.950937 23470 leader_election.cc:302] T 294d4bcc1e7a477cb56455581277a60f P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:46:30.952072 25852 raft_consensus.cc:2781] T 2424fd9bd7f14e508fd106cbfda786a4 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260809 15:46:30.952564 25852 raft_consensus.cc:482] T 2424fd9bd7f14e508fd106cbfda786a4 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:46:30.953192 25852 raft_consensus.cc:3037] T 2424fd9bd7f14e508fd106cbfda786a4 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:46:30.950865 25793 raft_consensus.cc:482] T f9e7cadcd24d45fcbfa84209225e8f2d P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:46:30.925429 25869 raft_consensus.cc:504] T 0da4d5283c8a46878bd39b85e5c71f66 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:30.956441 25869 leader_election.cc:288] T 0da4d5283c8a46878bd39b85e5c71f66 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 3 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:30.958568 25907 leader_election.cc:288] T 213543cdbe814c89a17a963469922c1b P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:30.865073 25501 leader_election.cc:288] T 213543cdbe814c89a17a963469922c1b P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 2 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:30.870736 25941 raft_consensus.cc:482] T 0da4d5283c8a46878bd39b85e5c71f66 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:30.959971 25941 raft_consensus.cc:504] T 0da4d5283c8a46878bd39b85e5c71f66 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:30.960790 23285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2db057fd10e149a9895dff6ef8c9f038" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:30.962445 23283 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53562 (request call id 1538) took 1121 ms. Trace:
I20260809 15:46:30.916320 25808 raft_consensus.cc:482] T 297c82ed906d4b48b184985eed3de601 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:30.962890 23283 rpcz_store.cc:276] 0809 15:46:29.840722 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:29.840881 (+ 159us) service_pool.cc:227] Handling call
0809 15:46:29.859033 (+ 18152us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:30.441574 (+582541us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:46:30.441603 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 15:46:30.441640 (+ 37us) raft_consensus.cc:1631] Marking committed up to 1
0809 15:46:30.441666 (+ 26us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:30.962164 (+520498us) spinlock_profiling.cc:243] Waited 582 ms on lock 0x7b540030eb54. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e9220f51 00007f34e921f99e 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772 00007f34e764a94c 00007f34e7649d86
0809 15:46:30.962213 (+ 49us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:30.962333 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1339341184}
W20260809 15:46:30.943476 23353 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 68968.7us
W20260809 15:46:30.947278 23517 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56766 (request call id 1655) took 1921 ms (client timeout 1915 ms). Trace:
I20260809 15:46:30.947461 25948 raft_consensus.cc:2781] T f7715d71dbac447b9351fc5eb36c5e80 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260809 15:46:30.944608 25933 raft_consensus.cc:504] T cee02fff2c9c48a19f43eb358a89e7b0 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:30.582559 25899 leader_election.cc:288] T 763e58d6ba644eea830a6c1664d39c9b P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:30.949175 25773 leader_election.cc:288] T 3296ef50be254fc29cfc77d22a8f7a23 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:30.817431 25477 raft_consensus.cc:482] T 763e58d6ba644eea830a6c1664d39c9b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:30.553016 25507 raft_consensus.cc:504] T a3622f6e3e3641cea1960e3a31498bb7 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:46:30.566560 25911 log.cc:924] Time spent T d2d4f749fcfd4fb3bfb8115af3471598 P c0f38c1908414566aa9115bb2cbdd85d: Append to log took a long time: real 0.329s user 0.000s sys 0.000s
I20260809 15:46:30.893105 25816 leader_election.cc:288] T 472816994d814cb198b69e5248697440 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:30.965164 23353 leader_election.cc:302] T c5c2e425a74b4e64ae70b5def750434c P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:46:30.909164 23286 raft_consensus.cc:1261] T cef3e7277d7d4760824a3436ed4a847d P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20260809 15:46:30.576256 25863 consensus_meta.cc:287] T 538a228ddfb54bc6a36528d5ff07597b P 65c36d164b0540f099d592f38307ee6c: Time spent flushing consensus metadata: real 1.411s user 0.010s sys 0.005s
I20260809 15:46:30.954555 25793 raft_consensus.cc:504] T f9e7cadcd24d45fcbfa84209225e8f2d P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:30.870823 25942 raft_consensus.cc:482] T 00a1881267a14ba8b78c81040f17a014 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
W20260809 15:46:30.572937 25478 consensus_meta.cc:287] T 7660dfa2ef694927b8eda4100a2bcd9b P 61e69695258f48baa984c9d395e8ce3f: Time spent flushing consensus metadata: real 1.227s user 0.000s sys 0.008s
I20260809 15:46:30.556046 25931 raft_consensus.cc:482] T df20671a7b874dbb98c0ef28371c1308 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:46:30.952247 24410 leader_election.cc:302] T e14d118b2eaa42aa8a7cd4025aba7128 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c; no voters:
I20260809 15:46:30.563645 25789 raft_consensus.cc:482] T df20671a7b874dbb98c0ef28371c1308 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:46:30.717809 25695 raft_consensus.cc:482] T ae6d2c08a7a94f58ac8dd698e3fd456c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:30.964604 23394 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55320 (request call id 1076) took 1831 ms. Trace:
I20260809 15:46:31.101673 23399 raft_consensus.cc:2454] T 213543cdbe814c89a17a963469922c1b P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 2.
I20260809 15:46:31.109390 23402 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d3be1e356b254d22b995be32f1c12f6f"
dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
mode: GRACEFUL
new_leader_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
from {username='slave'} at 127.0.0.1:51280
W20260809 15:46:30.965049 23517 rpcz_store.cc:269] 0809 15:46:29.025840 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:29.026043 (+ 203us) service_pool.cc:227] Handling call
0809 15:46:30.947063 (+1921020us) spinlock_profiling.cc:243] Waited 1.37 s on lock 0x7b54000753d4. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e92242ae 00007f34ee161c27 00007f34e7849c04 00007f34e7849b92 00007f34e7849ac1 00007f34e7849a3c 00007f34e7848cf2 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:46:30.947155 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3142914816}
I20260809 15:46:30.586865 25900 raft_consensus.cc:482] T 8e4ffcf83c234ae7910ddef3ec0b4bda P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting leader election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
I20260809 15:46:31.111151 23404 raft_consensus.cc:2379] T 4564dfc6d4094c1b9240b367ccecf3b1 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c in current term 3: Already voted for candidate c0f38c1908414566aa9115bb2cbdd85d in this term.
I20260809 15:46:30.912333 24448 raft_consensus.cc:2866] T 16ca412a0dae4feebe85dd01bf1a9173 P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 15:46:31.133329 25694 raft_consensus.cc:482] T d2d4f749fcfd4fb3bfb8115af3471598 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:31.012281 25929 raft_consensus.cc:504] T f9e7cadcd24d45fcbfa84209225e8f2d P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:30.962612 25941 leader_election.cc:288] T 0da4d5283c8a46878bd39b85e5c71f66 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:31.120132 23526 raft_consensus.cc:1261] T d3e94b7e50754a81b3d72a07476cb7b9 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Refusing update from remote peer 61e69695258f48baa984c9d395e8ce3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260809 15:46:31.120062 23527 raft_consensus.cc:2354] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 3. Current term is 4.
I20260809 15:46:30.967061 25573 leader_election.cc:288] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
W20260809 15:46:31.120954 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 189799us
I20260809 15:46:31.125953 23273 raft_consensus.cc:2454] T a3622f6e3e3641cea1960e3a31498bb7 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 3.
I20260809 15:46:30.579560 25947 raft_consensus.cc:482] T 0a51c0baaa6e432997c9d3552af4f254 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
W20260809 15:46:31.127384 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 180998us
W20260809 15:46:31.317675 23470 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 366795us
I20260809 15:46:31.137476 23394 rpcz_store.cc:276] 0809 15:46:29.132935 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:29.133046 (+ 111us) service_pool.cc:227] Handling call
0809 15:46:29.141820 (+ 8774us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:30.395825 (+1254005us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:30.964391 (+568566us) spinlock_profiling.cc:243] Waited 769 ms on lock 0x7b54002206d4. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e9220f51 00007f34e921f99e 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772 00007f34e764a94c 00007f34e7649d86
0809 15:46:30.964548 (+ 157us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1769503616}
W20260809 15:46:31.170531 24410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 218344us
I20260809 15:46:31.383836 23470 leader_election.cc:302] T 213543cdbe814c89a17a963469922c1b P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:46:31.384167 24410 leader_election.cc:302] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:46:31.283289 23519 raft_consensus.cc:2454] T 294d4bcc1e7a477cb56455581277a60f P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 4.
W20260809 15:46:31.135164 23353 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 170062us
I20260809 15:46:30.573292 25905 raft_consensus.cc:504] T 2db057fd10e149a9895dff6ef8c9f038 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:30.909792 25700 consensus_queue.cc:237] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
W20260809 15:46:31.137784 23345 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.599s user 0.108s sys 0.121s
I20260809 15:46:31.113273 25948 raft_consensus.cc:482] T f7715d71dbac447b9351fc5eb36c5e80 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting leader election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
I20260809 15:46:31.064816 25808 raft_consensus.cc:504] T 297c82ed906d4b48b184985eed3de601 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:30.697214 25879 raft_consensus.cc:482] T 30c1740ef6874df0a332fbd6fd4b03ce P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:31.010908 25840 raft_consensus.cc:504] T 953eea268e17491ca76f3a555860a512 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:31.138679 23396 raft_consensus.cc:2379] T a3622f6e3e3641cea1960e3a31498bb7 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c in current term 3: Already voted for candidate c0f38c1908414566aa9115bb2cbdd85d in this term.
I20260809 15:46:31.138463 25789 raft_consensus.cc:3037] T df20671a7b874dbb98c0ef28371c1308 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:31.136997 25931 raft_consensus.cc:3037] T df20671a7b874dbb98c0ef28371c1308 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Advancing to term 3
W20260809 15:46:31.283454 24406 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 797695us
I20260809 15:46:31.170128 23402 raft_consensus.cc:595] T d3be1e356b254d22b995be32f1c12f6f P c0f38c1908414566aa9115bb2cbdd85d [term 1 LEADER]: Received request to transfer leadership to TS 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:31.133620 25639 raft_consensus.cc:482] T 0c48a3049897452383257aa757834fb5 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
W20260809 15:46:31.400877 23345 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.862s user 0.108s sys 0.121s
I20260809 15:46:31.140553 23395 raft_consensus.cc:2379] T 2424fd9bd7f14e508fd106cbfda786a4 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c in current term 4: Already voted for candidate c0f38c1908414566aa9115bb2cbdd85d in this term.
I20260809 15:46:31.225044 25898 raft_consensus.cc:2781] T e14d118b2eaa42aa8a7cd4025aba7128 P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Leader election won for term 4
I20260809 15:46:31.327450 24448 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38734 (request call id 193) took 1442 ms. Trace:
I20260809 15:46:31.268064 25967 raft_consensus.cc:482] T d2d4f749fcfd4fb3bfb8115af3471598 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:31.266969 25587 raft_consensus.cc:482] T ba56725e42f74a1baa4c04d9660727f9 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:31.266479 25869 raft_consensus.cc:482] T cef3e7277d7d4760824a3436ed4a847d P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:31.138141 25478 raft_consensus.cc:504] T 7660dfa2ef694927b8eda4100a2bcd9b P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
W20260809 15:46:31.184749 24468 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38734 (request call id 190) took 1933 ms (client timeout 2070 ms). Trace:
I20260809 15:46:31.283723 25731 raft_consensus.cc:482] T 16ca412a0dae4feebe85dd01bf1a9173 P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:31.265837 25532 leader_election.cc:288] T a88421dba80d4d3388aa5eb62e23853b P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:31.177948 23520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3aa6a9a52804df0ba9b0ca3c82f38c3" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:31.403618 23228 leader_election.cc:302] T 0acc47ce21c3422698d1ef7277c3ed63 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:46:31.322465 25921 raft_consensus.cc:482] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:46:31.265224 25951 raft_consensus.cc:482] T 907bacfa79d04a55a68c8dbe60056462 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
I20260809 15:46:31.136539 25885 raft_consensus.cc:2781] T 2424fd9bd7f14e508fd106cbfda786a4 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260809 15:46:31.135123 25863 raft_consensus.cc:504] T 538a228ddfb54bc6a36528d5ff07597b P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:31.139602 25953 raft_consensus.cc:2781] T c5c2e425a74b4e64ae70b5def750434c P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260809 15:46:31.400089 25694 raft_consensus.cc:504] T d2d4f749fcfd4fb3bfb8115af3471598 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:31.319931 25883 leader_election.cc:288] T 563daf42d7784db195ea99c3cb21b370 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:31.437145 23392 raft_consensus.cc:2454] T 2db057fd10e149a9895dff6ef8c9f038 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 2.
I20260809 15:46:31.432395 25941 raft_consensus.cc:2781] T 213543cdbe814c89a17a963469922c1b P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260809 15:46:31.428599 25946 raft_consensus.cc:2781] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260809 15:46:31.180104 25652 leader_election.cc:288] T 4b30f28311964a0b9149a50495e83d8f P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:31.170902 25942 raft_consensus.cc:504] T 00a1881267a14ba8b78c81040f17a014 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:46:31.333155 25913 consensus_meta.cc:287] T 0acc47ce21c3422698d1ef7277c3ed63 P c0f38c1908414566aa9115bb2cbdd85d: Time spent flushing consensus metadata: real 0.893s user 0.001s sys 0.009s
I20260809 15:46:31.382033 25873 raft_consensus.cc:482] T 297c82ed906d4b48b184985eed3de601 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:31.422606 25948 raft_consensus.cc:3037] T f7715d71dbac447b9351fc5eb36c5e80 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Advancing to term 3
W20260809 15:46:31.429997 25849 consensus_meta.cc:287] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 65c36d164b0540f099d592f38307ee6c: Time spent flushing consensus metadata: real 1.930s user 0.005s sys 0.005s
I20260809 15:46:31.179289 25793 leader_election.cc:288] T f9e7cadcd24d45fcbfa84209225e8f2d P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:31.446902 23394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3622f6e3e3641cea1960e3a31498bb7" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:31.382373 25900 raft_consensus.cc:3037] T 8e4ffcf83c234ae7910ddef3ec0b4bda P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:31.424434 25937 raft_consensus.cc:2741] T 294d4bcc1e7a477cb56455581277a60f P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20260809 15:46:31.439904 25639 raft_consensus.cc:504] T 0c48a3049897452383257aa757834fb5 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:31.454154 24455 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0741ab44d99439abc3aa3bdd27ee160" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:31.454857 23393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4564dfc6d4094c1b9240b367ccecf3b1" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:31.455456 23396 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55320 (request call id 1086) took 1166 ms. Trace:
I20260809 15:46:31.455582 25943 raft_consensus.cc:482] T d3be1e356b254d22b995be32f1c12f6f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:46:31.447306 25746 raft_consensus.cc:482] T 3296ef50be254fc29cfc77d22a8f7a23 P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
W20260809 15:46:31.457652 25962 log.cc:924] Time spent T 297c82ed906d4b48b184985eed3de601 P c0f38c1908414566aa9115bb2cbdd85d: Append to log took a long time: real 0.518s user 0.002s sys 0.000s
W20260809 15:46:31.399433 25960 log.cc:924] Time spent T 472816994d814cb198b69e5248697440 P 65c36d164b0540f099d592f38307ee6c: Append to log took a long time: real 0.408s user 0.001s sys 0.000s
W20260809 15:46:31.461707 23468 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.68:34689, user_credentials={real_user=slave}} blocked reactor thread for 465261us
I20260809 15:46:31.455029 23527 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56770 (request call id 1153) took 1745 ms. Trace:
I20260809 15:46:31.381197 25947 raft_consensus.cc:504] T 0a51c0baaa6e432997c9d3552af4f254 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:31.404505 25882 consensus_queue.cc:1035] T d3e94b7e50754a81b3d72a07476cb7b9 P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.048s
I20260809 15:46:31.177239 25695 raft_consensus.cc:504] T ae6d2c08a7a94f58ac8dd698e3fd456c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:31.460214 25899 raft_consensus.cc:979] T d3be1e356b254d22b995be32f1c12f6f P c0f38c1908414566aa9115bb2cbdd85d: : Instructing follower 67b3aeadfa7a432abe8af4d292e04ab9 to start an election
I20260809 15:46:31.133625 25477 raft_consensus.cc:504] T 763e58d6ba644eea830a6c1664d39c9b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:46:31.558463 24410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 135487us
W20260809 15:46:31.691931 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 288363us
I20260809 15:46:31.693336 23228 leader_election.cc:302] T df20671a7b874dbb98c0ef28371c1308 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:46:31.694509 23394 raft_consensus.cc:2379] T a3622f6e3e3641cea1960e3a31498bb7 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 in current term 3: Already voted for candidate c0f38c1908414566aa9115bb2cbdd85d in this term.
I20260809 15:46:31.444829 25864 raft_consensus.cc:482] T ba56725e42f74a1baa4c04d9660727f9 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:31.695333 23517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "763e58d6ba644eea830a6c1664d39c9b" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:31.695434 24410 leader_election.cc:302] T 294d4bcc1e7a477cb56455581277a60f P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c; no voters:
I20260809 15:46:31.691401 23396 rpcz_store.cc:276] 0809 15:46:30.288889 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:30.570428 (+281539us) service_pool.cc:227] Handling call
0809 15:46:31.455212 (+884784us) spinlock_profiling.cc:243] Waited 565 ms on lock 0x7b540009ecd4. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e92242ae 00007f34ee161c27 00007f34e7849c04 00007f34e7849b92 00007f34e7849ac1 00007f34e7849a3c 00007f34e7848cf2 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:46:31.455400 (+ 188us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1299811584}
I20260809 15:46:31.699187 25959 raft_consensus.cc:1067] T d3be1e356b254d22b995be32f1c12f6f P c0f38c1908414566aa9115bb2cbdd85d [term 1 LEADER]: Signalling peer 67b3aeadfa7a432abe8af4d292e04ab9 to start an election
I20260809 15:46:31.699601 25979 consensus_queue.cc:1035] T d3e94b7e50754a81b3d72a07476cb7b9 P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20260809 15:46:31.463338 25587 raft_consensus.cc:504] T ba56725e42f74a1baa4c04d9660727f9 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:31.700287 23522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "213543cdbe814c89a17a963469922c1b" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:31.700619 25639 leader_election.cc:288] T 0c48a3049897452383257aa757834fb5 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:31.701165 23519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e17dcb5efe45b8b6a8c42da03b4819" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:31.702389 23519 raft_consensus.cc:2454] T 28e17dcb5efe45b8b6a8c42da03b4819 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 2.
W20260809 15:46:31.703337 23404 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55320 (request call id 1078) took 1700 ms (client timeout 1937 ms). Trace:
I20260809 15:46:31.697343 23527 rpcz_store.cc:276] 0809 15:46:29.709243 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:29.709451 (+ 208us) service_pool.cc:227] Handling call
0809 15:46:31.454842 (+1745391us) spinlock_profiling.cc:243] Waited 1.24 s on lock 0x7b54000f7dd4. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e92242ae 00007f34ee161c27 00007f34e7849c04 00007f34e7849b92 00007f34e7849ac1 00007f34e7849a3c 00007f34e7848cf2 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:46:31.454974 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2861566208}
I20260809 15:46:31.439013 23470 leader_election.cc:302] T a3622f6e3e3641cea1960e3a31498bb7 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
W20260809 15:46:31.703732 23404 rpcz_store.cc:269] 0809 15:46:30.002843 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:30.003438 (+ 595us) service_pool.cc:227] Handling call
0809 15:46:31.703153 (+1699715us) spinlock_profiling.cc:243] Waited 615 ms on lock 0x7b540009fbd4. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e92242ae 00007f34ee161c27 00007f34e7849c04 00007f34e7849b92 00007f34e7849ac1 00007f34e7849a3c 00007f34e7848cf2 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:46:31.703287 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1414228736}
W20260809 15:46:31.599781 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 157556us
I20260809 15:46:31.704728 23522 raft_consensus.cc:2454] T 213543cdbe814c89a17a963469922c1b P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 2.
I20260809 15:46:31.608134 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e4ffcf83c234ae7910ddef3ec0b4bda" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
W20260809 15:46:31.704485 23470 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 265534us
I20260809 15:46:31.702025 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16ca412a0dae4feebe85dd01bf1a9173" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:31.445560 25857 leader_election.cc:288] T 07fcabb569414bddbd39df6cedd24e64 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:31.707906 23273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d880de657e544468c82157769622a26" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:31.488420 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2424fd9bd7f14e508fd106cbfda786a4" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
W20260809 15:46:31.665108 25963 log.cc:924] Time spent T 472816994d814cb198b69e5248697440 P 61e69695258f48baa984c9d395e8ce3f: Append to log took a long time: real 0.266s user 0.000s sys 0.001s
W20260809 15:46:31.604234 25772 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.646s user 0.002s sys 0.014s
W20260809 15:46:31.672226 25861 consensus_meta.cc:287] T 4564dfc6d4094c1b9240b367ccecf3b1 P 65c36d164b0540f099d592f38307ee6c: Time spent flushing consensus metadata: real 0.736s user 0.011s sys 0.010s
I20260809 15:46:31.569990 24463 raft_consensus.cc:2354] T 7660dfa2ef694927b8eda4100a2bcd9b P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c for earlier term 3. Current term is 4.
I20260809 15:46:31.716753 23270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba56725e42f74a1baa4c04d9660727f9" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:31.727245 23282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a88421dba80d4d3388aa5eb62e23853b" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:31.677495 25905 leader_election.cc:288] T 2db057fd10e149a9895dff6ef8c9f038 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:31.467648 25868 leader_election.cc:288] T 6ae14a74a26f434db08c652e097385c8 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:31.467403 25507 leader_election.cc:288] T a3622f6e3e3641cea1960e3a31498bb7 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:31.467140 25869 raft_consensus.cc:504] T cef3e7277d7d4760824a3436ed4a847d P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:31.717108 23394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0acc47ce21c3422698d1ef7277c3ed63" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:31.647423 25933 leader_election.cc:288] T cee02fff2c9c48a19f43eb358a89e7b0 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:31.489940 25875 leader_election.cc:288] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 4 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:31.721477 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df20671a7b874dbb98c0ef28371c1308" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:31.722188 23283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7715d71dbac447b9351fc5eb36c5e80" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:31.722978 23526 raft_consensus.cc:1261] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260809 15:46:31.725581 23472 leader_election.cc:302] T 28e17dcb5efe45b8b6a8c42da03b4819 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters: 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:31.706915 25913 raft_consensus.cc:2726] T 0acc47ce21c3422698d1ef7277c3ed63 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 15:46:31.727638 23270 raft_consensus.cc:3037] T ba56725e42f74a1baa4c04d9660727f9 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:31.711830 23351 leader_election.cc:302] T 73f6f44c259547f1b05a3a434616fa54 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c
I20260809 15:46:31.464219 25785 leader_election.cc:288] T d0721a4403a440f383318a3cea9d1442 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:31.728984 23285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0da4d5283c8a46878bd39b85e5c71f66" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:31.770229 23285 raft_consensus.cc:2379] T 0da4d5283c8a46878bd39b85e5c71f66 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c in current term 3: Already voted for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in this term.
I20260809 15:46:31.730798 23282 raft_consensus.cc:3037] T a88421dba80d4d3388aa5eb62e23853b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:31.701238 25946 raft_consensus.cc:482] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:46:31.709257 25941 raft_consensus.cc:482] T 213543cdbe814c89a17a963469922c1b P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:46:31.709381 25885 raft_consensus.cc:482] T 2424fd9bd7f14e508fd106cbfda786a4 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:46:31.701586 25573 raft_consensus.cc:2781] T 0acc47ce21c3422698d1ef7277c3ed63 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260809 15:46:31.731251 23396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "538a228ddfb54bc6a36528d5ff07597b" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:31.619997 25478 leader_election.cc:288] T 7660dfa2ef694927b8eda4100a2bcd9b P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 4 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:31.704357 25953 raft_consensus.cc:482] T c5c2e425a74b4e64ae70b5def750434c P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting leader election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
I20260809 15:46:31.620674 25927 leader_election.cc:288] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:31.763492 25507 raft_consensus.cc:482] T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:31.800503 23283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "294d4bcc1e7a477cb56455581277a60f" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:31.801146 23283 raft_consensus.cc:2454] T 294d4bcc1e7a477cb56455581277a60f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 3.
I20260809 15:46:31.710748 25639 raft_consensus.cc:2781] T a3622f6e3e3641cea1960e3a31498bb7 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Leader election won for term 3
I20260809 15:46:31.452982 25898 raft_consensus.cc:686] T e14d118b2eaa42aa8a7cd4025aba7128 P 61e69695258f48baa984c9d395e8ce3f [term 4 LEADER]: Becoming Leader. State: Replica: 61e69695258f48baa984c9d395e8ce3f, State: Running, Role: LEADER
I20260809 15:46:31.445092 24448 rpcz_store.cc:276] 0809 15:46:29.878185 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:29.881500 (+ 3315us) service_pool.cc:227] Handling call
0809 15:46:29.944103 (+ 62603us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:31.273714 (+1329611us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:31.320168 (+ 46454us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:46:31.460615 25650 consensus_queue.cc:1035] T cef3e7277d7d4760824a3436ed4a847d P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.263s
I20260809 15:46:31.719408 25919 raft_consensus.cc:2781] T df20671a7b874dbb98c0ef28371c1308 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260809 15:46:31.717443 25652 raft_consensus.cc:482] T 00a1881267a14ba8b78c81040f17a014 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
W20260809 15:46:31.720541 25772 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.762s user 0.004s sys 0.014s
I20260809 15:46:31.623100 24453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "563daf42d7784db195ea99c3cb21b370" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:31.464074 25965 consensus_queue.cc:1035] T 16ca412a0dae4feebe85dd01bf1a9173 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.320s
I20260809 15:46:31.782732 23276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b30f28311964a0b9149a50495e83d8f" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:31.813357 23404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3aa6a9a52804df0ba9b0ca3c82f38c3" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:31.523464 25731 raft_consensus.cc:504] T 16ca412a0dae4feebe85dd01bf1a9173 P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:31.727885 25929 leader_election.cc:288] T f9e7cadcd24d45fcbfa84209225e8f2d P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:31.702335 25921 raft_consensus.cc:504] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:31.707870 25864 raft_consensus.cc:504] T ba56725e42f74a1baa4c04d9660727f9 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:31.834133 24453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3aa6a9a52804df0ba9b0ca3c82f38c3" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:31.711385 25873 raft_consensus.cc:504] T 297c82ed906d4b48b184985eed3de601 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:31.707465 25746 raft_consensus.cc:504] T 3296ef50be254fc29cfc77d22a8f7a23 P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:31.765821 23528 raft_consensus.cc:2379] T 538a228ddfb54bc6a36528d5ff07597b P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 in current term 4: Already voted for candidate 65c36d164b0540f099d592f38307ee6c in this term.
I20260809 15:46:31.779614 23396 raft_consensus.cc:2454] T 538a228ddfb54bc6a36528d5ff07597b P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 3.
I20260809 15:46:31.741947 25980 raft_consensus.cc:2781] T 294d4bcc1e7a477cb56455581277a60f P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Leader election won for term 4
I20260809 15:46:31.750519 25587 leader_election.cc:288] T ba56725e42f74a1baa4c04d9660727f9 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:31.449864 25879 raft_consensus.cc:504] T 30c1740ef6874df0a332fbd6fd4b03ce P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:31.719807 25951 raft_consensus.cc:504] T 907bacfa79d04a55a68c8dbe60056462 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:31.768822 25868 raft_consensus.cc:2781] T 28e17dcb5efe45b8b6a8c42da03b4819 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260809 15:46:31.802695 23529 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56766 (request call id 1668) took 1388 ms. Trace:
W20260809 15:46:31.467020 24468 rpcz_store.cc:269] 0809 15:46:29.205252 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:29.212080 (+ 6828us) service_pool.cc:227] Handling call
0809 15:46:31.130535 (+1918455us) spinlock_profiling.cc:243] Waited 739 ms on lock 0x7b5400510754. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e92242ae 00007f34ee161c27 00007f34e7849c04 00007f34e7849b92 00007f34e7849ac1 00007f34e7849a3c 00007f34e7848cf2 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:46:31.138425 (+ 7890us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1699631488}
I20260809 15:46:31.828315 24455 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3296ef50be254fc29cfc77d22a8f7a23" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:31.727658 25861 raft_consensus.cc:504] T 4564dfc6d4094c1b9240b367ccecf3b1 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:31.806113 23274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ae14a74a26f434db08c652e097385c8" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:31.807163 23400 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55320 (request call id 1082) took 1519 ms. Trace:
W20260809 15:46:31.806157 24410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 93071.2us
I20260809 15:46:31.855479 23274 raft_consensus.cc:2454] T 6ae14a74a26f434db08c652e097385c8 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 3.
I20260809 15:46:31.780367 25941 raft_consensus.cc:3037] T 213543cdbe814c89a17a963469922c1b P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:31.763617 23286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "953eea268e17491ca76f3a555860a512" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:31.711216 25849 raft_consensus.cc:504] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:31.865044 25864 leader_election.cc:288] T ba56725e42f74a1baa4c04d9660727f9 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:31.783802 25885 raft_consensus.cc:3037] T 2424fd9bd7f14e508fd106cbfda786a4 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:46:31.779592 25946 raft_consensus.cc:3037] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:46:31.813745 23283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0721a4403a440f383318a3cea9d1442" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:31.842265 25477 leader_election.cc:288] T 763e58d6ba644eea830a6c1664d39c9b P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:31.783614 25840 leader_election.cc:288] T 953eea268e17491ca76f3a555860a512 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:31.836362 25816 consensus_queue.cc:1035] T cef3e7277d7d4760824a3436ed4a847d P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.021s
W20260809 15:46:31.766165 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 54419.8us
I20260809 15:46:31.810726 25507 raft_consensus.cc:504] T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:31.839994 25982 raft_consensus.cc:482] T d3be1e356b254d22b995be32f1c12f6f P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:46:31.860846 25965 consensus_queue.cc:1035] T 16ca412a0dae4feebe85dd01bf1a9173 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.031s
I20260809 15:46:31.813834 23270 raft_consensus.cc:2454] T ba56725e42f74a1baa4c04d9660727f9 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 3.
I20260809 15:46:31.809818 25919 raft_consensus.cc:482] T df20671a7b874dbb98c0ef28371c1308 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:46:31.829855 23277 raft_consensus.cc:1226] T 16ca412a0dae4feebe85dd01bf1a9173 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Rejecting Update request from peer 61e69695258f48baa984c9d395e8ce3f for earlier term 2. Current term is 3. Ops: []
I20260809 15:46:31.815793 25956 raft_consensus.cc:2781] T 73f6f44c259547f1b05a3a434616fa54 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader election won for term 3
I20260809 15:46:31.834244 25947 leader_election.cc:288] T 0a51c0baaa6e432997c9d3552af4f254 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:31.775143 23397 raft_consensus.cc:2454] T 0acc47ce21c3422698d1ef7277c3ed63 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 2.
I20260809 15:46:31.463614 25967 raft_consensus.cc:504] T d2d4f749fcfd4fb3bfb8115af3471598 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:31.800053 25875 consensus_queue.cc:1046] T 3924b648461e463b9d7925e5671d68ae P 65c36d164b0540f099d592f38307ee6c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.030s
I20260809 15:46:31.838876 23396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ae14a74a26f434db08c652e097385c8" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:31.840322 24472 raft_consensus.cc:2454] T 16ca412a0dae4feebe85dd01bf1a9173 P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 3.
I20260809 15:46:31.819949 25478 raft_consensus.cc:482] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:31.844012 25868 raft_consensus.cc:482] T 28e17dcb5efe45b8b6a8c42da03b4819 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Starting leader election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
I20260809 15:46:31.817514 25639 raft_consensus.cc:686] T a3622f6e3e3641cea1960e3a31498bb7 P 65c36d164b0540f099d592f38307ee6c [term 3 LEADER]: Becoming Leader. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:46:31.828975 25791 leader_election.cc:288] T a88421dba80d4d3388aa5eb62e23853b P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:31.801402 23285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e14d118b2eaa42aa8a7cd4025aba7128" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:31.850181 25879 leader_election.cc:288] T 30c1740ef6874df0a332fbd6fd4b03ce P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:31.851135 25980 raft_consensus.cc:686] T 294d4bcc1e7a477cb56455581277a60f P 61e69695258f48baa984c9d395e8ce3f [term 4 LEADER]: Becoming Leader. State: Replica: 61e69695258f48baa984c9d395e8ce3f, State: Running, Role: LEADER
I20260809 15:46:31.718394 25943 raft_consensus.cc:504] T d3be1e356b254d22b995be32f1c12f6f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:31.829314 23393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0da4d5283c8a46878bd39b85e5c71f66" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:31.844588 23529 rpcz_store.cc:276] 0809 15:46:30.404042 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:30.405328 (+ 1286us) service_pool.cc:227] Handling call
0809 15:46:30.405602 (+ 274us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 15:46:30.431572 (+ 25970us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:46:30.431603 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 15:46:30.456576 (+ 24973us) log.cc:842] Serialized 26 byte log entry
0809 15:46:30.919072 (+462496us) raft_consensus.cc:1631] Marking committed up to 4
0809 15:46:30.919108 (+ 36us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:31.179803 (+260695us) spinlock_profiling.cc:243] Waited 25.6 ms on lock 0x7b54001df6d4. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e9220f51 00007f34e921f99e 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772 00007f34e764a94c 00007f34e7649d86
0809 15:46:31.179851 (+ 48us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 15:46:31.779985 (+600134us) raft_consensus.cc:1671] finished
0809 15:46:31.780004 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:31.792384 (+ 12380us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":409,"raft.run_cpu_time_us":110,"raft.run_wall_time_us":110,"spinlock_wait_cycles":58931456,"thread_start_us":430965,"threads_started":1,"wal-append.queue_time_us":435564}
I20260809 15:46:31.839370 25785 raft_consensus.cc:482] T 0c29937a80724ddb9d07a27534a146f3 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:31.836792 25695 leader_election.cc:288] T ae6d2c08a7a94f58ac8dd698e3fd456c P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:31.828274 25789 raft_consensus.cc:504] T df20671a7b874dbb98c0ef28371c1308 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:31.852178 25863 leader_election.cc:288] T 538a228ddfb54bc6a36528d5ff07597b P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 4 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:31.859184 23274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5712f6b6fffc4cc5a8ddfbe63e4097df" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:31.837391 23276 raft_consensus.cc:2454] T 4b30f28311964a0b9149a50495e83d8f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 1.
I20260809 15:46:31.860086 24410 leader_election.cc:302] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:31.855419 23400 rpcz_store.cc:276] 0809 15:46:30.287818 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:30.287951 (+ 133us) service_pool.cc:227] Handling call
0809 15:46:30.406670 (+118719us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 15:46:30.691595 (+284925us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:46:30.691627 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0809 15:46:30.796997 (+105370us) log.cc:842] Serialized 52 byte log entry
0809 15:46:30.916145 (+119148us) raft_consensus.cc:1631] Marking committed up to 1
0809 15:46:30.916180 (+ 35us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:30.916257 (+ 77us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 15:46:31.456924 (+540667us) spinlock_profiling.cc:243] Waited 172 ms on lock 0x7b44000232cc. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e613cf11 00007f34e620aa95 00007f34e620b4b2 00007f34e8eb3511 00007f34e8eaee1a 00007f34e8ea2bee 00007f34e8ea1f36 00007f34e8ea1b58 00007f34e8eab551 00007f34e8eab509 00007f34e8eab499
0809 15:46:31.806889 (+349965us) raft_consensus.cc:1671] finished
0809 15:46:31.806910 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:31.807106 (+ 196us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":74177,"raft.queue_time_us":167592,"raft.run_cpu_time_us":101,"raft.run_wall_time_us":101,"spinlock_wait_cycles":395186048,"thread_start_us":319224,"threads_started":2,"wal-append.queue_time_us":142445}
I20260809 15:46:31.817813 25652 raft_consensus.cc:504] T 00a1881267a14ba8b78c81040f17a014 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:31.866609 23283 raft_consensus.cc:2396] T d0721a4403a440f383318a3cea9d1442 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260809 15:46:31.782614 25573 raft_consensus.cc:482] T 0acc47ce21c3422698d1ef7277c3ed63 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:46:31.867487 23521 raft_consensus.cc:3037] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:31.860078 23404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba56725e42f74a1baa4c04d9660727f9" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:31.852382 24448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d880de657e544468c82157769622a26" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:31.764712 23345 leader_election.cc:302] T e71a05ba1d4d42bd95c3f85f9313399e P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 61e69695258f48baa984c9d395e8ce3f, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:31.877461 23282 raft_consensus.cc:2454] T a88421dba80d4d3388aa5eb62e23853b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 3.
I20260809 15:46:31.791697 25953 raft_consensus.cc:3037] T c5c2e425a74b4e64ae70b5def750434c P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:31.867113 23298 tablet_copy_service.cc:342] P 67b3aeadfa7a432abe8af4d292e04ab9: Request end of tablet copy session 61e69695258f48baa984c9d395e8ce3f-0acc47ce21c3422698d1ef7277c3ed63 received from {username='slave'} at 127.0.0.1:55434
I20260809 15:46:31.866458 25898 consensus_queue.cc:237] T e14d118b2eaa42aa8a7cd4025aba7128 P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
W20260809 15:46:31.960368 23353 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 518612us
I20260809 15:46:32.185555 23404 raft_consensus.cc:3037] T ba56725e42f74a1baa4c04d9660727f9 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:32.190938 23285 raft_consensus.cc:2379] T e14d118b2eaa42aa8a7cd4025aba7128 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 61e69695258f48baa984c9d395e8ce3f in current term 4: Already voted for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in this term.
W20260809 15:46:32.186087 23345 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.68:34689, user_credentials={real_user=slave}} blocked reactor thread for 421425us
I20260809 15:46:32.195015 23345 leader_election.cc:302] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:46:31.866349 24451 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38724 (request call id 408) took 1432 ms. Trace:
I20260809 15:46:32.198549 23282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e4ffcf83c234ae7910ddef3ec0b4bda" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:32.200688 23282 raft_consensus.cc:2454] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 3.
I20260809 15:46:32.207805 23282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e14d118b2eaa42aa8a7cd4025aba7128" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:32.210816 23282 raft_consensus.cc:2379] T e14d118b2eaa42aa8a7cd4025aba7128 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 61e69695258f48baa984c9d395e8ce3f in current term 4: Already voted for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in this term.
I20260809 15:46:32.212953 23282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "294d4bcc1e7a477cb56455581277a60f" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
W20260809 15:46:32.157378 25852 consensus_meta.cc:287] T 2424fd9bd7f14e508fd106cbfda786a4 P 65c36d164b0540f099d592f38307ee6c: Time spent flushing consensus metadata: real 1.204s user 0.009s sys 0.002s
I20260809 15:46:32.218487 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "294d4bcc1e7a477cb56455581277a60f" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:32.201722 23270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e4ffcf83c234ae7910ddef3ec0b4bda" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
W20260809 15:46:32.197788 23228 connection.cc:232] RPC call timeout handler was delayed by -1.0823881579563022s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1587s for a response
I20260809 15:46:32.204521 23396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9e7cadcd24d45fcbfa84209225e8f2d" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:31.871021 23520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7660dfa2ef694927b8eda4100a2bcd9b" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
W20260809 15:46:32.184916 25851 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.221s user 0.010s sys 0.022s
I20260809 15:46:32.219328 23270 raft_consensus.cc:2454] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 3.
I20260809 15:46:32.216620 23282 raft_consensus.cc:2454] T 294d4bcc1e7a477cb56455581277a60f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 3.
I20260809 15:46:32.221153 23517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0741ab44d99439abc3aa3bdd27ee160" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:31.899621 24453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee02fff2c9c48a19f43eb358a89e7b0" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:31.917693 25921 leader_election.cc:288] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:32.222113 23526 raft_consensus.cc:2354] T 294d4bcc1e7a477cb56455581277a60f P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 3. Current term is 4.
I20260809 15:46:32.221975 23283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c48a3049897452383257aa757834fb5" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:32.227029 23400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5c2e425a74b4e64ae70b5def750434c" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:32.223112 23396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "953eea268e17491ca76f3a555860a512" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:32.184546 25882 raft_consensus.cc:482] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:32.158139 25746 leader_election.cc:288] T 3296ef50be254fc29cfc77d22a8f7a23 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:32.226167 23517 raft_consensus.cc:2454] T c0741ab44d99439abc3aa3bdd27ee160 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 2.
I20260809 15:46:32.156265 25849 leader_election.cc:288] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 2 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:32.194350 25868 raft_consensus.cc:3037] T 28e17dcb5efe45b8b6a8c42da03b4819 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:32.222011 23520 raft_consensus.cc:3037] T 7660dfa2ef694927b8eda4100a2bcd9b P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Advancing to term 4
I20260809 15:46:31.901422 25873 leader_election.cc:288] T 297c82ed906d4b48b184985eed3de601 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:31.896062 25507 leader_election.cc:288] T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:32.215135 23393 raft_consensus.cc:2454] T 0da4d5283c8a46878bd39b85e5c71f66 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 2.
I20260809 15:46:32.191968 25573 raft_consensus.cc:3037] T 0acc47ce21c3422698d1ef7277c3ed63 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:32.121497 25927 raft_consensus.cc:482] T 05ad7d8fdca64e3da527280ca5be9280 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:46:32.198513 25675 raft_consensus.cc:2726] T e71a05ba1d4d42bd95c3f85f9313399e P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 15:46:32.227865 23396 raft_consensus.cc:2454] T 953eea268e17491ca76f3a555860a512 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 2.
I20260809 15:46:32.267854 23396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7660dfa2ef694927b8eda4100a2bcd9b" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:32.267766 25980 consensus_queue.cc:237] T 294d4bcc1e7a477cb56455581277a60f P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:32.268484 23396 raft_consensus.cc:3037] T 7660dfa2ef694927b8eda4100a2bcd9b P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:46:32.268798 26005 raft_consensus.cc:3032] T 3924b648461e463b9d7925e5671d68ae P 65c36d164b0540f099d592f38307ee6c [term 3 LEADER]: Stepping down as leader of term 3
I20260809 15:46:32.269119 26005 raft_consensus.cc:729] T 3924b648461e463b9d7925e5671d68ae P 65c36d164b0540f099d592f38307ee6c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:46:32.234854 23277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e14d118b2eaa42aa8a7cd4025aba7128" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:32.227192 23228 leader_election.cc:302] T a3622f6e3e3641cea1960e3a31498bb7 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:32.270473 23519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ced718c1a5c4da79cb8c1ebd1435b87" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:32.271139 23519 raft_consensus.cc:2454] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 1.
I20260809 15:46:32.234325 25956 raft_consensus.cc:686] T 73f6f44c259547f1b05a3a434616fa54 P c0f38c1908414566aa9115bb2cbdd85d [term 3 LEADER]: Becoming Leader. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:46:32.199504 25785 raft_consensus.cc:504] T 0c29937a80724ddb9d07a27534a146f3 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:32.244027 23468 leader_election.cc:302] T 3d880de657e544468c82157769622a26 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 61e69695258f48baa984c9d395e8ce3f, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:32.204598 25929 raft_consensus.cc:482] T 998c6cee72cf40228c6dba064c0db2da P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:32.250512 23521 raft_consensus.cc:2454] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 3.
I20260809 15:46:32.205515 23298 tablet_copy_service.cc:433] P 67b3aeadfa7a432abe8af4d292e04ab9: ending tablet copy session 61e69695258f48baa984c9d395e8ce3f-0acc47ce21c3422698d1ef7277c3ed63 on tablet 0acc47ce21c3422698d1ef7277c3ed63 with peer 61e69695258f48baa984c9d395e8ce3f
I20260809 15:46:32.198748 25478 raft_consensus.cc:504] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:32.200820 25919 raft_consensus.cc:3037] T df20671a7b874dbb98c0ef28371c1308 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:32.258569 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "213543cdbe814c89a17a963469922c1b" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:32.227792 25852 raft_consensus.cc:504] T 2424fd9bd7f14e508fd106cbfda786a4 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:32.292546 23517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0da4d5283c8a46878bd39b85e5c71f66" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:32.293136 23517 raft_consensus.cc:3037] T 0da4d5283c8a46878bd39b85e5c71f66 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:31.927858 25931 raft_consensus.cc:504] T df20671a7b874dbb98c0ef28371c1308 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:32.226090 23286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "294d4bcc1e7a477cb56455581277a60f" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:32.257189 25951 leader_election.cc:288] T 907bacfa79d04a55a68c8dbe60056462 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:32.234618 25899 raft_consensus.cc:2781] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader election won for term 3
W20260809 15:46:32.225554 25851 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.272s user 0.010s sys 0.022s
W20260809 15:46:32.271209 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 44089.8us
I20260809 15:46:32.270437 23277 raft_consensus.cc:2379] T e14d118b2eaa42aa8a7cd4025aba7128 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 61e69695258f48baa984c9d395e8ce3f in current term 4: Already voted for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in this term.
I20260809 15:46:32.199425 25869 leader_election.cc:288] T cef3e7277d7d4760824a3436ed4a847d P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:32.247347 25875 raft_consensus.cc:482] T d3e94b7e50754a81b3d72a07476cb7b9 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:32.278067 25965 raft_consensus.cc:2726] T 3d880de657e544468c82157769622a26 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 15:46:32.243456 25861 leader_election.cc:288] T 4564dfc6d4094c1b9240b367ccecf3b1 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:32.237033 25791 raft_consensus.cc:482] T c0741ab44d99439abc3aa3bdd27ee160 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
W20260809 15:46:32.294488 24410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 434464us
I20260809 15:46:32.318291 25791 raft_consensus.cc:504] T c0741ab44d99439abc3aa3bdd27ee160 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:32.269893 26005 consensus_queue.cc:260] T 3924b648461e463b9d7925e5671d68ae P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:32.303135 23394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b30f28311964a0b9149a50495e83d8f" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:32.319733 25959 raft_consensus.cc:482] T cef3e7277d7d4760824a3436ed4a847d P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:32.320703 23394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3296ef50be254fc29cfc77d22a8f7a23" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:32.220176 25982 raft_consensus.cc:504] T d3be1e356b254d22b995be32f1c12f6f P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:32.243680 25882 raft_consensus.cc:504] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:32.309135 23472 leader_election.cc:302] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:32.275775 23276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "538a228ddfb54bc6a36528d5ff07597b" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:32.378516 23276 raft_consensus.cc:3037] T 538a228ddfb54bc6a36528d5ff07597b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:46:32.276772 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16ca412a0dae4feebe85dd01bf1a9173" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:32.380132 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee02fff2c9c48a19f43eb358a89e7b0" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:32.277971 25873 raft_consensus.cc:2726] T a3622f6e3e3641cea1960e3a31498bb7 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 15:46:32.321851 23402 raft_consensus.cc:2379] T df20671a7b874dbb98c0ef28371c1308 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 in current term 3: Already voted for candidate c0f38c1908414566aa9115bb2cbdd85d in this term.
I20260809 15:46:32.305223 25992 raft_consensus.cc:2726] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 15:46:32.321307 23394 raft_consensus.cc:2454] T 3296ef50be254fc29cfc77d22a8f7a23 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 1.
I20260809 15:46:32.273200 25927 raft_consensus.cc:504] T 05ad7d8fdca64e3da527280ca5be9280 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:46:32.377961 23472 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 68927.5us
I20260809 15:46:32.386986 23393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0721a4403a440f383318a3cea9d1442" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
W20260809 15:46:32.389113 23395 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55320 (request call id 1081) took 2101 ms (client timeout 1647 ms). Trace:
I20260809 15:46:32.308153 24448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f9d6565aa4c45a3b3f01210d0eb71f7" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:32.310472 23286 raft_consensus.cc:3037] T 294d4bcc1e7a477cb56455581277a60f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:46:32.312254 25851 raft_consensus.cc:686] T 998c6cee72cf40228c6dba064c0db2da P 65c36d164b0540f099d592f38307ee6c [term 5 LEADER]: Becoming Leader. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:46:32.285459 25929 raft_consensus.cc:504] T 998c6cee72cf40228c6dba064c0db2da P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:32.316826 25899 raft_consensus.cc:686] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d [term 3 LEADER]: Becoming Leader. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:46:32.403501 24410 leader_election.cc:302] T 3296ef50be254fc29cfc77d22a8f7a23 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:46:32.320279 25959 raft_consensus.cc:504] T cef3e7277d7d4760824a3436ed4a847d P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:32.409284 25990 raft_consensus.cc:2781] T 3296ef50be254fc29cfc77d22a8f7a23 P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260809 15:46:32.409654 25990 raft_consensus.cc:482] T 3296ef50be254fc29cfc77d22a8f7a23 P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Starting leader election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
I20260809 15:46:32.410001 25990 raft_consensus.cc:3037] T 3296ef50be254fc29cfc77d22a8f7a23 P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Advancing to term 2
W20260809 15:46:32.411679 23470 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 70918.1us
I20260809 15:46:32.398061 25965 raft_consensus.cc:2726] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 15:46:32.400715 24453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a823d35f4f348d1b52ba9e6d80b00d2" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f"
I20260809 15:46:32.410147 25899 consensus_queue.cc:237] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:32.307448 25639 consensus_queue.cc:237] T a3622f6e3e3641cea1960e3a31498bb7 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:32.441684 23353 leader_election.cc:302] T 28e17dcb5efe45b8b6a8c42da03b4819 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters: 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:32.439850 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "953eea268e17491ca76f3a555860a512" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:32.321692 25875 raft_consensus.cc:504] T d3e94b7e50754a81b3d72a07476cb7b9 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:32.405355 24448 raft_consensus.cc:2454] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 1.
I20260809 15:46:32.460011 23526 raft_consensus.cc:2454] T 953eea268e17491ca76f3a555860a512 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 2.
W20260809 15:46:32.440052 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 129935us
I20260809 15:46:32.427356 23470 leader_election.cc:302] T 6ae14a74a26f434db08c652e097385c8 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters: c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:32.466836 23228 leader_election.cc:302] T 4564dfc6d4094c1b9240b367ccecf3b1 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:32.467730 23470 leader_election.cc:302] T a88421dba80d4d3388aa5eb62e23853b P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
W20260809 15:46:32.389652 23395 rpcz_store.cc:269] 0809 15:46:30.287472 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:30.291282 (+ 3810us) service_pool.cc:227] Handling call
0809 15:46:32.388777 (+2097495us) spinlock_profiling.cc:243] Waited 504 ms on lock 0x7b5400192b54. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e92242ae 00007f34ee161c27 00007f34e7849c04 00007f34e7849b92 00007f34e7849ac1 00007f34e7849a3c 00007f34e7848cf2 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:46:32.388994 (+ 217us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1159337216}
I20260809 15:46:32.470386 23228 leader_election.cc:302] T 538a228ddfb54bc6a36528d5ff07597b P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:46:32.475160 23517 raft_consensus.cc:2454] T 0da4d5283c8a46878bd39b85e5c71f66 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 3.
I20260809 15:46:32.479842 23394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472816994d814cb198b69e5248697440" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
W20260809 15:46:32.440003 25998 log.cc:924] Time spent T 297c82ed906d4b48b184985eed3de601 P 65c36d164b0540f099d592f38307ee6c: Append to log took a long time: real 0.139s user 0.000s sys 0.000s
I20260809 15:46:32.440120 26005 raft_consensus.cc:3037] T 3924b648461e463b9d7925e5671d68ae P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Advancing to term 4
W20260809 15:46:32.801656 25961 log.cc:924] Time spent T 3924b648461e463b9d7925e5671d68ae P 61e69695258f48baa984c9d395e8ce3f: Append to log took a long time: real 1.859s user 0.000s sys 0.007s
I20260809 15:46:32.435456 25478 leader_election.cc:288] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:32.463299 25808 leader_election.cc:288] T 297c82ed906d4b48b184985eed3de601 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:32.466547 25992 raft_consensus.cc:482] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:46:32.434172 25943 leader_election.cc:288] T d3be1e356b254d22b995be32f1c12f6f P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:32.202507 24451 rpcz_store.cc:276] 0809 15:46:30.379771 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:30.399213 (+ 19442us) service_pool.cc:227] Handling call
0809 15:46:30.490620 (+ 91407us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 15:46:30.672870 (+182250us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:46:30.690120 (+ 17250us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 15:46:30.891485 (+201365us) log.cc:842] Serialized 26 byte log entry
0809 15:46:31.132301 (+240816us) raft_consensus.cc:1631] Marking committed up to 4
0809 15:46:31.136888 (+ 4587us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:31.169349 (+ 32461us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 15:46:31.791968 (+622619us) raft_consensus.cc:1671] finished
0809 15:46:31.799917 (+ 7949us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:31.811102 (+ 11185us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":14961,"raft.run_cpu_time_us":101,"raft.run_wall_time_us":100,"thread_start_us":227312,"threads_started":1,"wal-append.queue_time_us":331981}
W20260809 15:46:32.618196 26000 log.cc:924] Time spent T d0721a4403a440f383318a3cea9d1442 P c0f38c1908414566aa9115bb2cbdd85d: Append to log took a long time: real 0.218s user 0.001s sys 0.000s
I20260809 15:46:32.626396 24448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3e94b7e50754a81b3d72a07476cb7b9" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:32.727381 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07fcabb569414bddbd39df6cedd24e64" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:32.440102 24453 raft_consensus.cc:2354] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c for earlier term 2. Current term is 3.
I20260809 15:46:32.782876 23277 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e71a05ba1d4d42bd95c3f85f9313399e"
dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
mode: GRACEFUL
new_leader_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
from {username='slave'} at 127.0.0.1:33468
I20260809 15:46:32.817021 23277 raft_consensus.cc:595] T e71a05ba1d4d42bd95c3f85f9313399e P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 LEADER]: Received request to transfer leadership to TS c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:32.553231 25851 consensus_queue.cc:237] T 998c6cee72cf40228c6dba064c0db2da P 65c36d164b0540f099d592f38307ee6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:46:32.785458 25946 consensus_meta.cc:287] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 61e69695258f48baa984c9d395e8ce3f: Time spent flushing consensus metadata: real 0.915s user 0.021s sys 0.033s
I20260809 15:46:32.785812 23518 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56766 (request call id 1673) took 2378 ms. Trace:
I20260809 15:46:32.790189 24455 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7660dfa2ef694927b8eda4100a2bcd9b" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:32.812477 25921 raft_consensus.cc:3037] T 4564dfc6d4094c1b9240b367ccecf3b1 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:32.791915 23402 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55332 (request call id 1175) took 1082 ms. Trace:
W20260809 15:46:32.788146 23520 consensus_meta.cc:287] T 7660dfa2ef694927b8eda4100a2bcd9b P 65c36d164b0540f099d592f38307ee6c: Time spent flushing consensus metadata: real 0.512s user 0.006s sys 0.044s
I20260809 15:46:32.840852 23520 raft_consensus.cc:2454] T 7660dfa2ef694927b8eda4100a2bcd9b P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 4.
I20260809 15:46:32.830456 25822 leader_election.cc:288] T 4564dfc6d4094c1b9240b367ccecf3b1 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:32.841993 23520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30c1740ef6874df0a332fbd6fd4b03ce" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:32.830175 25951 raft_consensus.cc:2781] T 28e17dcb5efe45b8b6a8c42da03b4819 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260809 15:46:32.843634 26001 raft_consensus.cc:482] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:32.842311 24410 leader_election.cc:302] T 7660dfa2ef694927b8eda4100a2bcd9b P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c; no voters:
I20260809 15:46:32.406566 25956 consensus_queue.cc:237] T 73f6f44c259547f1b05a3a434616fa54 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:32.465466 25873 raft_consensus.cc:482] T 0c29937a80724ddb9d07a27534a146f3 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 15:46:32.799800 23396 consensus_meta.cc:287] T 7660dfa2ef694927b8eda4100a2bcd9b P c0f38c1908414566aa9115bb2cbdd85d: Time spent flushing consensus metadata: real 0.531s user 0.007s sys 0.008s
I20260809 15:46:32.512246 25675 consensus_queue.cc:1035] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.084s
W20260809 15:46:32.800364 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 330040us
I20260809 15:46:32.849299 23228 leader_election.cc:398] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 3 election: Vote denied by peer 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553) with higher term. Message: Invalid argument: T 5712f6b6fffc4cc5a8ddfbe63e4097df P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 3. Current term is 4.
I20260809 15:46:32.850075 23468 leader_election.cc:398] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 2 election: Vote denied by peer 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689) with higher term. Message: Invalid argument: T 2a823d35f4f348d1b52ba9e6d80b00d2 P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c for earlier term 2. Current term is 3.
W20260809 15:46:32.801656 23470 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 333962us
I20260809 15:46:32.850368 23468 leader_election.cc:401] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20260809 15:46:32.850749 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae6d2c08a7a94f58ac8dd698e3fd456c" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:32.802035 23394 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55320 (request call id 1096) took 1054 ms. Trace:
I20260809 15:46:32.801213 23517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9e7cadcd24d45fcbfa84209225e8f2d" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:32.851347 23394 rpcz_store.cc:276] 0809 15:46:31.747886 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:32.383618 (+635732us) service_pool.cc:227] Handling call
0809 15:46:32.801976 (+418358us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:46:32.851732 23517 raft_consensus.cc:2454] T f9e7cadcd24d45fcbfa84209225e8f2d P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 2.
W20260809 15:46:32.805944 23404 consensus_meta.cc:287] T ba56725e42f74a1baa4c04d9660727f9 P c0f38c1908414566aa9115bb2cbdd85d: Time spent flushing consensus metadata: real 0.612s user 0.007s sys 0.006s
I20260809 15:46:32.850389 23228 leader_election.cc:398] T 294d4bcc1e7a477cb56455581277a60f P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 3 election: Vote denied by peer 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553) with higher term. Message: Invalid argument: T 294d4bcc1e7a477cb56455581277a60f P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 3. Current term is 4.
I20260809 15:46:32.852444 23404 raft_consensus.cc:2454] T ba56725e42f74a1baa4c04d9660727f9 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 3.
I20260809 15:46:32.852795 23228 leader_election.cc:401] T 294d4bcc1e7a477cb56455581277a60f P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20260809 15:46:32.853010 23404 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55320 (request call id 1097) took 1104 ms. Trace:
W20260809 15:46:32.853343 23472 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 49173.5us
I20260809 15:46:32.848896 23396 raft_consensus.cc:2454] T 7660dfa2ef694927b8eda4100a2bcd9b P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 4.
I20260809 15:46:32.853298 23404 rpcz_store.cc:276] 0809 15:46:31.748285 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:31.859863 (+111578us) service_pool.cc:227] Handling call
0809 15:46:32.852958 (+993095us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:46:32.854683 23396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ced718c1a5c4da79cb8c1ebd1435b87" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:32.800846 23223 leader_election.cc:302] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:46:32.788914 25785 leader_election.cc:288] T 0c29937a80724ddb9d07a27534a146f3 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
W20260809 15:46:32.816458 25948 consensus_meta.cc:287] T f7715d71dbac447b9351fc5eb36c5e80 P c0f38c1908414566aa9115bb2cbdd85d: Time spent flushing consensus metadata: real 1.099s user 0.008s sys 0.013s
I20260809 15:46:32.872328 25943 raft_consensus.cc:2781] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260809 15:46:32.873057 25982 leader_election.cc:288] T d3be1e356b254d22b995be32f1c12f6f P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:32.872869 25948 raft_consensus.cc:504] T f7715d71dbac447b9351fc5eb36c5e80 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:32.873028 25943 raft_consensus.cc:482] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
W20260809 15:46:32.873785 25868 consensus_meta.cc:287] T 28e17dcb5efe45b8b6a8c42da03b4819 P 65c36d164b0540f099d592f38307ee6c: Time spent flushing consensus metadata: real 0.616s user 0.010s sys 0.003s
I20260809 15:46:32.874140 25868 raft_consensus.cc:504] T 28e17dcb5efe45b8b6a8c42da03b4819 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:32.872445 23517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2db057fd10e149a9895dff6ef8c9f038" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:32.875345 25943 raft_consensus.cc:3037] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:32.801927 25772 raft_consensus.cc:2781] T 6ae14a74a26f434db08c652e097385c8 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260809 15:46:32.815801 25900 consensus_meta.cc:287] T 8e4ffcf83c234ae7910ddef3ec0b4bda P c0f38c1908414566aa9115bb2cbdd85d: Time spent flushing consensus metadata: real 1.096s user 0.003s sys 0.016s
W20260809 15:46:32.816615 25573 consensus_meta.cc:287] T 0acc47ce21c3422698d1ef7277c3ed63 P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent flushing consensus metadata: real 0.549s user 0.003s sys 0.014s
I20260809 15:46:32.790390 25967 leader_election.cc:288] T d2d4f749fcfd4fb3bfb8115af3471598 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
W20260809 15:46:32.818653 25885 consensus_meta.cc:287] T 2424fd9bd7f14e508fd106cbfda786a4 P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent flushing consensus metadata: real 0.953s user 0.012s sys 0.006s
I20260809 15:46:32.789533 25882 leader_election.cc:288] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:32.832373 24452 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38734 (request call id 194) took 2532 ms. Trace:
W20260809 15:46:32.840286 23528 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56770 (request call id 1150) took 3420 ms (client timeout 2191 ms). Trace:
I20260809 15:46:32.847417 25898 raft_consensus.cc:2781] T 7660dfa2ef694927b8eda4100a2bcd9b P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Leader election won for term 4
I20260809 15:46:32.469866 25965 raft_consensus.cc:482] T d1a134ba33ea4076bbaac0ae460c17ba P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:46:32.834111 23402 rpcz_store.cc:276] 0809 15:46:31.709410 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:31.721242 (+ 11832us) service_pool.cc:227] Handling call
0809 15:46:32.791746 (+1070504us) spinlock_profiling.cc:243] Waited 570 ms on lock 0x7b540001dbd4. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e92242ae 00007f34ee161c27 00007f34e7849c04 00007f34e7849b92 00007f34e7849ac1 00007f34e7849a3c 00007f34e7848cf2 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:46:32.791863 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1311195392}
I20260809 15:46:32.844630 24453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5712f6b6fffc4cc5a8ddfbe63e4097df" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f"
I20260809 15:46:32.850828 23470 leader_election.cc:302] T ba56725e42f74a1baa4c04d9660727f9 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:46:32.817687 23522 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56766 (request call id 1683) took 1120 ms. Trace:
I20260809 15:46:32.785676 25929 leader_election.cc:288] T 998c6cee72cf40228c6dba064c0db2da P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
W20260809 15:46:32.856539 23223 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.68:34689, user_credentials={real_user=slave}} blocked reactor thread for 55769.7us
I20260809 15:46:32.856875 23396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0da4d5283c8a46878bd39b85e5c71f66" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:32.872421 25863 raft_consensus.cc:2726] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689) with higher term. Message: Invalid argument: T 2a823d35f4f348d1b52ba9e6d80b00d2 P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c for earlier term 2. Current term is 3.
I20260809 15:46:32.885294 23522 rpcz_store.cc:276] 0809 15:46:31.696923 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:31.700022 (+ 3099us) service_pool.cc:227] Handling call
0809 15:46:32.817553 (+1117531us) spinlock_profiling.cc:243] Waited 2.9 ms on lock 0x7b54000569d4. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e92242ae 00007f34ee161c27 00007f34e7849c04 00007f34e7849b92 00007f34e7849ac1 00007f34e7849a3c 00007f34e7848cf2 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:46:32.817638 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6673792}
I20260809 15:46:32.857285 23228 leader_election.cc:302] T c0741ab44d99439abc3aa3bdd27ee160 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 61e69695258f48baa984c9d395e8ce3f
I20260809 15:46:32.863865 23404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "213543cdbe814c89a17a963469922c1b" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:32.864527 23394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "907bacfa79d04a55a68c8dbe60056462" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:32.869433 25848 leader_election.cc:288] T d0721a4403a440f383318a3cea9d1442 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:32.847893 24455 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a51c0baaa6e432997c9d3552af4f254" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:32.876163 25772 raft_consensus.cc:482] T 6ae14a74a26f434db08c652e097385c8 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:46:32.870945 24451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3be1e356b254d22b995be32f1c12f6f" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:32.871765 24448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472816994d814cb198b69e5248697440" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:32.871896 25875 leader_election.cc:288] T d3e94b7e50754a81b3d72a07476cb7b9 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
W20260809 15:46:32.872313 25953 consensus_meta.cc:287] T c5c2e425a74b4e64ae70b5def750434c P c0f38c1908414566aa9115bb2cbdd85d: Time spent flushing consensus metadata: real 0.668s user 0.006s sys 0.009s
I20260809 15:46:32.831416 25931 leader_election.cc:288] T df20671a7b874dbb98c0ef28371c1308 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:32.846028 23393 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55332 (request call id 1172) took 1283 ms. Trace:
W20260809 15:46:32.882563 23528 rpcz_store.cc:269] 0809 15:46:29.420328 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:29.455794 (+ 35466us) service_pool.cc:227] Handling call
0809 15:46:32.840052 (+3384258us) spinlock_profiling.cc:243] Waited 1.92 s on lock 0x7b540016e754. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e92242ae 00007f34ee161c27 00007f34e7849c04 00007f34e7849b92 00007f34e7849ac1 00007f34e7849a3c 00007f34e7848cf2 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:46:32.840202 (+ 150us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4410414848}
I20260809 15:46:32.887593 23396 raft_consensus.cc:3037] T 0da4d5283c8a46878bd39b85e5c71f66 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:32.887620 23394 raft_consensus.cc:2454] T 907bacfa79d04a55a68c8dbe60056462 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 2.
I20260809 15:46:32.844285 26001 raft_consensus.cc:504] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:32.828964 25946 raft_consensus.cc:504] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:32.889107 23470 leader_election.cc:302] T 4b30f28311964a0b9149a50495e83d8f P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:46:32.910226 23394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "563daf42d7784db195ea99c3cb21b370" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:32.883991 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3be1e356b254d22b995be32f1c12f6f" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:32.890753 23404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9e7cadcd24d45fcbfa84209225e8f2d" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:32.891177 23527 raft_consensus.cc:1226] T 28e17dcb5efe45b8b6a8c42da03b4819 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Rejecting Update request from peer 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 2. Current term is 3. Ops: []
I20260809 15:46:32.870151 25795 leader_election.cc:288] T f7715d71dbac447b9351fc5eb36c5e80 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:32.888695 21289 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260809 15:46:32.818414 25992 raft_consensus.cc:504] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:32.912052 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e4ffcf83c234ae7910ddef3ec0b4bda" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:32.900115 23522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5712f6b6fffc4cc5a8ddfbe63e4097df" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:32.912446 23404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a88421dba80d4d3388aa5eb62e23853b" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:32.891861 25967 raft_consensus.cc:2781] T ba56725e42f74a1baa4c04d9660727f9 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Leader election won for term 3
I20260809 15:46:32.903127 23228 leader_election.cc:302] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:46:32.901047 23517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad8a3b25d7b49e5a2142d5709edd0c7" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:32.855556 25873 raft_consensus.cc:504] T 0c29937a80724ddb9d07a27534a146f3 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:32.875044 25990 raft_consensus.cc:504] T 3296ef50be254fc29cfc77d22a8f7a23 P 61e69695258f48baa984c9d395e8ce3f [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:32.882236 23395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad8a3b25d7b49e5a2142d5709edd0c7" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:32.888936 24453 raft_consensus.cc:3037] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:46:32.913797 23404 raft_consensus.cc:2454] T a88421dba80d4d3388aa5eb62e23853b P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 2.
I20260809 15:46:32.911098 23394 raft_consensus.cc:2454] T 563daf42d7784db195ea99c3cb21b370 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 2.
I20260809 15:46:32.894501 25929 raft_consensus.cc:2781] T c0741ab44d99439abc3aa3bdd27ee160 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260809 15:46:32.921574 23517 raft_consensus.cc:2362] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 3. Re-sending same reply.
I20260809 15:46:32.922137 23528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "763e58d6ba644eea830a6c1664d39c9b" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:32.899991 25772 raft_consensus.cc:3037] T 6ae14a74a26f434db08c652e097385c8 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:46:32.902245 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "297c82ed906d4b48b184985eed3de601" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:32.899560 25898 raft_consensus.cc:686] T 7660dfa2ef694927b8eda4100a2bcd9b P 61e69695258f48baa984c9d395e8ce3f [term 4 LEADER]: Becoming Leader. State: Replica: 61e69695258f48baa984c9d395e8ce3f, State: Running, Role: LEADER
I20260809 15:46:32.913605 23522 raft_consensus.cc:2379] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 61e69695258f48baa984c9d395e8ce3f in current term 4: Already voted for candidate 65c36d164b0540f099d592f38307ee6c in this term.
I20260809 15:46:32.923499 23404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae6d2c08a7a94f58ac8dd698e3fd456c" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:32.902855 23520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3e94b7e50754a81b3d72a07476cb7b9" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:32.927709 23528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba56725e42f74a1baa4c04d9660727f9" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:32.936077 23282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2d4f749fcfd4fb3bfb8115af3471598" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:32.964535 24471 raft_consensus.cc:2379] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 in current term 4: Already voted for candidate 61e69695258f48baa984c9d395e8ce3f in this term.
I20260809 15:46:32.884074 25885 raft_consensus.cc:504] T 2424fd9bd7f14e508fd106cbfda786a4 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:32.935604 23277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c29937a80724ddb9d07a27534a146f3" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:32.940646 23285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df20671a7b874dbb98c0ef28371c1308" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:32.886952 25965 raft_consensus.cc:504] T d1a134ba33ea4076bbaac0ae460c17ba P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:32.964811 23282 raft_consensus.cc:2454] T d2d4f749fcfd4fb3bfb8115af3471598 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 3.
I20260809 15:46:32.942714 23396 raft_consensus.cc:2454] T 0da4d5283c8a46878bd39b85e5c71f66 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 3.
I20260809 15:46:32.968463 23396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3296ef50be254fc29cfc77d22a8f7a23" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:32.968657 23282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05ad7d8fdca64e3da527280ca5be9280" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:32.969133 23396 raft_consensus.cc:3037] T 3296ef50be254fc29cfc77d22a8f7a23 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:32.969516 23282 raft_consensus.cc:2454] T 05ad7d8fdca64e3da527280ca5be9280 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 2.
I20260809 15:46:32.970371 25863 raft_consensus.cc:482] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:32.970633 23394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba56725e42f74a1baa4c04d9660727f9" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:32.970361 24448 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38734 (request call id 196) took 1146 ms. Trace:
I20260809 15:46:32.971364 23394 raft_consensus.cc:2379] T ba56725e42f74a1baa4c04d9660727f9 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 in current term 3: Already voted for candidate 65c36d164b0540f099d592f38307ee6c in this term.
I20260809 15:46:32.971033 25863 raft_consensus.cc:504] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:32.971242 24448 rpcz_store.cc:276] 0809 15:46:31.819469 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:32.871511 (+1052042us) service_pool.cc:227] Handling call
0809 15:46:32.965615 (+ 94104us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:46:32.908643 24451 raft_consensus.cc:2454] T d3be1e356b254d22b995be32f1c12f6f P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 1.
I20260809 15:46:32.972946 25863 leader_election.cc:288] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:32.970412 25924 raft_consensus.cc:482] T e71a05ba1d4d42bd95c3f85f9313399e P 61e69695258f48baa984c9d395e8ce3f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
I20260809 15:46:32.973152 24451 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38724 (request call id 411) took 1246 ms. Trace:
I20260809 15:46:32.974202 25746 raft_consensus.cc:482] T 0c48a3049897452383257aa757834fb5 P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
I20260809 15:46:32.929884 24455 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3924b648461e463b9d7925e5671d68ae" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f"
I20260809 15:46:32.955812 23404 raft_consensus.cc:2454] T ae6d2c08a7a94f58ac8dd698e3fd456c P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 3.
I20260809 15:46:32.883647 25573 raft_consensus.cc:504] T 0acc47ce21c3422698d1ef7277c3ed63 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } attrs { replace: true } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } }
I20260809 15:46:32.931197 25848 raft_consensus.cc:2781] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260809 15:46:32.973683 25924 raft_consensus.cc:504] T e71a05ba1d4d42bd95c3f85f9313399e P 61e69695258f48baa984c9d395e8ce3f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:32.972792 24448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3e94b7e50754a81b3d72a07476cb7b9" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:32.881698 25900 raft_consensus.cc:504] T 8e4ffcf83c234ae7910ddef3ec0b4bda P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:32.908509 25953 raft_consensus.cc:504] T c5c2e425a74b4e64ae70b5def750434c P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:33.003671 23282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2424fd9bd7f14e508fd106cbfda786a4" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:32.980584 23277 raft_consensus.cc:2454] T 0c29937a80724ddb9d07a27534a146f3 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 3.
I20260809 15:46:32.935850 25951 raft_consensus.cc:482] T 28e17dcb5efe45b8b6a8c42da03b4819 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting leader election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
I20260809 15:46:33.137712 23402 raft_consensus.cc:2454] T 8e4ffcf83c234ae7910ddef3ec0b4bda P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 3.
I20260809 15:46:33.138665 25900 leader_election.cc:288] T 8e4ffcf83c234ae7910ddef3ec0b4bda P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:33.138954 25573 leader_election.cc:288] T 0acc47ce21c3422698d1ef7277c3ed63 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 3 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:33.139570 25851 raft_consensus.cc:482] T cee02fff2c9c48a19f43eb358a89e7b0 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
I20260809 15:46:33.030753 25965 leader_election.cc:288] T d1a134ba33ea4076bbaac0ae460c17ba P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:33.140146 25639 raft_consensus.cc:482] T e14d118b2eaa42aa8a7cd4025aba7128 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:32.927652 26006 raft_consensus.cc:2781] T a88421dba80d4d3388aa5eb62e23853b P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Leader election won for term 3
I20260809 15:46:32.927930 25795 raft_consensus.cc:2781] T 4b30f28311964a0b9149a50495e83d8f P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260809 15:46:32.999959 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3622f6e3e3641cea1960e3a31498bb7" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:33.141520 25795 raft_consensus.cc:482] T 4b30f28311964a0b9149a50495e83d8f P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Starting leader election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:46:33.034682 23522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba56725e42f74a1baa4c04d9660727f9" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:33.011394 23517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a88421dba80d4d3388aa5eb62e23853b" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:33.142678 25848 raft_consensus.cc:482] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:46:32.935927 25990 leader_election.cc:288] T 3296ef50be254fc29cfc77d22a8f7a23 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 2 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
W20260809 15:46:32.986980 25919 consensus_meta.cc:287] T df20671a7b874dbb98c0ef28371c1308 P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent flushing consensus metadata: real 0.674s user 0.005s sys 0.017s
I20260809 15:46:32.999470 25652 leader_election.cc:288] T 00a1881267a14ba8b78c81040f17a014 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
W20260809 15:46:33.003952 25941 consensus_meta.cc:287] T 213543cdbe814c89a17a963469922c1b P 65c36d164b0540f099d592f38307ee6c: Time spent flushing consensus metadata: real 1.139s user 0.016s sys 0.027s
I20260809 15:46:33.143790 25851 raft_consensus.cc:504] T cee02fff2c9c48a19f43eb358a89e7b0 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:33.143606 25919 raft_consensus.cc:504] T df20671a7b874dbb98c0ef28371c1308 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:33.144147 25941 raft_consensus.cc:504] T 213543cdbe814c89a17a963469922c1b P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:33.140957 26006 raft_consensus.cc:686] T a88421dba80d4d3388aa5eb62e23853b P 65c36d164b0540f099d592f38307ee6c [term 3 LEADER]: Becoming Leader. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
W20260809 15:46:33.057024 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 153935us
I20260809 15:46:33.142443 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad8a3b25d7b49e5a2142d5709edd0c7" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:33.147585 23228 leader_election.cc:302] T cee02fff2c9c48a19f43eb358a89e7b0 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c
I20260809 15:46:33.148808 23228 leader_election.cc:302] T 0da4d5283c8a46878bd39b85e5c71f66 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:46:33.149694 23228 leader_election.cc:302] T f9e7cadcd24d45fcbfa84209225e8f2d P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:46:32.924804 25929 raft_consensus.cc:482] T c0741ab44d99439abc3aa3bdd27ee160 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
W20260809 15:46:32.927328 23399 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55320 (request call id 1093) took 2636 ms (client timeout 2760 ms). Trace:
I20260809 15:46:33.140693 25639 raft_consensus.cc:504] T e14d118b2eaa42aa8a7cd4025aba7128 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:33.057375 25942 leader_election.cc:288] T 00a1881267a14ba8b78c81040f17a014 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:33.137204 23282 raft_consensus.cc:2379] T 2424fd9bd7f14e508fd106cbfda786a4 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c in current term 4: Already voted for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in this term.
I20260809 15:46:32.997118 25882 raft_consensus.cc:482] T 0a51c0baaa6e432997c9d3552af4f254 P 61e69695258f48baa984c9d395e8ce3f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
I20260809 15:46:33.142884 24448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "907bacfa79d04a55a68c8dbe60056462" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:33.145329 25982 raft_consensus.cc:482] T 3d880de657e544468c82157769622a26 P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
I20260809 15:46:33.147039 23526 raft_consensus.cc:2454] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 3.
I20260809 15:46:33.215114 24410 leader_election.cc:302] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c; no voters: c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:33.121850 23396 raft_consensus.cc:2454] T 3296ef50be254fc29cfc77d22a8f7a23 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 2.
I20260809 15:46:33.216630 23396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2424fd9bd7f14e508fd106cbfda786a4" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
W20260809 15:46:33.157279 24463 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38734 (request call id 192) took 3658 ms (client timeout 1521 ms). Trace:
I20260809 15:46:33.202714 25913 raft_consensus.cc:482] T 4bcac4b24e7a4448abdc934bba5a624e P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
I20260809 15:46:33.155126 23283 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53554 (request call id 1161) took 1318 ms. Trace:
W20260809 15:46:33.135711 23286 consensus_meta.cc:287] T 294d4bcc1e7a477cb56455581277a60f P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent flushing consensus metadata: real 0.728s user 0.003s sys 0.009s
I20260809 15:46:33.147907 23472 leader_election.cc:302] T d0721a4403a440f383318a3cea9d1442 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:33.159219 23274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4564dfc6d4094c1b9240b367ccecf3b1" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:32.973726 24451 rpcz_store.cc:276] 0809 15:46:31.727182 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:32.972368 (+1245186us) service_pool.cc:227] Handling call
0809 15:46:32.972673 (+ 305us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:32.972827 (+ 154us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:46:32.972854 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 15:46:32.972894 (+ 40us) raft_consensus.cc:1631] Marking committed up to 4
0809 15:46:32.972942 (+ 48us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:32.972994 (+ 52us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:32.973102 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:46:32.931872 25694 leader_election.cc:288] T d2d4f749fcfd4fb3bfb8115af3471598 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
W20260809 15:46:33.145507 23276 consensus_meta.cc:287] T 538a228ddfb54bc6a36528d5ff07597b P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent flushing consensus metadata: real 0.767s user 0.012s sys 0.004s
I20260809 15:46:33.204648 23285 raft_consensus.cc:2379] T df20671a7b874dbb98c0ef28371c1308 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c in current term 3: Already voted for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in this term.
I20260809 15:46:32.928712 25948 leader_election.cc:288] T f7715d71dbac447b9351fc5eb36c5e80 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:33.163218 21289 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 15:46:33.158793 25943 raft_consensus.cc:504] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:32.964468 25967 raft_consensus.cc:686] T ba56725e42f74a1baa4c04d9660727f9 P 65c36d164b0540f099d592f38307ee6c [term 3 LEADER]: Becoming Leader. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:46:33.161250 23117 catalog_manager.cc:5495] T 16ca412a0dae4feebe85dd01bf1a9173 P 61e69695258f48baa984c9d395e8ce3f reported cstate change: term changed from 1 to 2, leader changed from 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65) to 61e69695258f48baa984c9d395e8ce3f (127.20.202.68). New cstate: current_term: 2 leader_uuid: "61e69695258f48baa984c9d395e8ce3f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260809 15:46:33.159708 26019 log.cc:924] Time spent T 28e17dcb5efe45b8b6a8c42da03b4819 P c0f38c1908414566aa9115bb2cbdd85d: Append to log took a long time: real 0.251s user 0.000s sys 0.000s
I20260809 15:46:33.207762 23400 raft_consensus.cc:2379] T c5c2e425a74b4e64ae70b5def750434c P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c in current term 3: Already voted for candidate c0f38c1908414566aa9115bb2cbdd85d in this term.
I20260809 15:46:33.137733 25951 raft_consensus.cc:3037] T 28e17dcb5efe45b8b6a8c42da03b4819 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:33.153383 23470 leader_election.cc:302] T d2d4f749fcfd4fb3bfb8115af3471598 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:46:33.159691 25873 leader_election.cc:288] T 0c29937a80724ddb9d07a27534a146f3 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
W20260809 15:46:33.246706 26017 log.cc:924] Time spent T 16ca412a0dae4feebe85dd01bf1a9173 P 67b3aeadfa7a432abe8af4d292e04ab9: Append to log took a long time: real 0.358s user 0.000s sys 0.001s
I20260809 15:46:33.003984 25868 leader_election.cc:288] T 28e17dcb5efe45b8b6a8c42da03b4819 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:33.142072 25795 raft_consensus.cc:3037] T 4b30f28311964a0b9149a50495e83d8f P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:33.197206 25848 raft_consensus.cc:3037] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:33.005023 23520 raft_consensus.cc:2454] T d3e94b7e50754a81b3d72a07476cb7b9 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 4.
W20260809 15:46:33.217540 24463 rpcz_store.cc:269] 0809 15:46:29.498663 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:29.527137 (+ 28474us) service_pool.cc:227] Handling call
0809 15:46:33.157028 (+3629891us) spinlock_profiling.cc:243] Waited 1.66 s on lock 0x7b54007116d4. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e92242ae 00007f34ee161c27 00007f34e7849c04 00007f34e7849b92 00007f34e7849ac1 00007f34e7849a3c 00007f34e7848cf2 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:46:33.157188 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3821028736}
I20260809 15:46:33.137089 23394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "763e58d6ba644eea830a6c1664d39c9b" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:32.995051 24455 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38724 (request call id 410) took 1271 ms. Trace:
I20260809 15:46:33.253487 23520 raft_consensus.cc:1261] T e14d118b2eaa42aa8a7cd4025aba7128 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Refusing update from remote peer 61e69695258f48baa984c9d395e8ce3f: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260809 15:46:33.253604 25868 raft_consensus.cc:2781] T d2d4f749fcfd4fb3bfb8115af3471598 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260809 15:46:33.253957 23394 raft_consensus.cc:2454] T 763e58d6ba644eea830a6c1664d39c9b P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 2.
I20260809 15:46:33.144966 25924 leader_election.cc:288] T e71a05ba1d4d42bd95c3f85f9313399e P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:33.218873 24410 leader_election.cc:302] T 3296ef50be254fc29cfc77d22a8f7a23 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:46:33.217417 23396 raft_consensus.cc:2379] T 2424fd9bd7f14e508fd106cbfda786a4 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c in current term 4: Already voted for candidate c0f38c1908414566aa9115bb2cbdd85d in this term.
I20260809 15:46:33.262259 23396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7baa2492df0340c4a5ac241959ce1ac3" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:33.140983 23404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7660dfa2ef694927b8eda4100a2bcd9b" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:33.203085 25990 raft_consensus.cc:482] T 472816994d814cb198b69e5248697440 P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:46:33.205206 25929 raft_consensus.cc:3037] T c0741ab44d99439abc3aa3bdd27ee160 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:33.223601 23283 rpcz_store.cc:276] 0809 15:46:31.836767 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:32.234713 (+397946us) service_pool.cc:227] Handling call
0809 15:46:32.252054 (+ 17341us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:32.252229 (+ 175us) raft_consensus.cc:1520] Early marking committed up to index 3
0809 15:46:32.521544 (+269315us) log.cc:842] Serialized 14 byte log entry
0809 15:46:33.152637 (+631093us) log.cc:842] Serialized 14 byte log entry
0809 15:46:33.154840 (+ 2203us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 15:46:33.154883 (+ 43us) raft_consensus.cc:1631] Marking committed up to 3
0809 15:46:33.154920 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:33.154979 (+ 59us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:33.155077 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":598185,"prepare.run_cpu_time_us":63,"prepare.run_wall_time_us":66,"raft.queue_time_us":67211,"raft.run_cpu_time_us":141,"raft.run_wall_time_us":8190,"thread_start_us":622159,"threads_started":3,"wal-append.queue_time_us":87189}
W20260809 15:46:33.226680 23472 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 78864.6us
I20260809 15:46:33.279605 23472 leader_election.cc:302] T 472816994d814cb198b69e5248697440 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:33.225859 25882 raft_consensus.cc:504] T 0a51c0baaa6e432997c9d3552af4f254 P 61e69695258f48baa984c9d395e8ce3f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:33.226861 23282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00a1881267a14ba8b78c81040f17a014" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:33.206413 25900 raft_consensus.cc:482] T 7baa2492df0340c4a5ac241959ce1ac3 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
I20260809 15:46:33.244659 25946 leader_election.cc:288] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 4 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:33.211661 25573 raft_consensus.cc:2781] T 0da4d5283c8a46878bd39b85e5c71f66 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader election won for term 3
W20260809 15:46:33.231122 23223 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.68:34689, user_credentials={real_user=slave}} blocked reactor thread for 224338us
W20260809 15:46:33.228916 23353 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 424521us
I20260809 15:46:33.210028 26026 raft_consensus.cc:2726] T cee02fff2c9c48a19f43eb358a89e7b0 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 15:46:33.290640 23223 leader_election.cc:302] T d3be1e356b254d22b995be32f1c12f6f P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:46:33.256102 25868 raft_consensus.cc:482] T d2d4f749fcfd4fb3bfb8115af3471598 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:46:33.225124 25478 raft_consensus.cc:2781] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260809 15:46:33.236910 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 87287.8us
I20260809 15:46:33.295543 25478 raft_consensus.cc:482] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:46:33.296031 25478 raft_consensus.cc:3037] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:46:32.882381 24452 rpcz_store.cc:276] 0809 15:46:30.299543 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:30.300735 (+ 1192us) service_pool.cc:227] Handling call
0809 15:46:30.330254 (+ 29519us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 15:46:30.410410 (+ 80156us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:46:30.413710 (+ 3300us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 15:46:30.533612 (+119902us) log.cc:842] Serialized 26 byte log entry
0809 15:46:30.917672 (+384060us) raft_consensus.cc:1631] Marking committed up to 5
0809 15:46:30.921517 (+ 3845us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:30.944802 (+ 23285us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 15:46:32.792367 (+1847565us) spinlock_profiling.cc:243] Waited 176 ms on lock 0x7b44000232cc. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e613cf11 00007f34e620aa95 00007f34e620b4b2 00007f34e8eb3511 00007f34e8eaee1a 00007f34e8ea2bee 00007f34e8ea1f36 00007f34e8ea1b58 00007f34e8eab551 00007f34e8eab509 00007f34e8eab499
0809 15:46:32.811310 (+ 18943us) raft_consensus.cc:1671] finished
0809 15:46:32.811328 (+ 18us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:32.832269 (+ 20941us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":12215,"raft.run_cpu_time_us":182,"raft.run_wall_time_us":49677,"spinlock_wait_cycles":404235648,"thread_start_us":128134,"threads_started":1,"wal-append.queue_time_us":377911}
I20260809 15:46:33.241611 24451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e4ffcf83c234ae7910ddef3ec0b4bda" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f"
I20260809 15:46:33.299638 24451 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38722 (request call id 355) took 1077 ms. Trace:
I20260809 15:46:33.299925 25899 raft_consensus.cc:482] T 998c6cee72cf40228c6dba064c0db2da P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:33.300474 23353 leader_election.cc:302] T 953eea268e17491ca76f3a555860a512 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:46:33.300562 25899 raft_consensus.cc:504] T 998c6cee72cf40228c6dba064c0db2da P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:33.302441 23283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e17dcb5efe45b8b6a8c42da03b4819" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:33.303611 23283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1a134ba33ea4076bbaac0ae460c17ba" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:33.304123 23283 raft_consensus.cc:2454] T d1a134ba33ea4076bbaac0ae460c17ba P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 1.
I20260809 15:46:33.306157 25899 leader_election.cc:288] T 998c6cee72cf40228c6dba064c0db2da P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:33.228169 25913 raft_consensus.cc:504] T 4bcac4b24e7a4448abdc934bba5a624e P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:33.307807 23270 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53554 (request call id 1164) took 1416 ms. Trace:
I20260809 15:46:33.308234 23270 rpcz_store.cc:276] 0809 15:46:31.891179 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:32.248205 (+357026us) service_pool.cc:227] Handling call
0809 15:46:32.248491 (+ 286us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 15:46:32.248779 (+ 288us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:46:32.248805 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 15:46:32.249170 (+ 365us) log.cc:842] Serialized 26 byte log entry
0809 15:46:32.411298 (+162128us) raft_consensus.cc:1631] Marking committed up to 4
0809 15:46:32.411335 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:32.411394 (+ 59us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 15:46:33.307603 (+896209us) raft_consensus.cc:1671] finished
0809 15:46:33.307622 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:33.307754 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":559,"raft.run_cpu_time_us":137,"raft.run_wall_time_us":12220,"thread_start_us":128245,"threads_started":1,"wal-append.queue_time_us":561040}
I20260809 15:46:33.309366 25913 leader_election.cc:288] T 4bcac4b24e7a4448abdc934bba5a624e P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:33.230667 25652 raft_consensus.cc:2726] T d0721a4403a440f383318a3cea9d1442 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260809 15:46:33.258502 25921 raft_consensus.cc:2726] T 4564dfc6d4094c1b9240b367ccecf3b1 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 15:46:33.227576 23395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ced718c1a5c4da79cb8c1ebd1435b87" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
W20260809 15:46:33.248564 23470 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 95652.8us
I20260809 15:46:32.827651 23518 rpcz_store.cc:276] 0809 15:46:30.407224 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:30.569545 (+162321us) service_pool.cc:227] Handling call
0809 15:46:30.569845 (+ 300us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 15:46:31.168848 (+599003us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:46:31.171069 (+ 2221us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 15:46:31.401210 (+230141us) log.cc:842] Serialized 26 byte log entry
0809 15:46:31.789122 (+387912us) raft_consensus.cc:1631] Marking committed up to 1
0809 15:46:31.800979 (+ 11857us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:32.785158 (+984179us) spinlock_profiling.cc:243] Waited 525 ms on lock 0x7b54000910d4. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e9220f51 00007f34e921f99e 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772 00007f34e764a94c 00007f34e7649d86
0809 15:46:32.785229 (+ 71us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 15:46:32.785325 (+ 96us) raft_consensus.cc:1671] finished
0809 15:46:32.785339 (+ 14us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:32.785714 (+ 375us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2409,"raft.queue_time_us":2193,"raft.run_cpu_time_us":156,"raft.run_wall_time_us":32957,"spinlock_wait_cycles":1207141632,"thread_start_us":278531,"threads_started":1,"wal-append.queue_time_us":404050,"wal-append.run_cpu_time_us":2685,"wal-append.run_wall_time_us":1081676}
I20260809 15:46:33.227097 23286 raft_consensus.cc:2454] T 294d4bcc1e7a477cb56455581277a60f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 4.
W20260809 15:46:33.316140 26027 log.cc:924] Time spent T 998c6cee72cf40228c6dba064c0db2da P 65c36d164b0540f099d592f38307ee6c: Append to log took a long time: real 0.071s user 0.000s sys 0.000s
I20260809 15:46:33.232218 23276 raft_consensus.cc:2454] T 538a228ddfb54bc6a36528d5ff07597b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 4.
I20260809 15:46:33.236038 25694 raft_consensus.cc:2781] T f9e7cadcd24d45fcbfa84209225e8f2d P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260809 15:46:33.316825 23276 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53554 (request call id 1165) took 1334 ms. Trace:
I20260809 15:46:33.316926 25694 raft_consensus.cc:482] T f9e7cadcd24d45fcbfa84209225e8f2d P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Starting leader election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:46:33.317512 25694 raft_consensus.cc:3037] T f9e7cadcd24d45fcbfa84209225e8f2d P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:33.317412 23276 rpcz_store.cc:276] 0809 15:46:31.982269 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:32.233872 (+251603us) service_pool.cc:227] Handling call
0809 15:46:33.316769 (+1082897us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:46:33.290932 25875 raft_consensus.cc:2726] T 472816994d814cb198b69e5248697440 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260809 15:46:33.319448 25477 raft_consensus.cc:2726] T 294d4bcc1e7a477cb56455581277a60f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553) with higher term. Message: Invalid argument: T 294d4bcc1e7a477cb56455581277a60f P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 3. Current term is 4.
I20260809 15:46:33.263119 23404 raft_consensus.cc:2354] T 7660dfa2ef694927b8eda4100a2bcd9b P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c for earlier term 3. Current term is 4.
I20260809 15:46:33.283247 23282 raft_consensus.cc:2454] T 00a1881267a14ba8b78c81040f17a014 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 3.
I20260809 15:46:32.995201 25746 raft_consensus.cc:504] T 0c48a3049897452383257aa757834fb5 P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:33.230604 25772 raft_consensus.cc:504] T 6ae14a74a26f434db08c652e097385c8 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:33.331864 25772 leader_election.cc:288] T 6ae14a74a26f434db08c652e097385c8 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 4 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:33.333745 25883 raft_consensus.cc:482] T 99bdb93503db4de4a1cfb0161ea0c07c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:46:33.255710 24455 rpcz_store.cc:276] 0809 15:46:31.723793 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:32.929621 (+1205828us) service_pool.cc:227] Handling call
0809 15:46:32.994992 (+ 65371us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:46:33.288277 23394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a88421dba80d4d3388aa5eb62e23853b" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:32.909412 23393 rpcz_store.cc:276] 0809 15:46:31.562630 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:32.388190 (+825560us) service_pool.cc:227] Handling call
0809 15:46:32.388572 (+ 382us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:32.388875 (+ 303us) raft_consensus.cc:1520] Early marking committed up to index 2
0809 15:46:32.389317 (+ 442us) log.cc:842] Serialized 14 byte log entry
0809 15:46:32.805015 (+415698us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 15:46:32.816124 (+ 11109us) raft_consensus.cc:1631] Marking committed up to 2
0809 15:46:32.818325 (+ 2201us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:32.826237 (+ 7912us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:32.833902 (+ 7665us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":10234,"prepare.queue_time_us":8622,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":36,"raft.queue_time_us":123631,"raft.run_cpu_time_us":129,"raft.run_wall_time_us":130,"thread_start_us":285204,"threads_started":1,"wal-append.queue_time_us":490103}
W20260809 15:46:33.154170 23392 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55320 (request call id 1092) took 2855 ms (client timeout 1826 ms). Trace:
I20260809 15:46:33.307025 25956 raft_consensus.cc:2781] T 953eea268e17491ca76f3a555860a512 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260809 15:46:33.309026 25851 leader_election.cc:288] T cee02fff2c9c48a19f43eb358a89e7b0 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:33.310412 23274 raft_consensus.cc:3037] T 4564dfc6d4094c1b9240b367ccecf3b1 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:33.352272 23228 leader_election.cc:302] T 213543cdbe814c89a17a963469922c1b P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:33.311012 23470 leader_election.cc:302] T 05ad7d8fdca64e3da527280ca5be9280 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:46:33.156205 25898 consensus_queue.cc:237] T 7660dfa2ef694927b8eda4100a2bcd9b P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
W20260809 15:46:33.204730 23399 rpcz_store.cc:269] 0809 15:46:30.290630 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:30.572677 (+282047us) service_pool.cc:227] Handling call
0809 15:46:32.924528 (+2351851us) spinlock_profiling.cc:243] Waited 61.6 ms on lock 0x7b54003755d4. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e92242ae 00007f34ee161c27 00007f34e7849c04 00007f34e7849b92 00007f34e7849ac1 00007f34e7849a3c 00007f34e7848cf2 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:46:32.927240 (+ 2712us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":141595392}
I20260809 15:46:33.312266 23353 leader_election.cc:302] T 213543cdbe814c89a17a963469922c1b P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:46:33.290594 25900 raft_consensus.cc:504] T 7baa2492df0340c4a5ac241959ce1ac3 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:33.315007 25573 raft_consensus.cc:686] T 0da4d5283c8a46878bd39b85e5c71f66 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 LEADER]: Becoming Leader. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Running, Role: LEADER
I20260809 15:46:33.315234 23395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998c6cee72cf40228c6dba064c0db2da" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
W20260809 15:46:33.315991 23286 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55434 (request call id 118) took 1420 ms (client timeout 1708 ms). Trace:
I20260809 15:46:33.225520 25982 raft_consensus.cc:504] T 3d880de657e544468c82157769622a26 P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:33.320606 23404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e71a05ba1d4d42bd95c3f85f9313399e" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:33.321684 23472 leader_election.cc:302] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:33.363750 23404 raft_consensus.cc:2454] T e71a05ba1d4d42bd95c3f85f9313399e P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 2.
I20260809 15:46:33.364394 23472 leader_election.cc:398] T 7660dfa2ef694927b8eda4100a2bcd9b P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 pre-election: Vote denied by peer c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263) with higher term. Message: Invalid argument: T 7660dfa2ef694927b8eda4100a2bcd9b P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c for earlier term 3. Current term is 4.
I20260809 15:46:33.367969 23472 leader_election.cc:401] T 7660dfa2ef694927b8eda4100a2bcd9b P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20260809 15:46:33.370828 25851 raft_consensus.cc:2726] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 15:46:33.249059 26006 consensus_queue.cc:237] T a88421dba80d4d3388aa5eb62e23853b P 65c36d164b0540f099d592f38307ee6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:33.372244 25746 leader_election.cc:288] T 0c48a3049897452383257aa757834fb5 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:33.372547 25851 raft_consensus.cc:2726] T 7660dfa2ef694927b8eda4100a2bcd9b P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263) with higher term. Message: Invalid argument: T 7660dfa2ef694927b8eda4100a2bcd9b P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c for earlier term 3. Current term is 4.
I20260809 15:46:33.373404 25943 leader_election.cc:288] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 2 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:33.374166 23274 raft_consensus.cc:2454] T 4564dfc6d4094c1b9240b367ccecf3b1 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 3.
I20260809 15:46:33.334636 25795 raft_consensus.cc:504] T 4b30f28311964a0b9149a50495e83d8f P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:33.327973 25639 leader_election.cc:288] T e14d118b2eaa42aa8a7cd4025aba7128 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:33.325966 25967 consensus_queue.cc:237] T ba56725e42f74a1baa4c04d9660727f9 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:46:33.334790 24410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 115989us
I20260809 15:46:33.379537 25795 leader_election.cc:288] T 4b30f28311964a0b9149a50495e83d8f P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 2 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:33.254923 24448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad8a3b25d7b49e5a2142d5709edd0c7" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:33.380174 25573 consensus_queue.cc:237] T 0da4d5283c8a46878bd39b85e5c71f66 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:33.290874 25924 raft_consensus.cc:2781] T 3296ef50be254fc29cfc77d22a8f7a23 P 61e69695258f48baa984c9d395e8ce3f [term 2 FOLLOWER]: Leader election won for term 2
I20260809 15:46:33.339527 23394 raft_consensus.cc:3037] T a88421dba80d4d3388aa5eb62e23853b P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:33.311828 25868 raft_consensus.cc:3037] T d2d4f749fcfd4fb3bfb8115af3471598 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:46:33.342792 23345 leader_election.cc:302] T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c
I20260809 15:46:33.349004 25956 raft_consensus.cc:482] T 953eea268e17491ca76f3a555860a512 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting leader election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
I20260809 15:46:33.344192 25841 leader_election.cc:288] T 2424fd9bd7f14e508fd106cbfda786a4 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 4 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
W20260809 15:46:33.342525 23392 rpcz_store.cc:269] 0809 15:46:30.290395 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:30.479238 (+188843us) service_pool.cc:227] Handling call
0809 15:46:33.136182 (+2656944us) spinlock_profiling.cc:243] Waited 517 ms on lock 0x7b5400192654. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e92242ae 00007f34ee161c27 00007f34e7849c04 00007f34e7849b92 00007f34e7849ac1 00007f34e7849a3c 00007f34e7848cf2 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:46:33.145577 (+ 9395us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1188890496}
I20260809 15:46:33.338747 25953 leader_election.cc:288] T c5c2e425a74b4e64ae70b5def750434c P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:33.349790 23282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3296ef50be254fc29cfc77d22a8f7a23" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:33.350188 23270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "294d4bcc1e7a477cb56455581277a60f" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:33.353318 23228 leader_election.cc:302] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters: 61e69695258f48baa984c9d395e8ce3f
I20260809 15:46:33.337314 26040 raft_consensus.cc:2781] T d3be1e356b254d22b995be32f1c12f6f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260809 15:46:33.355127 23470 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 44232.7us
I20260809 15:46:33.396083 23282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e71a05ba1d4d42bd95c3f85f9313399e" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
W20260809 15:46:33.357913 23353 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 45695.6us
I20260809 15:46:33.397174 23282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3be1e356b254d22b995be32f1c12f6f" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:33.394516 23270 raft_consensus.cc:2362] T 294d4bcc1e7a477cb56455581277a60f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 4. Re-sending same reply.
I20260809 15:46:33.359913 23283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5712f6b6fffc4cc5a8ddfbe63e4097df" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:33.358986 23396 raft_consensus.cc:2454] T 7baa2492df0340c4a5ac241959ce1ac3 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 1.
I20260809 15:46:33.310828 24451 rpcz_store.cc:276] 0809 15:46:32.221902 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:33.298589 (+1076687us) service_pool.cc:227] Handling call
0809 15:46:33.299017 (+ 428us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:33.299199 (+ 182us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:46:33.299229 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 15:46:33.299287 (+ 58us) raft_consensus.cc:1631] Marking committed up to 5
0809 15:46:33.299320 (+ 33us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:33.299419 (+ 99us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:33.299578 (+ 159us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:46:33.321730 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cef3e7277d7d4760824a3436ed4a847d" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:33.354910 25921 raft_consensus.cc:482] T 73f6f44c259547f1b05a3a434616fa54 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:33.402467 24453 raft_consensus.cc:2454] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 4.
I20260809 15:46:33.403007 24453 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38734 (request call id 197) took 1582 ms. Trace:
I20260809 15:46:33.347864 25883 raft_consensus.cc:504] T 99bdb93503db4de4a1cfb0161ea0c07c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:33.377419 25982 leader_election.cc:288] T 3d880de657e544468c82157769622a26 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:33.320037 25869 raft_consensus.cc:2741] T 538a228ddfb54bc6a36528d5ff07597b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
W20260809 15:46:33.329644 25978 log.cc:924] Time spent T 0c29937a80724ddb9d07a27534a146f3 P c0f38c1908414566aa9115bb2cbdd85d: Append to log took a long time: real 1.716s user 0.003s sys 0.005s
I20260809 15:46:33.415200 25694 raft_consensus.cc:504] T f9e7cadcd24d45fcbfa84209225e8f2d P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:33.379410 24455 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee02fff2c9c48a19f43eb358a89e7b0" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
W20260809 15:46:33.417023 24471 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38722 (request call id 349) took 1868 ms (client timeout 2177 ms). Trace:
I20260809 15:46:33.417168 24455 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e4ffcf83c234ae7910ddef3ec0b4bda" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:33.418304 24455 raft_consensus.cc:2379] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 in current term 4: Already voted for candidate 61e69695258f48baa984c9d395e8ce3f in this term.
I20260809 15:46:33.418045 25478 raft_consensus.cc:504] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:33.284941 25990 raft_consensus.cc:504] T 472816994d814cb198b69e5248697440 P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:33.419461 24448 raft_consensus.cc:2379] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c in current term 4: Already voted for candidate 61e69695258f48baa984c9d395e8ce3f in this term.
I20260809 15:46:33.421139 25990 leader_election.cc:288] T 472816994d814cb198b69e5248697440 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:33.327183 25941 leader_election.cc:288] T 213543cdbe814c89a17a963469922c1b P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:33.421548 25883 leader_election.cc:288] T 99bdb93503db4de4a1cfb0161ea0c07c P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:33.407954 25746 consensus_queue.cc:1035] T e14d118b2eaa42aa8a7cd4025aba7128 P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.014s
W20260809 15:46:33.386620 24406 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 63308.6us
I20260809 15:46:33.422991 25478 leader_election.cc:288] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 4 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:33.358829 25772 raft_consensus.cc:2781] T 05ad7d8fdca64e3da527280ca5be9280 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260809 15:46:33.423460 25772 raft_consensus.cc:482] T 05ad7d8fdca64e3da527280ca5be9280 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Starting leader election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:46:33.423804 25772 raft_consensus.cc:3037] T 05ad7d8fdca64e3da527280ca5be9280 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Advancing to term 3
W20260809 15:46:33.424459 26005 consensus_meta.cc:287] T 3924b648461e463b9d7925e5671d68ae P 65c36d164b0540f099d592f38307ee6c: Time spent flushing consensus metadata: real 0.622s user 0.002s sys 0.008s
I20260809 15:46:33.356746 26026 raft_consensus.cc:2726] T 213543cdbe814c89a17a963469922c1b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260809 15:46:33.362810 25899 raft_consensus.cc:2781] T 213543cdbe814c89a17a963469922c1b P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260809 15:46:33.383183 23276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e14d118b2eaa42aa8a7cd4025aba7128" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:33.432840 23276 raft_consensus.cc:2379] T e14d118b2eaa42aa8a7cd4025aba7128 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 61e69695258f48baa984c9d395e8ce3f in current term 4: Already voted for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in this term.
I20260809 15:46:33.433225 25731 leader_election.cc:288] T 16ca412a0dae4feebe85dd01bf1a9173 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:33.433979 23276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "297c82ed906d4b48b184985eed3de601" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:33.429348 25899 raft_consensus.cc:482] T 213543cdbe814c89a17a963469922c1b P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:46:33.436441 25899 raft_consensus.cc:3037] T 213543cdbe814c89a17a963469922c1b P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:33.398613 23270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "953eea268e17491ca76f3a555860a512" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:33.354101 25951 raft_consensus.cc:504] T 28e17dcb5efe45b8b6a8c42da03b4819 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:33.399597 23228 leader_election.cc:302] T a88421dba80d4d3388aa5eb62e23853b P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:46:33.400746 23399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73f6f44c259547f1b05a3a434616fa54" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:33.400882 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d880de657e544468c82157769622a26" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:33.396145 26040 raft_consensus.cc:482] T d3be1e356b254d22b995be32f1c12f6f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:46:33.402230 25921 raft_consensus.cc:504] T 73f6f44c259547f1b05a3a434616fa54 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:33.364518 23404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0da4d5283c8a46878bd39b85e5c71f66" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:33.454937 23404 raft_consensus.cc:2379] T 0da4d5283c8a46878bd39b85e5c71f66 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c in current term 3: Already voted for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in this term.
I20260809 15:46:33.455662 23404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4564dfc6d4094c1b9240b367ccecf3b1" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:33.456161 23404 raft_consensus.cc:2379] T 4564dfc6d4094c1b9240b367ccecf3b1 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c in current term 3: Already voted for candidate c0f38c1908414566aa9115bb2cbdd85d in this term.
I20260809 15:46:33.392659 23393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00a1881267a14ba8b78c81040f17a014" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:33.363519 25919 leader_election.cc:288] T df20671a7b874dbb98c0ef28371c1308 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 3 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:33.457985 23404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cef3e7277d7d4760824a3436ed4a847d" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:33.458505 23404 raft_consensus.cc:2454] T cef3e7277d7d4760824a3436ed4a847d P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 3.
W20260809 15:46:33.362622 23286 rpcz_store.cc:269] 0809 15:46:31.895325 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:31.956256 (+ 60931us) service_pool.cc:227] Handling call
0809 15:46:33.315932 (+1359676us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:46:33.404399 23283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "763e58d6ba644eea830a6c1664d39c9b" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:33.405007 23529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e14d118b2eaa42aa8a7cd4025aba7128" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:33.405454 25882 leader_election.cc:288] T 0a51c0baaa6e432997c9d3552af4f254 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
W20260809 15:46:33.405670 23521 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56766 (request call id 1670) took 2999 ms (client timeout 1880 ms). Trace:
I20260809 15:46:33.460008 23529 raft_consensus.cc:2379] T e14d118b2eaa42aa8a7cd4025aba7128 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 in current term 4: Already voted for candidate 61e69695258f48baa984c9d395e8ce3f in this term.
I20260809 15:46:33.406008 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c48a3049897452383257aa757834fb5" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
W20260809 15:46:33.459726 23521 rpcz_store.cc:269] 0809 15:46:30.406459 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:30.407378 (+ 919us) service_pool.cc:227] Handling call
0809 15:46:33.405513 (+2998135us) spinlock_profiling.cc:243] Waited 1.36 s on lock 0x7b54001077d4. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e92242ae 00007f34ee161c27 00007f34e7849c04 00007f34e7849b92 00007f34e7849ac1 00007f34e7849a3c 00007f34e7848cf2 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:46:33.405609 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3135335296}
I20260809 15:46:33.406306 26041 raft_consensus.cc:482] T 30c1740ef6874df0a332fbd6fd4b03ce P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:33.469940 23404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00a1881267a14ba8b78c81040f17a014" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:33.407476 24451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16ca412a0dae4feebe85dd01bf1a9173" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:33.408162 25900 leader_election.cc:288] T 7baa2492df0340c4a5ac241959ce1ac3 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:33.376601 23395 raft_consensus.cc:2362] T 998c6cee72cf40228c6dba064c0db2da P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 5. Re-sending same reply.
I20260809 15:46:33.408056 23470 leader_election.cc:302] T d1a134ba33ea4076bbaac0ae460c17ba P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:46:33.389648 25956 raft_consensus.cc:3037] T 953eea268e17491ca76f3a555860a512 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:33.393000 25913 raft_consensus.cc:2726] T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260809 15:46:33.410254 23353 leader_election.cc:302] T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters: 61e69695258f48baa984c9d395e8ce3f
I20260809 15:46:33.437525 24455 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bcac4b24e7a4448abdc934bba5a624e" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
W20260809 15:46:33.482528 23353 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 72362.4us
I20260809 15:46:33.351964 23518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "297c82ed906d4b48b184985eed3de601" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:33.486482 23518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2424fd9bd7f14e508fd106cbfda786a4" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:33.487427 23518 raft_consensus.cc:2379] T 2424fd9bd7f14e508fd106cbfda786a4 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 in current term 4: Already voted for candidate 65c36d164b0540f099d592f38307ee6c in this term.
I20260809 15:46:33.488328 26001 leader_election.cc:288] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:33.429072 23392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "297c82ed906d4b48b184985eed3de601" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:33.495862 23392 raft_consensus.cc:2454] T 297c82ed906d4b48b184985eed3de601 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 4.
I20260809 15:46:33.399221 25943 raft_consensus.cc:2781] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260809 15:46:33.437481 23527 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56770 (request call id 1166) took 1125 ms. Trace:
W20260809 15:46:33.429095 24471 rpcz_store.cc:269] 0809 15:46:31.548637 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:31.548755 (+ 118us) service_pool.cc:227] Handling call
0809 15:46:33.416838 (+1868083us) spinlock_profiling.cc:243] Waited 1.26 s on lock 0x7b5400545454. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e92242ae 00007f34ee161c27 00007f34e7849c04 00007f34e7849b92 00007f34e7849ac1 00007f34e7849a3c 00007f34e7848cf2 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:46:33.416960 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2894832896}
I20260809 15:46:33.438521 23228 leader_election.cc:302] T 563daf42d7784db195ea99c3cb21b370 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters: 61e69695258f48baa984c9d395e8ce3f
W20260809 15:46:33.498294 23397 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55320 (request call id 1079) took 3495 ms (client timeout 2688 ms). Trace:
W20260809 15:46:33.498677 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 60209.7us
I20260809 15:46:33.440052 23399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2db057fd10e149a9895dff6ef8c9f038" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:33.499019 23395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f9d6565aa4c45a3b3f01210d0eb71f7" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:33.499486 23399 raft_consensus.cc:2454] T 2db057fd10e149a9895dff6ef8c9f038 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 2.
W20260809 15:46:33.499728 24472 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38722 (request call id 348) took 1972 ms (client timeout 2050 ms). Trace:
W20260809 15:46:33.500078 24472 rpcz_store.cc:269] 0809 15:46:31.527299 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:31.527421 (+ 122us) service_pool.cc:227] Handling call
0809 15:46:33.499576 (+1972155us) spinlock_profiling.cc:243] Waited 134 ms on lock 0x7b54006667d4. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e92242ae 00007f34ee161c27 00007f34e7849c04 00007f34e7849b92 00007f34e7849ac1 00007f34e7849a3c 00007f34e7848cf2 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:46:33.499676 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":308519296}
I20260809 15:46:33.441126 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c29937a80724ddb9d07a27534a146f3" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:33.441840 25885 leader_election.cc:288] T 2424fd9bd7f14e508fd106cbfda786a4 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:33.445626 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c29937a80724ddb9d07a27534a146f3" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:33.446349 23280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9e7cadcd24d45fcbfa84209225e8f2d" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:33.429747 25959 leader_election.cc:288] T cef3e7277d7d4760824a3436ed4a847d P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:33.447613 23394 raft_consensus.cc:2454] T a88421dba80d4d3388aa5eb62e23853b P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 3.
I20260809 15:46:33.448402 23270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3296ef50be254fc29cfc77d22a8f7a23" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:33.408622 24453 rpcz_store.cc:276] 0809 15:46:31.821065 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:32.844206 (+1023141us) service_pool.cc:227] Handling call
0809 15:46:33.402944 (+558738us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 15:46:33.453701 23472 leader_election.cc:334] T 7baa2492df0340c4a5ac241959ce1ac3 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263): Timed out: RequestConsensusVote RPC to 127.20.202.66:36263 timed out after 1.665s (SENT)
I20260809 15:46:33.403177 25852 leader_election.cc:288] T 2424fd9bd7f14e508fd106cbfda786a4 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 4 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:33.460381 23529 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56770 (request call id 1157) took 1242 ms. Trace:
W20260809 15:46:33.464442 24410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 84096.9us
I20260809 15:46:33.475515 23404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a823d35f4f348d1b52ba9e6d80b00d2" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:33.478647 23521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00a1881267a14ba8b78c81040f17a014" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
W20260809 15:46:33.477834 23470 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 69862.7us
I20260809 15:46:33.456758 25942 raft_consensus.cc:2741] T a88421dba80d4d3388aa5eb62e23853b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260809 15:46:33.456881 26040 raft_consensus.cc:3037] T d3be1e356b254d22b995be32f1c12f6f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:33.482892 25694 leader_election.cc:288] T f9e7cadcd24d45fcbfa84209225e8f2d P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 3 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:33.496781 23527 rpcz_store.cc:276] 0809 15:46:32.312208 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:32.312325 (+ 117us) service_pool.cc:227] Handling call
0809 15:46:32.317958 (+ 5633us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:32.911653 (+593695us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:33.437278 (+525625us) spinlock_profiling.cc:243] Waited 571 ms on lock 0x7b540020f154. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e9220f51 00007f34e921f99e 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772 00007f34e764a94c 00007f34e7649d86
0809 15:46:33.437427 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1314328960}
I20260809 15:46:33.595515 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f9d6565aa4c45a3b3f01210d0eb71f7" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f"
I20260809 15:46:33.613692 24471 raft_consensus.cc:3037] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:33.613623 23472 leader_election.cc:302] T 7baa2492df0340c4a5ac241959ce1ac3 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:33.614239 23394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7715d71dbac447b9351fc5eb36c5e80" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:33.608189 23522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0acc47ce21c3422698d1ef7277c3ed63" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:33.614900 23394 raft_consensus.cc:2379] T f7715d71dbac447b9351fc5eb36c5e80 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c in current term 3: Already voted for candidate c0f38c1908414566aa9115bb2cbdd85d in this term.
I20260809 15:46:33.480755 24468 raft_consensus.cc:2454] T 0c48a3049897452383257aa757834fb5 P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 1.
W20260809 15:46:33.614877 23472 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 161204us
I20260809 15:46:33.616647 25967 raft_consensus.cc:2726] T 7baa2492df0340c4a5ac241959ce1ac3 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 15:46:33.603199 23519 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56770 (request call id 1165) took 1329 ms. Trace:
I20260809 15:46:33.615451 23522 raft_consensus.cc:3037] T 0acc47ce21c3422698d1ef7277c3ed63 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Advancing to term 3
W20260809 15:46:33.616655 23472 leader_election.cc:334] T 998c6cee72cf40228c6dba064c0db2da P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263): Timed out: RequestConsensusVote RPC to 127.20.202.66:36263 timed out after 1.672s (SENT)
W20260809 15:46:33.620055 23472 leader_election.cc:334] T 73f6f44c259547f1b05a3a434616fa54 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263): Timed out: RequestConsensusVote RPC to 127.20.202.66:36263 timed out after 1.783s (SENT)
I20260809 15:46:33.615018 23529 rpcz_store.cc:276] 0809 15:46:32.217858 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:32.218285 (+ 427us) service_pool.cc:227] Handling call
0809 15:46:33.404757 (+1186472us) spinlock_profiling.cc:243] Waited 1.12 ms on lock 0x7b5800004298. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e613cf11 00007f34e62c0851 00007f34e62c0702 00007f34ef1b0b09 00007f34ee0ec730 00007f34ee1b279b 00007f34ee1b2821 00007f34ee1621b0 00007f34ee161a94 00007f34e7849c04 00007f34e7849b92
0809 15:46:33.460332 (+ 55575us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2576256}
I20260809 15:46:33.620457 23472 leader_election.cc:302] T 73f6f44c259547f1b05a3a434616fa54 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:33.620906 23529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7baa2492df0340c4a5ac241959ce1ac3" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:33.613021 23526 raft_consensus.cc:2454] T 0c29937a80724ddb9d07a27534a146f3 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 3.
I20260809 15:46:33.618865 23280 raft_consensus.cc:2379] T f9e7cadcd24d45fcbfa84209225e8f2d P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c in current term 3: Already voted for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in this term.
I20260809 15:46:33.622771 23472 leader_election.cc:302] T 0da4d5283c8a46878bd39b85e5c71f66 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:33.613543 24410 leader_election.cc:302] T e71a05ba1d4d42bd95c3f85f9313399e P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d; no voters:
W20260809 15:46:33.617568 24468 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38734 (request call id 199) took 1781 ms (client timeout 1895 ms). Trace:
I20260809 15:46:33.488443 25900 raft_consensus.cc:2781] T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260809 15:46:33.498718 23397 rpcz_store.cc:269] 0809 15:46:30.003161 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:30.003514 (+ 353us) service_pool.cc:227] Handling call
0809 15:46:33.498114 (+3494600us) spinlock_profiling.cc:243] Waited 1.38 s on lock 0x7b540025e254. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e92242ae 00007f34ee161c27 00007f34e7849c04 00007f34e7849b92 00007f34e7849ac1 00007f34e7849a3c 00007f34e7848cf2 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:46:33.498230 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3165257600}
W20260809 15:46:33.622952 23280 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53554 (request call id 1157) took 1843 ms (client timeout 2109 ms). Trace:
I20260809 15:46:33.489411 23276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07fcabb569414bddbd39df6cedd24e64" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:33.616169 23228 leader_election.cc:302] T ae6d2c08a7a94f58ac8dd698e3fd456c P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c
I20260809 15:46:33.638094 23228 leader_election.cc:302] T 763e58d6ba644eea830a6c1664d39c9b P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c
I20260809 15:46:33.638803 23404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e4ffcf83c234ae7910ddef3ec0b4bda" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:33.639359 23404 raft_consensus.cc:3037] T 8e4ffcf83c234ae7910ddef3ec0b4bda P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:46:33.639787 23228 leader_election.cc:302] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters: 61e69695258f48baa984c9d395e8ce3f
I20260809 15:46:33.640033 23472 leader_election.cc:302] T 2db057fd10e149a9895dff6ef8c9f038 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters: 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:33.603820 24451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e14d118b2eaa42aa8a7cd4025aba7128" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
W20260809 15:46:33.624010 24468 rpcz_store.cc:269] 0809 15:46:31.836515 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:33.405749 (+1569234us) service_pool.cc:227] Handling call
0809 15:46:33.617488 (+211739us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:46:33.629948 23521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4564dfc6d4094c1b9240b367ccecf3b1" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:33.600582 24455 raft_consensus.cc:1226] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Rejecting Update request from peer c0f38c1908414566aa9115bb2cbdd85d for earlier term 3. Current term is 4. Ops: []
I20260809 15:46:33.631506 23397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b30f28311964a0b9149a50495e83d8f" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
W20260809 15:46:33.626008 23280 rpcz_store.cc:269] 0809 15:46:31.779285 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:31.779529 (+ 244us) service_pool.cc:227] Handling call
0809 15:46:33.446225 (+1666696us) spinlock_profiling.cc:243] Waited 3.5 ms on lock 0x7b5800003398. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e613cf11 00007f34e62c0851 00007f34e62c0702 00007f34ef1b0b09 00007f34ee0ec730 00007f34ee1b279b 00007f34ee1b2821 00007f34ee1621b0 00007f34ee161a94 00007f34e7849c04 00007f34e7849b92
0809 15:46:33.622910 (+176685us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":8053504}
I20260809 15:46:33.632405 25852 raft_consensus.cc:2726] T 73f6f44c259547f1b05a3a434616fa54 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260809 15:46:33.481897 25793 raft_consensus.cc:2781] T d1a134ba33ea4076bbaac0ae460c17ba P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260809 15:46:33.500599 23399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b30f28311964a0b9149a50495e83d8f" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:33.655998 25793 raft_consensus.cc:482] T d1a134ba33ea4076bbaac0ae460c17ba P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Starting leader election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:46:33.656543 25793 raft_consensus.cc:3037] T d1a134ba33ea4076bbaac0ae460c17ba P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:33.634218 23470 leader_election.cc:302] T 538a228ddfb54bc6a36528d5ff07597b P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:46:33.656993 23399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e17dcb5efe45b8b6a8c42da03b4819" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:33.596333 25951 leader_election.cc:288] T 28e17dcb5efe45b8b6a8c42da03b4819 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:33.635120 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7715d71dbac447b9351fc5eb36c5e80" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:33.635167 23468 leader_election.cc:302] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c; no voters: 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:33.661094 23470 leader_election.cc:302] T 00a1881267a14ba8b78c81040f17a014 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters: c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:33.483690 26041 raft_consensus.cc:504] T 30c1740ef6874df0a332fbd6fd4b03ce P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:33.640913 23228 leader_election.cc:302] T cef3e7277d7d4760824a3436ed4a847d P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:46:33.618382 23270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "213543cdbe814c89a17a963469922c1b" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:33.665975 23270 raft_consensus.cc:2454] T 213543cdbe814c89a17a963469922c1b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 2.
I20260809 15:46:33.666853 23270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2db057fd10e149a9895dff6ef8c9f038" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:33.568042 23223 leader_election.cc:302] T 16ca412a0dae4feebe85dd01bf1a9173 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c
I20260809 15:46:33.618139 23519 rpcz_store.cc:276] 0809 15:46:32.274721 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:32.274821 (+ 100us) service_pool.cc:227] Handling call
0809 15:46:32.296242 (+ 21421us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:32.296448 (+ 206us) raft_consensus.cc:1520] Early marking committed up to index 2
0809 15:46:32.316694 (+ 20246us) log.cc:842] Serialized 14 byte log entry
0809 15:46:33.448128 (+1131434us) spinlock_profiling.cc:243] Waited 59 ms on lock 0x7b18002da720. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34ef1978b1 00007f34ef1a4661 00007f34e62da345 00007f34ef111a39 00007f34e62f78bb 00007f34e62f5cbf 00007f34e62f7e22 00007f34e8ea15bb 00007f34e8ea7063 00007f34e8ea757b 00007f34e922a88a
0809 15:46:33.602896 (+154768us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 15:46:33.602946 (+ 50us) raft_consensus.cc:1631] Marking committed up to 2
0809 15:46:33.602985 (+ 39us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:33.603044 (+ 59us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:33.603148 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":17607,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":1148,"raft.queue_time_us":93,"raft.run_cpu_time_us":76,"raft.run_wall_time_us":76,"spinlock_wait_cycles":135717248,"thread_start_us":154404,"threads_started":1}
I20260809 15:46:33.643774 23472 leader_election.cc:302] T f7715d71dbac447b9351fc5eb36c5e80 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:33.644228 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a51c0baaa6e432997c9d3552af4f254" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:33.646234 24471 raft_consensus.cc:2454] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 61e69695258f48baa984c9d395e8ce3f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 2.
I20260809 15:46:33.630798 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0721a4403a440f383318a3cea9d1442" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:33.546116 23286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30c1740ef6874df0a332fbd6fd4b03ce" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:33.647966 23394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0acc47ce21c3422698d1ef7277c3ed63" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:33.667871 23519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998c6cee72cf40228c6dba064c0db2da" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:33.648141 23521 raft_consensus.cc:2379] T 4564dfc6d4094c1b9240b367ccecf3b1 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d in current term 3: Already voted for candidate 65c36d164b0540f099d592f38307ee6c in this term.
I20260809 15:46:33.631398 26005 raft_consensus.cc:2726] T 0da4d5283c8a46878bd39b85e5c71f66 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 15:46:33.500914 25919 raft_consensus.cc:2781] T 563daf42d7784db195ea99c3cb21b370 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260809 15:46:33.557826 25999 raft_consensus.cc:482] T 3924b648461e463b9d7925e5671d68ae P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:33.648358 25694 raft_consensus.cc:2781] T ae6d2c08a7a94f58ac8dd698e3fd456c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260809 15:46:33.611871 25927 leader_election.cc:288] T 05ad7d8fdca64e3da527280ca5be9280 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:33.609148 25772 raft_consensus.cc:504] T 05ad7d8fdca64e3da527280ca5be9280 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:33.546399 25924 raft_consensus.cc:686] T 3296ef50be254fc29cfc77d22a8f7a23 P 61e69695258f48baa984c9d395e8ce3f [term 2 LEADER]: Becoming Leader. State: Replica: 61e69695258f48baa984c9d395e8ce3f, State: Running, Role: LEADER
I20260809 15:46:33.582068 23395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07fcabb569414bddbd39df6cedd24e64" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:33.561744 23392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2424fd9bd7f14e508fd106cbfda786a4" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:33.564589 23393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3be1e356b254d22b995be32f1c12f6f" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:33.640375 23529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16ca412a0dae4feebe85dd01bf1a9173" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:33.663323 23280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4564dfc6d4094c1b9240b367ccecf3b1" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:33.496692 25943 raft_consensus.cc:482] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
I20260809 15:46:33.635640 25478 raft_consensus.cc:2781] T e71a05ba1d4d42bd95c3f85f9313399e P 61e69695258f48baa984c9d395e8ce3f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260809 15:46:33.691906 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bcac4b24e7a4448abdc934bba5a624e" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:33.692261 23529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e17dcb5efe45b8b6a8c42da03b4819" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:33.668851 23519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a823d35f4f348d1b52ba9e6d80b00d2" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:33.669234 23402 raft_consensus.cc:2454] T 0a51c0baaa6e432997c9d3552af4f254 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 2.
I20260809 15:46:33.668004 23526 raft_consensus.cc:2454] T d0721a4403a440f383318a3cea9d1442 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 3.
I20260809 15:46:33.669579 23228 leader_election.cc:302] T 297c82ed906d4b48b184985eed3de601 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:46:33.693154 23519 raft_consensus.cc:2362] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 3. Re-sending same reply.
I20260809 15:46:33.677037 25694 raft_consensus.cc:482] T ae6d2c08a7a94f58ac8dd698e3fd456c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:33.624379 25900 raft_consensus.cc:482] T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:46:33.707949 25900 raft_consensus.cc:3037] T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Advancing to term 5
I20260809 15:46:33.709832 25694 raft_consensus.cc:3037] T ae6d2c08a7a94f58ac8dd698e3fd456c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:46:33.713020 25885 raft_consensus.cc:2781] T 297c82ed906d4b48b184985eed3de601 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260809 15:46:33.713521 25885 raft_consensus.cc:482] T 297c82ed906d4b48b184985eed3de601 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:33.714020 25885 raft_consensus.cc:3037] T 297c82ed906d4b48b184985eed3de601 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Advancing to term 5
I20260809 15:46:33.686530 25875 raft_consensus.cc:3037] T f7715d71dbac447b9351fc5eb36c5e80 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:33.672467 23270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a51c0baaa6e432997c9d3552af4f254" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:33.488302 23518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c48a3049897452383257aa757834fb5" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:33.648937 25501 raft_consensus.cc:2781] T 763e58d6ba644eea830a6c1664d39c9b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260809 15:46:33.721827 23270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba56725e42f74a1baa4c04d9660727f9" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:33.652122 26006 raft_consensus.cc:2781] T 2db057fd10e149a9895dff6ef8c9f038 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260809 15:46:33.675824 23472 leader_election.cc:302] T 28e17dcb5efe45b8b6a8c42da03b4819 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d
W20260809 15:46:33.724733 23472 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 48974.7us
I20260809 15:46:33.676414 23521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5c2e425a74b4e64ae70b5def750434c" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:33.658162 25942 raft_consensus.cc:2781] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260809 15:46:33.730746 25942 raft_consensus.cc:482] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:46:33.599748 25921 leader_election.cc:288] T 73f6f44c259547f1b05a3a434616fa54 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:33.605502 25868 raft_consensus.cc:504] T d2d4f749fcfd4fb3bfb8115af3471598 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:46:33.676641 23223 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.68:34689, user_credentials={real_user=slave}} blocked reactor thread for 108678us
I20260809 15:46:33.678206 23399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1a134ba33ea4076bbaac0ae460c17ba" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:33.675040 25848 raft_consensus.cc:504] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:33.691898 25478 raft_consensus.cc:482] T e71a05ba1d4d42bd95c3f85f9313399e P 61e69695258f48baa984c9d395e8ce3f [term 2 FOLLOWER]: Starting leader election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
I20260809 15:46:33.679237 24475 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38722 (request call id 347) took 2187 ms. Trace:
I20260809 15:46:33.740890 23399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99bdb93503db4de4a1cfb0161ea0c07c" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:33.680146 25883 raft_consensus.cc:2726] T 16ca412a0dae4feebe85dd01bf1a9173 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260809 15:46:33.681336 25999 raft_consensus.cc:504] T 3924b648461e463b9d7925e5671d68ae P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:33.681723 23283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e4ffcf83c234ae7910ddef3ec0b4bda" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:33.683224 23397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0721a4403a440f383318a3cea9d1442" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:33.607151 25929 raft_consensus.cc:504] T c0741ab44d99439abc3aa3bdd27ee160 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:33.670214 25852 raft_consensus.cc:2781] T 538a228ddfb54bc6a36528d5ff07597b P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Leader election won for term 4
I20260809 15:46:33.670711 25650 raft_consensus.cc:2781] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Leader election won for term 4
I20260809 15:46:33.763072 25848 leader_election.cc:288] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 2 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:33.673054 25573 raft_consensus.cc:2781] T cef3e7277d7d4760824a3436ed4a847d P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260809 15:46:33.671974 25849 raft_consensus.cc:2781] T 00a1881267a14ba8b78c81040f17a014 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260809 15:46:33.680074 25789 leader_election.cc:288] T df20671a7b874dbb98c0ef28371c1308 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:33.672250 25919 raft_consensus.cc:482] T 563daf42d7784db195ea99c3cb21b370 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
I20260809 15:46:33.764319 25573 raft_consensus.cc:482] T cef3e7277d7d4760824a3436ed4a847d P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:46:33.694139 23519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a823d35f4f348d1b52ba9e6d80b00d2" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:33.767338 25849 raft_consensus.cc:482] T 00a1881267a14ba8b78c81040f17a014 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Starting leader election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:46:33.768934 23519 raft_consensus.cc:2454] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 3.
I20260809 15:46:33.668339 23394 raft_consensus.cc:3037] T 0acc47ce21c3422698d1ef7277c3ed63 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:33.769944 23519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9e7cadcd24d45fcbfa84209225e8f2d" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:33.770498 23519 raft_consensus.cc:3037] T f9e7cadcd24d45fcbfa84209225e8f2d P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:33.695528 23280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "907bacfa79d04a55a68c8dbe60056462" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:33.698985 24406 leader_election.cc:302] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:33.677073 25816 raft_consensus.cc:482] T 99bdb93503db4de4a1cfb0161ea0c07c P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:46:33.763788 25852 raft_consensus.cc:686] T 538a228ddfb54bc6a36528d5ff07597b P 65c36d164b0540f099d592f38307ee6c [term 4 LEADER]: Becoming Leader. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:46:33.771842 25816 raft_consensus.cc:504] T 99bdb93503db4de4a1cfb0161ea0c07c P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:33.772624 25852 consensus_queue.cc:237] T 538a228ddfb54bc6a36528d5ff07597b P 65c36d164b0540f099d592f38307ee6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:33.699680 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05ad7d8fdca64e3da527280ca5be9280" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:33.700429 23228 leader_election.cc:302] T e14d118b2eaa42aa8a7cd4025aba7128 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c
I20260809 15:46:33.704013 24410 leader_election.cc:302] T 0a51c0baaa6e432997c9d3552af4f254 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:46:33.684082 24455 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38724 (request call id 413) took 1418 ms. Trace:
I20260809 15:46:33.780305 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2d4f749fcfd4fb3bfb8115af3471598" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:33.780561 24455 rpcz_store.cc:276] 0809 15:46:32.258446 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:33.599894 (+1341448us) service_pool.cc:227] Handling call
0809 15:46:33.600324 (+ 430us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:33.661350 (+ 61026us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:33.676435 (+ 15085us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:46:33.704378 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472816994d814cb198b69e5248697440" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:33.781841 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99bdb93503db4de4a1cfb0161ea0c07c" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:33.763237 25650 raft_consensus.cc:686] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 65c36d164b0540f099d592f38307ee6c [term 4 LEADER]: Becoming Leader. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:46:33.782301 23397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f9d6565aa4c45a3b3f01210d0eb71f7" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:33.783191 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba56725e42f74a1baa4c04d9660727f9" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:33.707093 26026 consensus_queue.cc:1046] T 28e17dcb5efe45b8b6a8c42da03b4819 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20260809 15:46:33.718842 23353 leader_election.cc:302] T 4bcac4b24e7a4448abdc934bba5a624e P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c
I20260809 15:46:33.665685 23404 raft_consensus.cc:2454] T 8e4ffcf83c234ae7910ddef3ec0b4bda P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 4.
I20260809 15:46:33.784837 23394 raft_consensus.cc:2454] T 0acc47ce21c3422698d1ef7277c3ed63 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 3.
I20260809 15:46:33.819406 23404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4564dfc6d4094c1b9240b367ccecf3b1" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:33.819545 23394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05ad7d8fdca64e3da527280ca5be9280" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:33.820096 23404 raft_consensus.cc:2379] T 4564dfc6d4094c1b9240b367ccecf3b1 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c in current term 3: Already voted for candidate c0f38c1908414566aa9115bb2cbdd85d in this term.
I20260809 15:46:33.689015 24453 raft_consensus.cc:1226] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Rejecting Update request from peer 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 3. Current term is 4. Ops: []
I20260809 15:46:33.693034 25943 raft_consensus.cc:3037] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:33.722601 25501 raft_consensus.cc:482] T 763e58d6ba644eea830a6c1664d39c9b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:33.729511 25791 leader_election.cc:288] T c0741ab44d99439abc3aa3bdd27ee160 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:33.729104 25793 raft_consensus.cc:504] T d1a134ba33ea4076bbaac0ae460c17ba P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:33.734480 23223 leader_election.cc:302] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:46:33.653669 25899 raft_consensus.cc:504] T 213543cdbe814c89a17a963469922c1b P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:33.740710 23521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99bdb93503db4de4a1cfb0161ea0c07c" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:33.829772 23521 raft_consensus.cc:2454] T 99bdb93503db4de4a1cfb0161ea0c07c P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 1.
I20260809 15:46:33.690598 23395 raft_consensus.cc:2454] T 07fcabb569414bddbd39df6cedd24e64 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 3.
I20260809 15:46:33.741333 24475 rpcz_store.cc:276] 0809 15:46:31.458442 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:31.458737 (+ 295us) service_pool.cc:227] Handling call
0809 15:46:31.459160 (+ 423us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:31.852413 (+393253us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:46:31.852453 (+ 40us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 15:46:31.852494 (+ 41us) raft_consensus.cc:1631] Marking committed up to 4
0809 15:46:31.852524 (+ 30us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:33.645522 (+1792998us) spinlock_profiling.cc:243] Waited 389 ms on lock 0x7b54005a1754. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e9220f51 00007f34e921f99e 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772 00007f34e764a94c 00007f34e7649d86
0809 15:46:33.645569 (+ 47us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:33.645678 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":893772544}
I20260809 15:46:33.727813 23392 raft_consensus.cc:2379] T 2424fd9bd7f14e508fd106cbfda786a4 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 in current term 4: Already voted for candidate c0f38c1908414566aa9115bb2cbdd85d in this term.
I20260809 15:46:33.751171 24451 raft_consensus.cc:1261] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260809 15:46:33.822399 25913 raft_consensus.cc:482] T 0acc47ce21c3422698d1ef7277c3ed63 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:33.727459 26006 raft_consensus.cc:482] T 2db057fd10e149a9895dff6ef8c9f038 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Starting leader election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
I20260809 15:46:33.841367 26006 raft_consensus.cc:3037] T 2db057fd10e149a9895dff6ef8c9f038 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:33.731241 25942 raft_consensus.cc:3037] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:46:33.758924 26041 leader_election.cc:288] T 30c1740ef6874df0a332fbd6fd4b03ce P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:33.761096 25929 leader_election.cc:288] T c0741ab44d99439abc3aa3bdd27ee160 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 3 election: Requested vote from peers 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:33.853339 24452 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38722 (request call id 354) took 1635 ms. Trace:
I20260809 15:46:33.761698 24448 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38722 (request call id 356) took 1539 ms. Trace:
I20260809 15:46:33.729457 23393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df20671a7b874dbb98c0ef28371c1308" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:33.853716 25694 raft_consensus.cc:504] T ae6d2c08a7a94f58ac8dd698e3fd456c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:33.854486 23393 raft_consensus.cc:2379] T df20671a7b874dbb98c0ef28371c1308 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c in current term 3: Already voted for candidate c0f38c1908414566aa9115bb2cbdd85d in this term.
I20260809 15:46:33.764225 23399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "538a228ddfb54bc6a36528d5ff07597b" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:33.738382 23276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3296ef50be254fc29cfc77d22a8f7a23" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:33.855500 23399 raft_consensus.cc:3037] T 538a228ddfb54bc6a36528d5ff07597b P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:46:33.855777 23393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9e7cadcd24d45fcbfa84209225e8f2d" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:33.733796 25868 leader_election.cc:288] T d2d4f749fcfd4fb3bfb8115af3471598 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 4 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:33.768574 25919 raft_consensus.cc:3037] T 563daf42d7784db195ea99c3cb21b370 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:33.743438 25478 raft_consensus.cc:3037] T e71a05ba1d4d42bd95c3f85f9313399e P 61e69695258f48baa984c9d395e8ce3f [term 2 FOLLOWER]: Advancing to term 3
W20260809 15:46:33.771962 24406 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 73002.7us
I20260809 15:46:33.831743 26026 raft_consensus.cc:2781] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 15:46:33.777100 23522 raft_consensus.cc:2454] T 0acc47ce21c3422698d1ef7277c3ed63 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 3.
W20260809 15:46:33.775579 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 75222.3us
I20260809 15:46:33.859931 23228 leader_election.cc:302] T 0c29937a80724ddb9d07a27534a146f3 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters: c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:33.860729 23228 leader_election.cc:302] T 00a1881267a14ba8b78c81040f17a014 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:33.861690 23228 leader_election.cc:302] T d0721a4403a440f383318a3cea9d1442 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:46:33.862254 23228 leader_election.cc:302] T 99bdb93503db4de4a1cfb0161ea0c07c P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:46:33.694729 23529 raft_consensus.cc:2379] T 28e17dcb5efe45b8b6a8c42da03b4819 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d in current term 3: Already voted for candidate 65c36d164b0540f099d592f38307ee6c in this term.
I20260809 15:46:33.775489 25848 raft_consensus.cc:2726] T e14d118b2eaa42aa8a7cd4025aba7128 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20260809 15:46:33.780118 24410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 76186.7us
I20260809 15:46:33.706208 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cef3e7277d7d4760824a3436ed4a847d" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:33.834467 26001 raft_consensus.cc:2726] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260809 15:46:33.722607 23270 raft_consensus.cc:2379] T ba56725e42f74a1baa4c04d9660727f9 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d in current term 3: Already voted for candidate 65c36d164b0540f099d592f38307ee6c in this term.
I20260809 15:46:33.820535 23402 raft_consensus.cc:2362] T ba56725e42f74a1baa4c04d9660727f9 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 3. Re-sending same reply.
I20260809 15:46:33.783542 23397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0acc47ce21c3422698d1ef7277c3ed63" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:33.822652 23273 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53554 (request call id 1156) took 2071 ms. Trace:
I20260809 15:46:33.780304 25898 raft_consensus.cc:2781] T 0a51c0baaa6e432997c9d3552af4f254 P 61e69695258f48baa984c9d395e8ce3f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260809 15:46:33.769218 25849 raft_consensus.cc:3037] T 00a1881267a14ba8b78c81040f17a014 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:46:33.781909 23528 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56770 (request call id 1173) took 1402 ms. Trace:
I20260809 15:46:33.784945 23285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e4ffcf83c234ae7910ddef3ec0b4bda" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:33.782325 23526 raft_consensus.cc:2454] T 472816994d814cb198b69e5248697440 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 3.
I20260809 15:46:33.781980 25772 leader_election.cc:288] T 05ad7d8fdca64e3da527280ca5be9280 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
W20260809 15:46:33.828164 23223 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.68:34689, user_credentials={real_user=slave}} blocked reactor thread for 93753us
I20260809 15:46:33.827404 23404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472816994d814cb198b69e5248697440" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:33.894752 23273 rpcz_store.cc:276] 0809 15:46:31.751393 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:31.751635 (+ 242us) service_pool.cc:227] Handling call
0809 15:46:33.577364 (+1825729us) spinlock_profiling.cc:243] Waited 13.4 ms on lock 0x7b5800003398. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e613cf11 00007f34e62c0851 00007f34e62c0702 00007f34ef1b0b09 00007f34ee0ec730 00007f34ee1b279b 00007f34ee1b2821 00007f34ee1762b0 00007f34ee1617f9 00007f34ee1612a2 00007f34e7847684
0809 15:46:33.577556 (+ 192us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:33.684491 (+106935us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:46:33.684525 (+ 34us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 15:46:33.684564 (+ 39us) raft_consensus.cc:1631] Marking committed up to 3
0809 15:46:33.684591 (+ 27us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:33.822434 (+137843us) spinlock_profiling.cc:243] Waited 107 ms on lock 0x7b540030fa54. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e9220f51 00007f34e921f99e 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772 00007f34e764a94c 00007f34e7649d86
0809 15:46:33.822492 (+ 58us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:33.822603 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":276339328}
I20260809 15:46:33.830376 23394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5c2e425a74b4e64ae70b5def750434c" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:33.766188 23286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df20671a7b874dbb98c0ef28371c1308" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:33.738535 23518 raft_consensus.cc:2454] T 0c48a3049897452383257aa757834fb5 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 1.
I20260809 15:46:33.838030 23519 raft_consensus.cc:2454] T f9e7cadcd24d45fcbfa84209225e8f2d P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 3.
I20260809 15:46:33.797729 25816 leader_election.cc:288] T 99bdb93503db4de4a1cfb0161ea0c07c P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:33.839053 23470 leader_election.cc:302] T f9e7cadcd24d45fcbfa84209225e8f2d P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d
W20260809 15:46:33.908046 23470 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 69114.6us
I20260809 15:46:33.822187 25951 raft_consensus.cc:2726] T 4bcac4b24e7a4448abdc934bba5a624e P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 15:46:33.909512 23285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7715d71dbac447b9351fc5eb36c5e80" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:33.822227 25956 raft_consensus.cc:504] T 953eea268e17491ca76f3a555860a512 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:33.845824 23472 leader_election.cc:302] T 07fcabb569414bddbd39df6cedd24e64 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters: 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:33.909592 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2d4f749fcfd4fb3bfb8115af3471598" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:33.828766 25501 raft_consensus.cc:3037] T 763e58d6ba644eea830a6c1664d39c9b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:33.849210 23521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df20671a7b874dbb98c0ef28371c1308" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
W20260809 15:46:33.819979 23353 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 101225us
I20260809 15:46:33.911908 23521 raft_consensus.cc:2379] T df20671a7b874dbb98c0ef28371c1308 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d in current term 3: Already voted for candidate 65c36d164b0540f099d592f38307ee6c in this term.
I20260809 15:46:33.786098 25573 raft_consensus.cc:3037] T cef3e7277d7d4760824a3436ed4a847d P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:46:33.863482 23529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2d4f749fcfd4fb3bfb8115af3471598" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:33.867961 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2424fd9bd7f14e508fd106cbfda786a4" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:33.868115 25477 raft_consensus.cc:2781] T 99bdb93503db4de4a1cfb0161ea0c07c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260809 15:46:33.848414 25913 raft_consensus.cc:504] T 0acc47ce21c3422698d1ef7277c3ed63 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:33.853744 24452 rpcz_store.cc:276] 0809 15:46:32.218335 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:33.297007 (+1078672us) service_pool.cc:227] Handling call
0809 15:46:33.297449 (+ 442us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:33.374970 (+ 77521us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:46:33.375004 (+ 34us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 15:46:33.375047 (+ 43us) raft_consensus.cc:1631] Marking committed up to 4
0809 15:46:33.375077 (+ 30us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:33.853051 (+477974us) spinlock_profiling.cc:243] Waited 77.3 ms on lock 0x7b540040f454. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e9220f51 00007f34e921f99e 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772 00007f34e764a94c 00007f34e7649d86
0809 15:46:33.853106 (+ 55us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:33.853248 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":177808768}
I20260809 15:46:33.869385 23395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3924b648461e463b9d7925e5671d68ae" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:33.869830 23393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df20671a7b874dbb98c0ef28371c1308" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:33.942559 23393 raft_consensus.cc:2379] T df20671a7b874dbb98c0ef28371c1308 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 in current term 3: Already voted for candidate c0f38c1908414566aa9115bb2cbdd85d in this term.
I20260809 15:46:33.943542 23393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad8a3b25d7b49e5a2142d5709edd0c7" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:33.944159 25992 leader_election.cc:288] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:33.824095 25931 consensus_queue.cc:1046] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.027s
I20260809 15:46:33.901479 26026 raft_consensus.cc:686] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 LEADER]: Becoming Leader. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Running, Role: LEADER
I20260809 15:46:33.845005 24453 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38722 (request call id 357) took 1370 ms. Trace:
I20260809 15:46:33.783244 25650 consensus_queue.cc:237] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 65c36d164b0540f099d592f38307ee6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:33.856151 23276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee02fff2c9c48a19f43eb358a89e7b0" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:33.894567 23270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05ad7d8fdca64e3da527280ca5be9280" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:33.895305 23522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad8a3b25d7b49e5a2142d5709edd0c7" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:33.897189 23528 rpcz_store.cc:276] 0809 15:46:32.379477 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:32.926172 (+546695us) service_pool.cc:227] Handling call
0809 15:46:33.781709 (+855537us) spinlock_profiling.cc:243] Waited 515 ms on lock 0x7b54000758d4. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e92242ae 00007f34ee161c27 00007f34e7849c04 00007f34e7849b92 00007f34e7849ac1 00007f34e7849a3c 00007f34e7848cf2 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:46:33.781843 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1184578432}
I20260809 15:46:33.898131 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998c6cee72cf40228c6dba064c0db2da" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:33.893956 23396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad8a3b25d7b49e5a2142d5709edd0c7" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:33.882287 24472 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38734 (request call id 202) took 1597 ms. Trace:
I20260809 15:46:33.823820 25695 raft_consensus.cc:3032] T 28e17dcb5efe45b8b6a8c42da03b4819 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 LEADER]: Stepping down as leader of term 2
I20260809 15:46:33.875236 23399 raft_consensus.cc:2454] T 538a228ddfb54bc6a36528d5ff07597b P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 4.
I20260809 15:46:34.091234 25695 raft_consensus.cc:729] T 28e17dcb5efe45b8b6a8c42da03b4819 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Running, Role: LEADER
I20260809 15:46:34.091430 23399 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55320 (request call id 1114) took 1091 ms. Trace:
I20260809 15:46:34.091758 23399 rpcz_store.cc:276] 0809 15:46:32.999944 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:33.763985 (+764041us) service_pool.cc:227] Handling call
0809 15:46:34.091380 (+327395us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:46:33.905016 23519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73f6f44c259547f1b05a3a434616fa54" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:34.092485 23519 raft_consensus.cc:2454] T 73f6f44c259547f1b05a3a434616fa54 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 3.
I20260809 15:46:33.906086 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3924b648461e463b9d7925e5671d68ae" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:34.092101 25695 consensus_queue.cc:260] T 28e17dcb5efe45b8b6a8c42da03b4819 P 67b3aeadfa7a432abe8af4d292e04ab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:34.093842 24471 raft_consensus.cc:2454] T 3924b648461e463b9d7925e5671d68ae P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 3.
I20260809 15:46:34.094076 25695 raft_consensus.cc:3037] T 28e17dcb5efe45b8b6a8c42da03b4819 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:33.906785 23273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99bdb93503db4de4a1cfb0161ea0c07c" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
W20260809 15:46:33.910789 23472 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 65033.5us
I20260809 15:46:33.888166 25793 leader_election.cc:288] T d1a134ba33ea4076bbaac0ae460c17ba P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 2 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:33.913128 23285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0741ab44d99439abc3aa3bdd27ee160" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:33.866783 24410 leader_election.cc:302] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c; no voters:
I20260809 15:46:34.095935 23285 raft_consensus.cc:2379] T c0741ab44d99439abc3aa3bdd27ee160 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c in current term 3: Already voted for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in this term.
W20260809 15:46:34.095945 24410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 229238us
I20260809 15:46:34.096386 23517 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56770 (request call id 1167) took 1721 ms. Trace:
W20260809 15:46:33.914687 23400 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55320 (request call id 1094) took 2167 ms (client timeout 1990 ms). Trace:
I20260809 15:46:33.878108 25942 raft_consensus.cc:504] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:33.915959 23277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b30f28311964a0b9149a50495e83d8f" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:33.916328 23404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2d4f749fcfd4fb3bfb8115af3471598" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:33.918862 23286 raft_consensus.cc:2379] T df20671a7b874dbb98c0ef28371c1308 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d in current term 3: Already voted for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in this term.
I20260809 15:46:33.882046 25883 raft_consensus.cc:2781] T d0721a4403a440f383318a3cea9d1442 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260809 15:46:33.889426 23392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a823d35f4f348d1b52ba9e6d80b00d2" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:33.881765 25929 raft_consensus.cc:2781] T 0c29937a80724ddb9d07a27534a146f3 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260809 15:46:33.866765 25863 raft_consensus.cc:2726] T 28e17dcb5efe45b8b6a8c42da03b4819 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260809 15:46:33.915048 23274 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53554 (request call id 1166) took 1708 ms. Trace:
I20260809 15:46:33.926460 23527 raft_consensus.cc:2379] T 2424fd9bd7f14e508fd106cbfda786a4 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d in current term 4: Already voted for candidate 65c36d164b0540f099d592f38307ee6c in this term.
I20260809 15:46:33.780454 25924 consensus_queue.cc:237] T 3296ef50be254fc29cfc77d22a8f7a23 P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:33.935107 23394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "953eea268e17491ca76f3a555860a512" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:33.867442 25999 leader_election.cc:288] T 3924b648461e463b9d7925e5671d68ae P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:33.936987 23529 raft_consensus.cc:2379] T d2d4f749fcfd4fb3bfb8115af3471598 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 in current term 4: Already voted for candidate 65c36d164b0540f099d592f38307ee6c in this term.
I20260809 15:46:33.938817 23470 leader_election.cc:302] T 0c29937a80724ddb9d07a27534a146f3 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:46:33.883587 25694 leader_election.cc:288] T ae6d2c08a7a94f58ac8dd698e3fd456c P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:33.896948 25898 raft_consensus.cc:482] T 0a51c0baaa6e432997c9d3552af4f254 P 61e69695258f48baa984c9d395e8ce3f [term 2 FOLLOWER]: Starting leader election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
I20260809 15:46:33.934190 25477 raft_consensus.cc:482] T 99bdb93503db4de4a1cfb0161ea0c07c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:46:33.941485 24451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0741ab44d99439abc3aa3bdd27ee160" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f"
I20260809 15:46:33.944065 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c29937a80724ddb9d07a27534a146f3" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:33.865999 26041 raft_consensus.cc:2726] T 00a1881267a14ba8b78c81040f17a014 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260809 15:46:33.883531 24455 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0741ab44d99439abc3aa3bdd27ee160" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:33.952383 23397 raft_consensus.cc:2379] T 0acc47ce21c3422698d1ef7277c3ed63 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c in current term 3: Already voted for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in this term.
I20260809 15:46:33.920591 25772 raft_consensus.cc:2726] T f9e7cadcd24d45fcbfa84209225e8f2d P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 15:46:33.955807 23521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df20671a7b874dbb98c0ef28371c1308" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:33.992857 24472 rpcz_store.cc:276] 0809 15:46:32.285279 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:33.613048 (+1327769us) service_pool.cc:227] Handling call
0809 15:46:33.633490 (+ 20442us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 15:46:33.635718 (+ 2228us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:46:33.635750 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 15:46:33.636185 (+ 435us) log.cc:842] Serialized 26 byte log entry
0809 15:46:33.766756 (+130571us) raft_consensus.cc:1631] Marking committed up to 4
0809 15:46:33.766791 (+ 35us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:33.766849 (+ 58us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 15:46:33.882101 (+115252us) raft_consensus.cc:1671] finished
0809 15:46:33.882120 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:33.882242 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1863,"raft.queue_time_us":2124,"raft.run_cpu_time_us":104,"raft.run_wall_time_us":105,"thread_start_us":130329,"threads_started":1,"wal-append.queue_time_us":231651}
I20260809 15:46:34.139761 25913 leader_election.cc:288] T 0acc47ce21c3422698d1ef7277c3ed63 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:34.105418 26065 raft_consensus.cc:3032] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P c0f38c1908414566aa9115bb2cbdd85d [term 3 LEADER]: Stepping down as leader of term 3
I20260809 15:46:34.077634 26026 consensus_queue.cc:237] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:34.097041 23472 leader_election.cc:302] T df20671a7b874dbb98c0ef28371c1308 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:34.161144 23404 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55332 (request call id 1191) took 1236 ms. Trace:
I20260809 15:46:34.161499 23404 rpcz_store.cc:276] 0809 15:46:32.924268 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:33.905824 (+981556us) service_pool.cc:227] Handling call
0809 15:46:34.161091 (+255267us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:46:34.162449 23404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2424fd9bd7f14e508fd106cbfda786a4" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:34.163116 23404 raft_consensus.cc:2379] T 2424fd9bd7f14e508fd106cbfda786a4 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c in current term 4: Already voted for candidate c0f38c1908414566aa9115bb2cbdd85d in this term.
I20260809 15:46:33.951604 23395 raft_consensus.cc:2454] T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 4.
I20260809 15:46:34.096098 25979 raft_consensus.cc:2781] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260809 15:46:34.081010 23270 raft_consensus.cc:3037] T 05ad7d8fdca64e3da527280ca5be9280 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:34.164320 25979 raft_consensus.cc:482] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:46:34.166658 25979 raft_consensus.cc:3037] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:46:34.166772 25942 leader_election.cc:288] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:34.083163 25868 raft_consensus.cc:482] T 294d4bcc1e7a477cb56455581277a60f P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:34.167550 25868 raft_consensus.cc:504] T 294d4bcc1e7a477cb56455581277a60f P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:34.160676 23286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5c2e425a74b4e64ae70b5def750434c" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:34.170166 25868 leader_election.cc:288] T 294d4bcc1e7a477cb56455581277a60f P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:34.083843 23522 raft_consensus.cc:3037] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:46:34.171286 23392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ae14a74a26f434db08c652e097385c8" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:34.172435 23392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df20671a7b874dbb98c0ef28371c1308" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:34.173318 23392 raft_consensus.cc:2379] T df20671a7b874dbb98c0ef28371c1308 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 in current term 3: Already voted for candidate c0f38c1908414566aa9115bb2cbdd85d in this term.
I20260809 15:46:34.174791 23392 raft_consensus.cc:1226] T f7715d71dbac447b9351fc5eb36c5e80 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Rejecting Update request from peer 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 2. Current term is 3. Ops: []
I20260809 15:46:34.176064 25898 raft_consensus.cc:3037] T 0a51c0baaa6e432997c9d3552af4f254 P 61e69695258f48baa984c9d395e8ce3f [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:34.177500 23286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998c6cee72cf40228c6dba064c0db2da" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
W20260809 15:46:34.178323 23470 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 239573us
I20260809 15:46:34.181376 23402 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55320 (request call id 1120) took 1180 ms. Trace:
I20260809 15:46:34.181722 23402 rpcz_store.cc:276] 0809 15:46:33.001396 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:33.921654 (+920258us) service_pool.cc:227] Handling call
0809 15:46:34.181317 (+259663us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:46:34.182855 23402 raft_consensus.cc:1226] T c5c2e425a74b4e64ae70b5def750434c P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Rejecting Update request from peer 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 2. Current term is 3. Ops: []
I20260809 15:46:34.163849 23404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73f6f44c259547f1b05a3a434616fa54" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
W20260809 15:46:34.183825 23400 rpcz_store.cc:269] 0809 15:46:31.747210 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:32.199475 (+452265us) service_pool.cc:227] Handling call
0809 15:46:33.914489 (+1715014us) spinlock_profiling.cc:243] Waited 976 ms on lock 0x7b540009f6d4. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e92242ae 00007f34ee161c27 00007f34e7849c04 00007f34e7849b92 00007f34e7849ac1 00007f34e7849a3c 00007f34e7848cf2 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:46:33.914623 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2245701504}
I20260809 15:46:34.184547 23404 raft_consensus.cc:1226] T 953eea268e17491ca76f3a555860a512 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Rejecting Update request from peer 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 2. Current term is 3. Ops: []
I20260809 15:46:34.164593 23395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "213543cdbe814c89a17a963469922c1b" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:34.185482 26005 raft_consensus.cc:2781] T 0c29937a80724ddb9d07a27534a146f3 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260809 15:46:34.168098 23283 raft_consensus.cc:2379] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 61e69695258f48baa984c9d395e8ce3f in current term 4: Already voted for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in this term.
I20260809 15:46:34.085269 23276 raft_consensus.cc:2454] T cee02fff2c9c48a19f43eb358a89e7b0 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 2.
I20260809 15:46:33.949034 24453 rpcz_store.cc:276] 0809 15:46:32.468965 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:33.649297 (+1180332us) service_pool.cc:227] Handling call
0809 15:46:33.675487 (+ 26190us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:33.826094 (+150607us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:33.839752 (+ 13658us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:46:33.901840 25841 consensus_queue.cc:1035] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.048s
I20260809 15:46:33.937752 25816 raft_consensus.cc:2781] T 07fcabb569414bddbd39df6cedd24e64 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260809 15:46:34.094892 23468 leader_election.cc:302] T 3924b648461e463b9d7925e5671d68ae P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c; no voters:
I20260809 15:46:34.096789 24410 leader_election.cc:302] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:46:34.096715 23285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7baa2492df0340c4a5ac241959ce1ac3" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:34.096725 23517 rpcz_store.cc:276] 0809 15:46:32.375219 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:32.925866 (+550647us) service_pool.cc:227] Handling call
0809 15:46:34.096213 (+1170347us) spinlock_profiling.cc:243] Waited 333 ms on lock 0x7b5400055fd4. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e92242ae 00007f34ee161c27 00007f34e7849c04 00007f34e7849b92 00007f34e7849ac1 00007f34e7849a3c 00007f34e7848cf2 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:46:34.096332 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":766774656}
I20260809 15:46:34.178267 23286 raft_consensus.cc:2454] T 998c6cee72cf40228c6dba064c0db2da P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 5.
I20260809 15:46:34.185730 23395 raft_consensus.cc:2379] T 213543cdbe814c89a17a963469922c1b P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c in current term 3: Already voted for candidate c0f38c1908414566aa9115bb2cbdd85d in this term.
W20260809 15:46:34.192332 24410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 95616.8us
I20260809 15:46:34.084043 25899 leader_election.cc:288] T 213543cdbe814c89a17a963469922c1b P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:34.183369 23402 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55332 (request call id 1174) took 2491 ms. Trace:
I20260809 15:46:34.189941 23519 raft_consensus.cc:1226] T d2d4f749fcfd4fb3bfb8115af3471598 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Rejecting Update request from peer c0f38c1908414566aa9115bb2cbdd85d for earlier term 3. Current term is 4. Ops: []
I20260809 15:46:34.191505 23394 raft_consensus.cc:2379] T 953eea268e17491ca76f3a555860a512 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c in current term 3: Already voted for candidate c0f38c1908414566aa9115bb2cbdd85d in this term.
I20260809 15:46:34.226344 23402 rpcz_store.cc:276] 0809 15:46:31.691691 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:34.182323 (+2490632us) service_pool.cc:227] Handling call
0809 15:46:34.182699 (+ 376us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:34.183161 (+ 462us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:34.183319 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:46:34.190378 23404 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55332 (request call id 1176) took 2467 ms. Trace:
I20260809 15:46:34.228070 23404 rpcz_store.cc:276] 0809 15:46:31.723173 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:34.184036 (+2460863us) service_pool.cc:227] Handling call
0809 15:46:34.184403 (+ 367us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:34.190132 (+ 5729us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:34.190333 (+ 201us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:46:34.189986 23522 raft_consensus.cc:2454] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 4.
I20260809 15:46:34.110338 23115 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:56222 (request call id 244) took 5888 ms. Trace:
W20260809 15:46:34.192250 23468 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.68:34689, user_credentials={real_user=slave}} blocked reactor thread for 97440.8us
I20260809 15:46:34.227805 24410 leader_election.cc:302] T 472816994d814cb198b69e5248697440 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c; no voters: c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:34.229729 24410 leader_election.cc:302] T 0c48a3049897452383257aa757834fb5 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c; no voters:
I20260809 15:46:34.192947 23285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2d4f749fcfd4fb3bfb8115af3471598" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:34.230880 23285 raft_consensus.cc:3037] T d2d4f749fcfd4fb3bfb8115af3471598 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:46:33.248840 21289 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 15:46:34.192113 23277 raft_consensus.cc:3037] T 4b30f28311964a0b9149a50495e83d8f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:34.169844 23274 rpcz_store.cc:276] 0809 15:46:32.207018 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:33.158967 (+951949us) service_pool.cc:227] Handling call
0809 15:46:33.914910 (+755943us) spinlock_profiling.cc:243] Waited 84.6 ms on lock 0x7b540008b454. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e92242ae 00007f34ee161c27 00007f34e7849c04 00007f34e7849b92 00007f34e7849ac1 00007f34e7849a3c 00007f34e7848cf2 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:46:33.914997 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":194552704}
I20260809 15:46:34.094798 23396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998c6cee72cf40228c6dba064c0db2da" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:34.232190 23274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e17dcb5efe45b8b6a8c42da03b4819" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:34.160688 23527 raft_consensus.cc:1226] T 6ae14a74a26f434db08c652e097385c8 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Rejecting Update request from peer c0f38c1908414566aa9115bb2cbdd85d for earlier term 3. Current term is 4. Ops: []
I20260809 15:46:34.191008 23399 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52856 (request call id 90) took 2325 ms. Trace:
I20260809 15:46:34.202646 23470 leader_election.cc:302] T cee02fff2c9c48a19f43eb358a89e7b0 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 61e69695258f48baa984c9d395e8ce3f
I20260809 15:46:34.034845 24452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c48a3049897452383257aa757834fb5" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:33.950918 23393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3e94b7e50754a81b3d72a07476cb7b9" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:33.856707 24448 rpcz_store.cc:276] 0809 15:46:32.222212 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:33.421007 (+1198795us) service_pool.cc:227] Handling call
0809 15:46:33.760786 (+339779us) spinlock_profiling.cc:243] Waited 7.91 ms on lock 0x7b5800005498. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e613cf11 00007f34e62c0851 00007f34e62c0702 00007f34ef1b0b09 00007f34ee0ec730 00007f34ee1b279b 00007f34ee1b2821 00007f34ee1762b0 00007f34ee1617f9 00007f34ee1612a2 00007f34e7847684
0809 15:46:33.761071 (+ 285us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:33.761299 (+ 228us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:46:33.761349 (+ 50us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 15:46:33.761397 (+ 48us) raft_consensus.cc:1631] Marking committed up to 4
0809 15:46:33.761428 (+ 31us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:33.761487 (+ 59us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:33.761623 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":18191744}
I20260809 15:46:34.189492 23529 raft_consensus.cc:1226] T f9e7cadcd24d45fcbfa84209225e8f2d P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Rejecting Update request from peer c0f38c1908414566aa9115bb2cbdd85d for earlier term 2. Current term is 3. Ops: []
W20260809 15:46:34.433831 25905 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.541s user 0.039s sys 0.062s
I20260809 15:46:34.190217 23270 raft_consensus.cc:2454] T 05ad7d8fdca64e3da527280ca5be9280 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 3.
I20260809 15:46:34.471629 23404 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52856 (request call id 94) took 2151 ms. Trace:
I20260809 15:46:34.471909 23270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cef3e7277d7d4760824a3436ed4a847d" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:34.192617 23521 raft_consensus.cc:2379] T df20671a7b874dbb98c0ef28371c1308 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 in current term 3: Already voted for candidate 65c36d164b0540f099d592f38307ee6c in this term.
W20260809 15:46:34.195815 23472 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 98853.1us
I20260809 15:46:34.471997 23404 rpcz_store.cc:276] 0809 15:46:32.319758 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:34.298380 (+1978622us) service_pool.cc:227] Handling call
0809 15:46:34.471162 (+172782us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:34.471341 (+ 179us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:46:34.471368 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 15:46:34.471403 (+ 35us) raft_consensus.cc:1631] Marking committed up to 5
0809 15:46:34.471430 (+ 27us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:34.471479 (+ 49us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:34.471578 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:46:34.473474 23472 leader_election.cc:302] T c5c2e425a74b4e64ae70b5def750434c P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:34.188253 25477 raft_consensus.cc:3037] T 99bdb93503db4de4a1cfb0161ea0c07c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:34.187237 23276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e14d118b2eaa42aa8a7cd4025aba7128" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
W20260809 15:46:34.427807 23353 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 345161us
I20260809 15:46:34.190053 25929 raft_consensus.cc:482] T 0c29937a80724ddb9d07a27534a146f3 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:46:34.107076 24468 raft_consensus.cc:1226] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 61e69695258f48baa984c9d395e8ce3f [term 2 FOLLOWER]: Rejecting Update request from peer c0f38c1908414566aa9115bb2cbdd85d for earlier term 1. Current term is 2. Ops: []
I20260809 15:46:34.192533 26001 raft_consensus.cc:2781] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Leader election won for term 4
I20260809 15:46:34.402524 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f9d6565aa4c45a3b3f01210d0eb71f7" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:33.953191 25900 raft_consensus.cc:504] T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:34.191937 25816 raft_consensus.cc:482] T 07fcabb569414bddbd39df6cedd24e64 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Starting leader election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
I20260809 15:46:34.193181 25868 raft_consensus.cc:2781] T 3924b648461e463b9d7925e5671d68ae P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260809 15:46:33.950330 25885 raft_consensus.cc:504] T 297c82ed906d4b48b184985eed3de601 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:34.434932 23399 rpcz_store.cc:276] 0809 15:46:31.865956 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:34.190317 (+2324361us) service_pool.cc:227] Handling call
0809 15:46:34.190598 (+ 281us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:34.190741 (+ 143us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:46:34.190765 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 15:46:34.190798 (+ 33us) raft_consensus.cc:1631] Marking committed up to 5
0809 15:46:34.190828 (+ 30us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:34.190877 (+ 49us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:34.190963 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:46:34.505477 23404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a88421dba80d4d3388aa5eb62e23853b" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:34.510666 23404 raft_consensus.cc:2362] T a88421dba80d4d3388aa5eb62e23853b P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 3. Re-sending same reply.
I20260809 15:46:34.434579 24410 leader_election.cc:302] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c; no voters: c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:34.360869 25992 raft_consensus.cc:2781] T 0c48a3049897452383257aa757834fb5 P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260809 15:46:34.434509 23468 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.68:34689, user_credentials={real_user=slave}} blocked reactor thread for 204704us
I20260809 15:46:34.511783 23468 leader_election.cc:302] T c0741ab44d99439abc3aa3bdd27ee160 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 61e69695258f48baa984c9d395e8ce3f, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:34.443799 23526 raft_consensus.cc:1226] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Rejecting Update request from peer c0f38c1908414566aa9115bb2cbdd85d for earlier term 3. Current term is 4. Ops: [3.5-3.5]
W20260809 15:46:34.454936 23470 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 252530us
I20260809 15:46:34.193981 26005 raft_consensus.cc:482] T 0c29937a80724ddb9d07a27534a146f3 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:46:34.435086 23396 raft_consensus.cc:2454] T 998c6cee72cf40228c6dba064c0db2da P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 5.
W20260809 15:46:34.433593 23228 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.571s user 0.025s sys 0.075s
W20260809 15:46:34.515736 23228 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.653s user 0.027s sys 0.076s
W20260809 15:46:34.516005 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 653751us
I20260809 15:46:34.516582 23228 leader_election.cc:302] T 0acc47ce21c3422698d1ef7277c3ed63 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:46:34.517329 23228 leader_election.cc:302] T f9e7cadcd24d45fcbfa84209225e8f2d P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:46:34.453930 23522 raft_consensus.cc:1226] T 4b30f28311964a0b9149a50495e83d8f P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Rejecting Update request from peer c0f38c1908414566aa9115bb2cbdd85d for earlier term 1. Current term is 2. Ops: []
I20260809 15:46:34.188335 25883 raft_consensus.cc:482] T d0721a4403a440f383318a3cea9d1442 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Starting leader election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:46:34.248245 23392 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55332 (request call id 1173) took 2653 ms. Trace:
I20260809 15:46:34.489110 25941 raft_consensus.cc:2726] T c5c2e425a74b4e64ae70b5def750434c P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 15:46:34.475998 23529 raft_consensus.cc:1261] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260809 15:46:34.476462 23521 raft_consensus.cc:1226] T 05ad7d8fdca64e3da527280ca5be9280 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Rejecting Update request from peer c0f38c1908414566aa9115bb2cbdd85d for earlier term 2. Current term is 3. Ops: []
I20260809 15:46:34.428823 24451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e4ffcf83c234ae7910ddef3ec0b4bda" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f"
I20260809 15:46:34.482930 23472 leader_election.cc:302] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d
W20260809 15:46:34.554739 23472 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 71870.9us
I20260809 15:46:34.555490 23472 leader_election.cc:302] T 2424fd9bd7f14e508fd106cbfda786a4 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:34.505574 25816 raft_consensus.cc:3037] T 07fcabb569414bddbd39df6cedd24e64 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:46:34.224686 25694 consensus_queue.cc:1046] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.236s
I20260809 15:46:34.206859 26040 raft_consensus.cc:504] T d3be1e356b254d22b995be32f1c12f6f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:34.473655 25772 raft_consensus.cc:2726] T df20671a7b874dbb98c0ef28371c1308 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260809 15:46:34.509660 23283 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55434 (request call id 130) took 1158 ms. Trace:
I20260809 15:46:34.569263 23283 rpcz_store.cc:276] 0809 15:46:33.351206 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:33.473930 (+122724us) service_pool.cc:227] Handling call
0809 15:46:33.681584 (+207654us) spinlock_profiling.cc:243] Waited 6.08 ms on lock 0x7b5800003398. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e613cf11 00007f34e62c0851 00007f34e62c0702 00007f34ef1b0b09 00007f34ee0ec730 00007f34ee1b279b 00007f34ee1b2821 00007f34ee1621b0 00007f34ee161a94 00007f34e7849c04 00007f34e7849b92
0809 15:46:34.509497 (+827913us) spinlock_profiling.cc:243] Waited 420 ms on lock 0x7b540008a054. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e92242ae 00007f34ee161c27 00007f34e7849c04 00007f34e7849b92 00007f34e7849ac1 00007f34e7849a3c 00007f34e7848cf2 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:46:34.509611 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":981076096}
I20260809 15:46:34.198295 26065 raft_consensus.cc:729] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P c0f38c1908414566aa9115bb2cbdd85d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:46:34.512743 23519 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56766 (request call id 1701) took 1006 ms. Trace:
I20260809 15:46:34.515545 25992 raft_consensus.cc:482] T 0c48a3049897452383257aa757834fb5 P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Starting leader election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
I20260809 15:46:34.536368 25883 raft_consensus.cc:3037] T d0721a4403a440f383318a3cea9d1442 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:46:34.489480 25929 raft_consensus.cc:3037] T 0c29937a80724ddb9d07a27534a146f3 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:46:34.515496 23396 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55332 (request call id 1197) took 1191 ms. Trace:
I20260809 15:46:34.575606 23396 rpcz_store.cc:276] 0809 15:46:33.323959 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:34.004242 (+680283us) service_pool.cc:227] Handling call
0809 15:46:34.515442 (+511200us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:46:34.529464 23394 raft_consensus.cc:1226] T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Rejecting Update request from peer 61e69695258f48baa984c9d395e8ce3f for earlier term 4. Current term is 5. Ops: [2.5-4.6]
W20260809 15:46:34.519158 24410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 84670.3us
I20260809 15:46:34.533427 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1a134ba33ea4076bbaac0ae460c17ba" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:34.561089 25956 leader_election.cc:288] T 953eea268e17491ca76f3a555860a512 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
W20260809 15:46:34.470347 25905 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.577s user 0.039s sys 0.062s
I20260809 15:46:34.537072 23522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "213543cdbe814c89a17a963469922c1b" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:34.579169 23282 raft_consensus.cc:2379] T d3be1e356b254d22b995be32f1c12f6f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 61e69695258f48baa984c9d395e8ce3f in current term 2: Already voted for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in this term.
I20260809 15:46:34.579564 23522 raft_consensus.cc:2379] T 213543cdbe814c89a17a963469922c1b P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d in current term 3: Already voted for candidate 65c36d164b0540f099d592f38307ee6c in this term.
W20260809 15:46:34.568120 25478 consensus_meta.cc:287] T e71a05ba1d4d42bd95c3f85f9313399e P 61e69695258f48baa984c9d395e8ce3f: Time spent flushing consensus metadata: real 0.685s user 0.004s sys 0.004s
I20260809 15:46:34.583595 23396 raft_consensus.cc:1226] T 3296ef50be254fc29cfc77d22a8f7a23 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Rejecting Update request from peer 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 1. Current term is 2. Ops: []
I20260809 15:46:34.537985 23228 leader_election.cc:302] T ba56725e42f74a1baa4c04d9660727f9 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:34.538841 23399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "213543cdbe814c89a17a963469922c1b" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:34.584482 23399 raft_consensus.cc:2379] T 213543cdbe814c89a17a963469922c1b P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c in current term 3: Already voted for candidate c0f38c1908414566aa9115bb2cbdd85d in this term.
I20260809 15:46:34.584744 23396 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55332 (request call id 1179) took 1750 ms. Trace:
I20260809 15:46:34.534199 25948 consensus_queue.cc:1046] T 6ae14a74a26f434db08c652e097385c8 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.041s
I20260809 15:46:34.585083 23396 rpcz_store.cc:276] 0809 15:46:32.833851 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:34.580932 (+1747081us) service_pool.cc:227] Handling call
0809 15:46:34.581206 (+ 274us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:34.584546 (+ 3340us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:34.584700 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:46:34.568912 26082 raft_consensus.cc:482] T a3622f6e3e3641cea1960e3a31498bb7 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:34.532923 25852 raft_consensus.cc:3037] T c0741ab44d99439abc3aa3bdd27ee160 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:34.544744 25885 leader_election.cc:288] T 297c82ed906d4b48b184985eed3de601 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:34.510821 25868 raft_consensus.cc:482] T 3924b648461e463b9d7925e5671d68ae P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:46:34.509374 23286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1a134ba33ea4076bbaac0ae460c17ba" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:34.548151 23285 raft_consensus.cc:2454] T d2d4f749fcfd4fb3bfb8115af3471598 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 4.
I20260809 15:46:34.548810 25898 raft_consensus.cc:504] T 0a51c0baaa6e432997c9d3552af4f254 P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:34.555415 23274 raft_consensus.cc:2454] T 28e17dcb5efe45b8b6a8c42da03b4819 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 3.
W20260809 15:46:34.479645 23351 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.529s user 0.019s sys 0.053s
I20260809 15:46:34.218750 25501 raft_consensus.cc:504] T 763e58d6ba644eea830a6c1664d39c9b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:34.560511 23472 leader_election.cc:302] T 953eea268e17491ca76f3a555860a512 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:34.561728 23277 raft_consensus.cc:2454] T 4b30f28311964a0b9149a50495e83d8f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 2.
I20260809 15:46:34.230353 23115 rpcz_store.cc:276] 0809 15:46:28.221967 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:28.222102 (+ 135us) service_pool.cc:227] Handling call
0809 15:46:34.110251 (+5888149us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0809 15:46:29.005018 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0809 15:46:29.097236 (+ 92218us) write_op.cc:432] Acquiring schema lock in shared mode
0809 15:46:29.100812 (+ 3576us) write_op.cc:435] Acquired schema lock
0809 15:46:29.107098 (+ 6286us) tablet.cc:660] Decoding operations
0809 15:46:29.188237 (+ 81139us) write_op.cc:620] Acquiring the partition lock for write op
0809 15:46:29.191755 (+ 3518us) write_op.cc:641] Partition lock acquired for write op
0809 15:46:29.201512 (+ 9757us) tablet.cc:683] Acquiring locks for 1 operations
0809 15:46:29.223195 (+ 21683us) tablet.cc:699] Row locks acquired
0809 15:46:29.224713 (+ 1518us) write_op.cc:260] PREPARE: finished
0809 15:46:29.277401 (+ 52688us) write_op.cc:270] Start()
0809 15:46:29.286404 (+ 9003us) write_op.cc:276] Timestamp: P: 1786290389244454 usec, L: 0
0809 15:46:29.288688 (+ 2284us) op_driver.cc:331] REPLICATION: starting
0809 15:46:29.362031 (+ 73343us) log.cc:842] Serialized 536 byte log entry
0809 15:46:29.510321 (+148290us) spinlock_profiling.cc:243] Waited 14.9 ms on lock 0x7b44000232cc. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e613cf11 00007f34e620aa95 00007f34e620b4b2 00007f34e8eb3511 00007f34e8eaee1a 00007f34e62da5fe 00007f34ef111a39 00007f34e62f78bb 00007f34e62f5cbf 00007f34e62f7e22 00007f34e8ea15bb
0809 15:46:31.137257 (+1626936us) op_driver.cc:444] REPLICATION: finished
0809 15:46:31.521494 (+384237us) write_op.cc:301] APPLY: starting
0809 15:46:31.525501 (+ 4007us) tablet.cc:1365] starting BulkCheckPresence
0809 15:46:31.538374 (+ 12873us) tablet.cc:1368] finished BulkCheckPresence
0809 15:46:31.545839 (+ 7465us) tablet.cc:1370] starting ApplyRowOperation cycle
0809 15:46:31.604310 (+ 58471us) tablet.cc:1381] finished ApplyRowOperation cycle
0809 15:46:31.663704 (+ 59394us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0809 15:46:31.674319 (+ 10615us) write_op.cc:312] APPLY: finished
0809 15:46:31.699129 (+ 24810us) log.cc:842] Serialized 23 byte log entry
0809 15:46:31.742822 (+ 43693us) write_op.cc:489] Releasing partition, row and schema locks
0809 15:46:31.742930 (+ 108us) write_op.cc:454] Released schema lock
0809 15:46:31.742977 (+ 47us) write_op.cc:341] FINISH: Updating metrics
0809 15:46:31.743938 (+ 961us) write_op.cc:489] Releasing partition, row and schema locks
0809 15:46:31.743970 (+ 32us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":2724,"reported_tablets":6,"child_traces":[["op",{"apply.queue_time_us":206559,"apply.run_cpu_time_us":34406,"apply.run_wall_time_us":279377,"num_ops":1,"prepare.queue_time_us":714058,"prepare.run_cpu_time_us":27806,"prepare.run_wall_time_us":600409,"raft.queue_time_us":654072,"raft.run_cpu_time_us":29058,"raft.run_wall_time_us":320598,"replication_time_us":1844728,"spinlock_wait_cycles":34297600,"thread_start_us":1203265,"threads_started":5,"wal-append.queue_time_us":520967,"wal-append.run_cpu_time_us":66598,"wal-append.run_wall_time_us":2723930}]]}
I20260809 15:46:34.562073 23397 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55320 (request call id 1118) took 1561 ms. Trace:
W20260809 15:46:34.535142 25919 consensus_meta.cc:287] T 563daf42d7784db195ea99c3cb21b370 P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent flushing consensus metadata: real 0.645s user 0.005s sys 0.003s
I20260809 15:46:34.565353 23404 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55332 (request call id 1178) took 1731 ms. Trace:
I20260809 15:46:34.582755 23395 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55332 (request call id 1177) took 1747 ms. Trace:
I20260809 15:46:34.607478 23404 rpcz_store.cc:276] 0809 15:46:32.833593 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:34.528624 (+1695031us) service_pool.cc:227] Handling call
0809 15:46:34.564949 (+ 36325us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:34.565094 (+ 145us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:46:34.565118 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 15:46:34.565153 (+ 35us) raft_consensus.cc:1631] Marking committed up to 2
0809 15:46:34.565181 (+ 28us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:34.565226 (+ 45us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:34.565307 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:46:34.509600 23393 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55320 (request call id 1123) took 1507 ms. Trace:
I20260809 15:46:34.527576 23520 raft_consensus.cc:1226] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 65c36d164b0540f099d592f38307ee6c [term 4 LEADER]: Rejecting Update request from peer 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 3. Current term is 4. Ops: []
I20260809 15:46:34.527544 26005 raft_consensus.cc:3037] T 0c29937a80724ddb9d07a27534a146f3 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:46:34.535784 25942 raft_consensus.cc:2781] T 0acc47ce21c3422698d1ef7277c3ed63 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader election won for term 3
I20260809 15:46:34.570380 23519 rpcz_store.cc:276] 0809 15:46:33.505881 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:34.093098 (+587217us) service_pool.cc:227] Handling call
0809 15:46:34.189779 (+ 96681us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:34.298505 (+108726us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:34.511809 (+213304us) spinlock_profiling.cc:243] Waited 96.4 ms on lock 0x7b54000906d0. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e921fa09 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:46:34.512501 (+ 692us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":221647744}
I20260809 15:46:34.478471 25967 raft_consensus.cc:2781] T cee02fff2c9c48a19f43eb358a89e7b0 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260809 15:46:34.553812 26038 raft_consensus.cc:2781] T f9e7cadcd24d45fcbfa84209225e8f2d P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader election won for term 3
I20260809 15:46:34.572788 23529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0acc47ce21c3422698d1ef7277c3ed63" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:34.562703 25650 raft_consensus.cc:2726] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260809 15:46:34.621287 26038 raft_consensus.cc:686] T f9e7cadcd24d45fcbfa84209225e8f2d P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 LEADER]: Becoming Leader. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Running, Role: LEADER
I20260809 15:46:34.499146 23276 raft_consensus.cc:2454] T e14d118b2eaa42aa8a7cd4025aba7128 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 4.
I20260809 15:46:34.622395 23276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ae14a74a26f434db08c652e097385c8" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:34.622184 26038 consensus_queue.cc:237] T f9e7cadcd24d45fcbfa84209225e8f2d P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:34.622970 23276 raft_consensus.cc:3037] T 6ae14a74a26f434db08c652e097385c8 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:46:34.557345 25746 raft_consensus.cc:2781] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Leader election won for term 4
I20260809 15:46:34.627575 25746 raft_consensus.cc:686] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 61e69695258f48baa984c9d395e8ce3f [term 4 LEADER]: Becoming Leader. State: Replica: 61e69695258f48baa984c9d395e8ce3f, State: Running, Role: LEADER
I20260809 15:46:34.628425 25746 consensus_queue.cc:237] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:34.604467 23286 raft_consensus.cc:3037] T d1a134ba33ea4076bbaac0ae460c17ba P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Advancing to term 2
W20260809 15:46:34.541020 26088 log.cc:924] Time spent T 16ca412a0dae4feebe85dd01bf1a9173 P 65c36d164b0540f099d592f38307ee6c: Append to log took a long time: real 0.060s user 0.000s sys 0.001s
I20260809 15:46:34.562784 25841 consensus_queue.cc:1046] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.021s
I20260809 15:46:34.587383 25694 raft_consensus.cc:2726] T ba56725e42f74a1baa4c04d9660727f9 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260809 15:46:34.587778 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 49876.5us
I20260809 15:46:34.553166 25931 consensus_queue.cc:1046] T f9e7cadcd24d45fcbfa84209225e8f2d P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.044s
I20260809 15:46:34.573698 25965 raft_consensus.cc:2726] T 2424fd9bd7f14e508fd106cbfda786a4 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260809 15:46:34.557163 25899 consensus_queue.cc:1046] T 4b30f28311964a0b9149a50495e83d8f P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.025s
I20260809 15:46:34.581259 25695 raft_consensus.cc:3032] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 LEADER]: Stepping down as leader of term 3
I20260809 15:46:34.597623 23402 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55332 (request call id 1181) took 1763 ms. Trace:
I20260809 15:46:34.643640 23402 rpcz_store.cc:276] 0809 15:46:32.834340 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:34.581456 (+1747116us) service_pool.cc:227] Handling call
0809 15:46:34.597222 (+ 15766us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:34.597360 (+ 138us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:46:34.597382 (+ 22us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 15:46:34.597417 (+ 35us) raft_consensus.cc:1631] Marking committed up to 4
0809 15:46:34.597444 (+ 27us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:34.597491 (+ 47us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:34.597576 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:46:34.577299 25992 raft_consensus.cc:3037] T 0c48a3049897452383257aa757834fb5 P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:34.592823 23521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "953eea268e17491ca76f3a555860a512" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
W20260809 15:46:34.600742 24475 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.520s user 0.000s sys 0.021s
W20260809 15:46:34.600920 23351 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.651s user 0.019s sys 0.053s
W20260809 15:46:34.646019 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 695900us
I20260809 15:46:34.611704 25941 raft_consensus.cc:3037] T 953eea268e17491ca76f3a555860a512 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:34.602887 23285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2424fd9bd7f14e508fd106cbfda786a4" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:34.650717 23285 raft_consensus.cc:2379] T 2424fd9bd7f14e508fd106cbfda786a4 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d in current term 4: Already voted for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in this term.
I20260809 15:46:34.603614 23277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "213543cdbe814c89a17a963469922c1b" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:34.603909 23399 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55320 (request call id 1119) took 1602 ms. Trace:
I20260809 15:46:34.630497 25883 raft_consensus.cc:504] T d0721a4403a440f383318a3cea9d1442 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:46:34.602988 24406 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 152433us
I20260809 15:46:34.605649 23397 rpcz_store.cc:276] 0809 15:46:33.000892 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:33.783189 (+782297us) service_pool.cc:227] Handling call
0809 15:46:34.561907 (+778718us) spinlock_profiling.cc:243] Waited 57 ms on lock 0x7b540025e254. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e92242ae 00007f34ee161c27 00007f34e7849c04 00007f34e7849b92 00007f34e7849ac1 00007f34e7849a3c 00007f34e7848cf2 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:46:34.562017 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":131001216}
W20260809 15:46:34.606841 23472 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 46401.5us
I20260809 15:46:34.683020 23397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a51c0baaa6e432997c9d3552af4f254" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:34.578576 25959 consensus_queue.cc:1046] T 05ad7d8fdca64e3da527280ca5be9280 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.026s
I20260809 15:46:34.598322 25868 raft_consensus.cc:3037] T 3924b648461e463b9d7925e5671d68ae P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Advancing to term 5
W20260809 15:46:34.602461 25573 consensus_meta.cc:287] T cef3e7277d7d4760824a3436ed4a847d P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent flushing consensus metadata: real 0.661s user 0.002s sys 0.007s
W20260809 15:46:34.602205 25875 consensus_meta.cc:287] T f7715d71dbac447b9351fc5eb36c5e80 P 65c36d164b0540f099d592f38307ee6c: Time spent flushing consensus metadata: real 0.875s user 0.010s sys 0.000s
I20260809 15:46:34.617127 23519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3e94b7e50754a81b3d72a07476cb7b9" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:34.588560 25478 raft_consensus.cc:504] T e71a05ba1d4d42bd95c3f85f9313399e P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:34.587548 26065 consensus_queue.cc:260] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P c0f38c1908414566aa9115bb2cbdd85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
W20260809 15:46:34.620592 25849 consensus_meta.cc:287] T 00a1881267a14ba8b78c81040f17a014 P 65c36d164b0540f099d592f38307ee6c: Time spent flushing consensus metadata: real 0.715s user 0.007s sys 0.001s
W20260809 15:46:34.620801 25943 consensus_meta.cc:287] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent flushing consensus metadata: real 0.791s user 0.009s sys 0.000s
I20260809 15:46:34.627286 25477 raft_consensus.cc:504] T 99bdb93503db4de4a1cfb0161ea0c07c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:34.584478 23283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3622f6e3e3641cea1960e3a31498bb7" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:34.621351 23529 raft_consensus.cc:2454] T 0acc47ce21c3422698d1ef7277c3ed63 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 3.
I20260809 15:46:34.642274 23404 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55332 (request call id 1198) took 1317 ms. Trace:
I20260809 15:46:34.634722 23228 leader_election.cc:302] T 73f6f44c259547f1b05a3a434616fa54 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:46:34.536098 23392 rpcz_store.cc:276] 0809 15:46:31.571698 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:34.174042 (+2602344us) service_pool.cc:227] Handling call
0809 15:46:34.174500 (+ 458us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:34.175131 (+ 631us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:34.224633 (+ 49502us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:46:34.649488 23351 leader_election.cc:302] T 2db057fd10e149a9895dff6ef8c9f038 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:34.651461 23399 rpcz_store.cc:276] 0809 15:46:33.001159 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:34.585217 (+1584058us) service_pool.cc:227] Handling call
0809 15:46:34.603491 (+ 18274us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:34.603652 (+ 161us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:46:34.603674 (+ 22us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 15:46:34.603711 (+ 37us) raft_consensus.cc:1631] Marking committed up to 2
0809 15:46:34.603737 (+ 26us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:34.603783 (+ 46us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:34.603866 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:46:34.568711 25913 consensus_queue.cc:1035] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.024s
I20260809 15:46:34.651482 23277 raft_consensus.cc:3037] T 213543cdbe814c89a17a963469922c1b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Advancing to term 3
W20260809 15:46:34.702431 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 67787.5us
I20260809 15:46:34.649137 25979 raft_consensus.cc:504] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:34.594887 26082 raft_consensus.cc:504] T a3622f6e3e3641cea1960e3a31498bb7 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:34.655323 23285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "294d4bcc1e7a477cb56455581277a60f" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:34.714289 24471 raft_consensus.cc:1226] T e71a05ba1d4d42bd95c3f85f9313399e P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Rejecting Update request from peer 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 2. Current term is 3. Ops: []
I20260809 15:46:34.714396 23285 raft_consensus.cc:2454] T 294d4bcc1e7a477cb56455581277a60f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 4.
I20260809 15:46:34.634938 24463 raft_consensus.cc:1226] T 0a51c0baaa6e432997c9d3552af4f254 P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Rejecting Update request from peer 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 2. Current term is 3. Ops: []
I20260809 15:46:34.633173 25948 consensus_queue.cc:1046] T d2d4f749fcfd4fb3bfb8115af3471598 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.026s
I20260809 15:46:34.654217 25900 leader_election.cc:288] T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 election: Requested vote from peers 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:34.664762 23276 raft_consensus.cc:2454] T 6ae14a74a26f434db08c652e097385c8 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 4.
I20260809 15:46:34.668196 25898 leader_election.cc:288] T 0a51c0baaa6e432997c9d3552af4f254 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 3 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:34.669118 26080 raft_consensus.cc:3032] T 6ae14a74a26f434db08c652e097385c8 P c0f38c1908414566aa9115bb2cbdd85d [term 3 LEADER]: Stepping down as leader of term 3
I20260809 15:46:34.642922 25695 raft_consensus.cc:729] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Running, Role: LEADER
I20260809 15:46:34.670295 23402 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55332 (request call id 1202) took 1329 ms. Trace:
I20260809 15:46:34.743464 23402 rpcz_store.cc:276] 0809 15:46:33.341413 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:34.644510 (+1303097us) service_pool.cc:227] Handling call
0809 15:46:34.669863 (+ 25353us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:34.670025 (+ 162us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:46:34.670049 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 15:46:34.670085 (+ 36us) raft_consensus.cc:1631] Marking committed up to 2
0809 15:46:34.670112 (+ 27us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:34.670155 (+ 43us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:34.670246 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:46:34.648761 25980 raft_consensus.cc:2781] T 472816994d814cb198b69e5248697440 P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260809 15:46:34.744344 23283 raft_consensus.cc:2379] T a3622f6e3e3641cea1960e3a31498bb7 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d in current term 3: Already voted for candidate 65c36d164b0540f099d592f38307ee6c in this term.
I20260809 15:46:34.634423 25942 raft_consensus.cc:686] T 0acc47ce21c3422698d1ef7277c3ed63 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 LEADER]: Becoming Leader. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Running, Role: LEADER
I20260809 15:46:34.609787 23393 rpcz_store.cc:276] 0809 15:46:33.002197 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:33.950620 (+948423us) service_pool.cc:227] Handling call
0809 15:46:34.506538 (+555918us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:46:34.612042 25919 raft_consensus.cc:504] T 563daf42d7784db195ea99c3cb21b370 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
W20260809 15:46:34.675936 26089 log.cc:924] Time spent T 5712f6b6fffc4cc5a8ddfbe63e4097df P 65c36d164b0540f099d592f38307ee6c: Append to log took a long time: real 0.077s user 0.000s sys 0.000s
I20260809 15:46:34.676513 23520 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56770 (request call id 1178) took 1353 ms. Trace:
I20260809 15:46:34.685925 23468 leader_election.cc:302] T 0c48a3049897452383257aa757834fb5 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 61e69695258f48baa984c9d395e8ce3f, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:34.700961 23404 rpcz_store.cc:276] 0809 15:46:33.324622 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:34.611251 (+1286629us) service_pool.cc:227] Handling call
0809 15:46:34.612703 (+ 1452us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:34.633304 (+ 20601us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:46:34.633335 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 15:46:34.633376 (+ 41us) raft_consensus.cc:1631] Marking committed up to 5
0809 15:46:34.633405 (+ 29us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:34.633981 (+ 576us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:34.634082 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:46:34.709543 24455 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "294d4bcc1e7a477cb56455581277a60f" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:34.710762 25953 raft_consensus.cc:3032] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P c0f38c1908414566aa9115bb2cbdd85d [term 1 LEADER]: Stepping down as leader of term 1
I20260809 15:46:34.712397 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3be1e356b254d22b995be32f1c12f6f" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f"
I20260809 15:46:34.715245 23285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d880de657e544468c82157769622a26" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
W20260809 15:46:34.769495 23517 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.570s user 0.014s sys 0.027s
W20260809 15:46:34.769804 23517 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.576s user 0.014s sys 0.027s
I20260809 15:46:34.690868 25573 raft_consensus.cc:504] T cef3e7277d7d4760824a3436ed4a847d P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:34.610507 23395 rpcz_store.cc:276] 0809 15:46:32.833292 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:34.427440 (+1594148us) service_pool.cc:227] Handling call
0809 15:46:34.510087 (+ 82647us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:34.552969 (+ 42882us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:46:34.556043 (+ 3074us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 15:46:34.559652 (+ 3609us) raft_consensus.cc:1631] Marking committed up to 1
0809 15:46:34.566379 (+ 6727us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:34.573554 (+ 7175us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:34.580681 (+ 7127us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:46:34.679284 25883 leader_election.cc:288] T d0721a4403a440f383318a3cea9d1442 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:34.791734 23517 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56766 (request call id 1712) took 1157 ms. Trace:
I20260809 15:46:34.690680 25875 raft_consensus.cc:2726] T f7715d71dbac447b9351fc5eb36c5e80 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 15:46:34.729918 23228 leader_election.cc:302] T 2424fd9bd7f14e508fd106cbfda786a4 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:34.691126 26040 leader_election.cc:288] T d3be1e356b254d22b995be32f1c12f6f P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 2 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:34.738955 23273 raft_consensus.cc:2379] T 99bdb93503db4de4a1cfb0161ea0c07c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c in current term 2: Already voted for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in this term.
I20260809 15:46:34.741518 23277 raft_consensus.cc:2454] T 213543cdbe814c89a17a963469922c1b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 3.
I20260809 15:46:34.618140 25967 raft_consensus.cc:482] T cee02fff2c9c48a19f43eb358a89e7b0 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Starting leader election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
I20260809 15:46:34.746969 23399 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52856 (request call id 103) took 1112 ms. Trace:
I20260809 15:46:34.711778 26065 raft_consensus.cc:3037] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:46:34.792076 23517 rpcz_store.cc:276] 0809 15:46:33.634413 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:34.192781 (+558368us) service_pool.cc:227] Handling call
0809 15:46:34.193035 (+ 254us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:34.193201 (+ 166us) raft_consensus.cc:1520] Early marking committed up to index 2
0809 15:46:34.193584 (+ 383us) log.cc:842] Serialized 14 byte log entry
0809 15:46:34.791443 (+597859us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 15:46:34.791498 (+ 55us) raft_consensus.cc:1631] Marking committed up to 2
0809 15:46:34.791539 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:34.791593 (+ 54us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:34.791688 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":21166,"prepare.queue_time_us":294,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":36,"raft.queue_time_us":24387,"raft.run_cpu_time_us":77,"raft.run_wall_time_us":78,"thread_start_us":576353,"threads_started":1}
I20260809 15:46:34.749068 23526 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56766 (request call id 1700) took 1232 ms. Trace:
W20260809 15:46:34.757948 23468 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.68:34689, user_credentials={real_user=slave}} blocked reactor thread for 72089us
I20260809 15:46:34.761606 23470 leader_election.cc:302] T 4564dfc6d4094c1b9240b367ccecf3b1 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters: c0f38c1908414566aa9115bb2cbdd85d
W20260809 15:46:34.864372 23470 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 102859us
I20260809 15:46:34.865060 23470 leader_election.cc:302] T 05ad7d8fdca64e3da527280ca5be9280 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters: c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:34.701330 25849 raft_consensus.cc:504] T 00a1881267a14ba8b78c81040f17a014 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:34.773442 23394 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52856 (request call id 92) took 2883 ms. Trace:
I20260809 15:46:34.742846 26080 raft_consensus.cc:729] T 6ae14a74a26f434db08c652e097385c8 P c0f38c1908414566aa9115bb2cbdd85d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
W20260809 15:46:34.720283 23345 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.68:34689, user_credentials={real_user=slave}} blocked reactor thread for 84775.2us
I20260809 15:46:34.701181 25943 raft_consensus.cc:504] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:34.757246 25852 raft_consensus.cc:2726] T c0741ab44d99439abc3aa3bdd27ee160 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260809 15:46:34.843916 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 114029us
I20260809 15:46:34.871671 23396 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55320 (request call id 1122) took 1869 ms. Trace:
I20260809 15:46:34.872669 23228 leader_election.cc:302] T d2d4f749fcfd4fb3bfb8115af3471598 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:34.873714 23228 leader_election.cc:302] T df20671a7b874dbb98c0ef28371c1308 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:34.787637 23404 raft_consensus.cc:1261] T 998c6cee72cf40228c6dba064c0db2da P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260809 15:46:34.874648 23404 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55320 (request call id 1140) took 1135 ms. Trace:
I20260809 15:46:34.874940 23404 rpcz_store.cc:276] 0809 15:46:33.739322 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:34.756860 (+1017538us) service_pool.cc:227] Handling call
0809 15:46:34.787338 (+ 30478us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:34.874417 (+ 87079us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:34.874598 (+ 181us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:46:34.845263 23277 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53554 (request call id 1177) took 1005 ms. Trace:
I20260809 15:46:34.875567 23472 leader_election.cc:302] T 0acc47ce21c3422698d1ef7277c3ed63 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:34.846603 24472 raft_consensus.cc:1226] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Rejecting Update request from peer 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20260809 15:46:34.847481 23270 raft_consensus.cc:2379] T cef3e7277d7d4760824a3436ed4a847d P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d in current term 4: Already voted for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in this term.
I20260809 15:46:34.847945 23399 rpcz_store.cc:276] 0809 15:46:33.634843 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:34.716404 (+1081561us) service_pool.cc:227] Handling call
0809 15:46:34.746548 (+ 30144us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:34.746706 (+ 158us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:46:34.746730 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 15:46:34.746764 (+ 34us) raft_consensus.cc:1631] Marking committed up to 5
0809 15:46:34.746791 (+ 27us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:34.746835 (+ 44us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:34.746921 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:46:34.713884 25694 raft_consensus.cc:2781] T 73f6f44c259547f1b05a3a434616fa54 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260809 15:46:34.768582 25953 raft_consensus.cc:729] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P c0f38c1908414566aa9115bb2cbdd85d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:46:34.850956 23526 rpcz_store.cc:276] 0809 15:46:33.505642 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:34.082185 (+576543us) service_pool.cc:227] Handling call
0809 15:46:34.240422 (+158237us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 15:46:34.527602 (+287180us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:34.737582 (+209980us) spinlock_profiling.cc:243] Waited 145 ms on lock 0x7b54000753d0. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e921fa09 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:46:34.737704 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":333886720}
I20260809 15:46:34.892505 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30c1740ef6874df0a332fbd6fd4b03ce" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:34.893615 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae6d2c08a7a94f58ac8dd698e3fd456c" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:34.774525 24453 raft_consensus.cc:1226] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Rejecting Update request from peer c0f38c1908414566aa9115bb2cbdd85d for earlier term 3. Current term is 4. Ops: [3.5-3.5]
I20260809 15:46:34.860095 23517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0741ab44d99439abc3aa3bdd27ee160" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:34.774761 25501 leader_election.cc:288] T 763e58d6ba644eea830a6c1664d39c9b P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 3 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:34.847539 25967 raft_consensus.cc:3037] T cee02fff2c9c48a19f43eb358a89e7b0 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Advancing to term 3
W20260809 15:46:34.842602 26006 consensus_meta.cc:287] T 2db057fd10e149a9895dff6ef8c9f038 P 65c36d164b0540f099d592f38307ee6c: Time spent flushing consensus metadata: real 1.001s user 0.010s sys 0.000s
I20260809 15:46:34.860646 23353 leader_election.cc:302] T 0acc47ce21c3422698d1ef7277c3ed63 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:46:34.866856 23470 leader_election.cc:302] T d2d4f749fcfd4fb3bfb8115af3471598 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters: c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:34.907953 23470 leader_election.cc:302] T 4b30f28311964a0b9149a50495e83d8f P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters: c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:34.908948 23470 leader_election.cc:302] T e14d118b2eaa42aa8a7cd4025aba7128 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 61e69695258f48baa984c9d395e8ce3f
I20260809 15:46:34.909927 23470 leader_election.cc:302] T 294d4bcc1e7a477cb56455581277a60f P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 61e69695258f48baa984c9d395e8ce3f
I20260809 15:46:34.866457 23394 rpcz_store.cc:276] 0809 15:46:31.876170 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:34.226430 (+2350260us) service_pool.cc:227] Handling call
0809 15:46:34.426747 (+200317us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 15:46:34.577555 (+150808us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:34.759035 (+181480us) spinlock_profiling.cc:243] Waited 102 ms on lock 0x7b54000023d4. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e9220f51 00007f34e921f99e 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772 00007f34e764a94c 00007f34e7649d86
0809 15:46:34.759208 (+ 173us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":235673344}
I20260809 15:46:34.876931 23472 leader_election.cc:302] T d3e94b7e50754a81b3d72a07476cb7b9 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:34.872006 23396 rpcz_store.cc:276] 0809 15:46:33.001933 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:34.585853 (+1583920us) service_pool.cc:227] Handling call
0809 15:46:34.586117 (+ 264us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:34.586288 (+ 171us) raft_consensus.cc:1520] Early marking committed up to index 3
0809 15:46:34.608347 (+ 22059us) log.cc:842] Serialized 14 byte log entry
0809 15:46:34.662227 (+ 53880us) spinlock_profiling.cc:243] Waited 1.27 ms on lock 0x7b18000482e0. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34ef1978b1 00007f34ef1a4661 00007f34e62da345 00007f34ef111a39 00007f34e62f78bb 00007f34e62f5cbf 00007f34e62f7e22 00007f34e8ea15bb 00007f34e8ea7063 00007f34e8ea757b 00007f34e922a88a
0809 15:46:34.871253 (+209026us) log.cc:842] Serialized 14 byte log entry
0809 15:46:34.871398 (+ 145us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 15:46:34.871437 (+ 39us) raft_consensus.cc:1631] Marking committed up to 3
0809 15:46:34.871470 (+ 33us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:34.871525 (+ 55us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:34.871622 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":20080,"prepare.run_cpu_time_us":31,"prepare.run_wall_time_us":32,"spinlock_wait_cycles":2929920,"thread_start_us":208644,"threads_started":1}
I20260809 15:46:34.870806 25863 raft_consensus.cc:2781] T 05ad7d8fdca64e3da527280ca5be9280 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Leader election won for term 3
I20260809 15:46:34.875852 23277 rpcz_store.cc:276] 0809 15:46:33.840265 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:34.603418 (+763153us) service_pool.cc:227] Handling call
0809 15:46:34.845210 (+241792us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:46:34.774236 25885 consensus_queue.cc:1046] T e71a05ba1d4d42bd95c3f85f9313399e P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.088s
W20260809 15:46:34.918098 23353 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 57531.8us
I20260809 15:46:34.848591 25883 raft_consensus.cc:2726] T 2424fd9bd7f14e508fd106cbfda786a4 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260809 15:46:34.902557 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0721a4403a440f383318a3cea9d1442" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:34.792773 23276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f9d6565aa4c45a3b3f01210d0eb71f7" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:34.905853 23517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "297c82ed906d4b48b184985eed3de601" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:34.906466 23280 raft_consensus.cc:2379] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d in current term 3: Already voted for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in this term.
I20260809 15:46:34.914476 23404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30c1740ef6874df0a332fbd6fd4b03ce" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:34.870842 25875 raft_consensus.cc:2781] T 4564dfc6d4094c1b9240b367ccecf3b1 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Leader election won for term 3
I20260809 15:46:34.923414 23517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ced718c1a5c4da79cb8c1ebd1435b87" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:34.924258 23517 raft_consensus.cc:3037] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:34.892537 26040 raft_consensus.cc:2726] T d2d4f749fcfd4fb3bfb8115af3471598 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260809 15:46:34.891881 25694 raft_consensus.cc:482] T 73f6f44c259547f1b05a3a434616fa54 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:46:34.935760 25694 raft_consensus.cc:3037] T 73f6f44c259547f1b05a3a434616fa54 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:46:34.936179 25943 leader_election.cc:288] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 3 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:34.937049 23517 raft_consensus.cc:2454] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 2.
I20260809 15:46:34.937654 25849 leader_election.cc:288] T 00a1881267a14ba8b78c81040f17a014 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 4 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:34.917961 25852 raft_consensus.cc:2781] T e14d118b2eaa42aa8a7cd4025aba7128 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260809 15:46:34.938444 23517 raft_consensus.cc:1261] T 73f6f44c259547f1b05a3a434616fa54 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260809 15:46:34.938792 23394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3be1e356b254d22b995be32f1c12f6f" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:34.889053 25965 raft_consensus.cc:2726] T 0acc47ce21c3422698d1ef7277c3ed63 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 15:46:34.940079 25919 leader_election.cc:288] T 563daf42d7784db195ea99c3cb21b370 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 3 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:34.923242 23277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16ca412a0dae4feebe85dd01bf1a9173" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:34.923324 23399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae6d2c08a7a94f58ac8dd698e3fd456c" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
W20260809 15:46:34.940019 23282 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55434 (request call id 127) took 1589 ms (client timeout 1577 ms). Trace:
W20260809 15:46:34.941834 23282 rpcz_store.cc:269] 0809 15:46:33.350651 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:33.396960 (+ 46309us) service_pool.cc:227] Handling call
0809 15:46:34.939851 (+1542891us) spinlock_profiling.cc:243] Waited 1.18 s on lock 0x7b54001554d4. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e92242ae 00007f34ee161c27 00007f34e7849c04 00007f34e7849b92 00007f34e7849ac1 00007f34e7849a3c 00007f34e7848cf2 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:46:34.939966 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2713162624}
I20260809 15:46:34.942735 23282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c48a3049897452383257aa757834fb5" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:34.944708 23277 raft_consensus.cc:2396] T 16ca412a0dae4feebe85dd01bf1a9173 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 61e69695258f48baa984c9d395e8ce3f for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260809 15:46:34.943809 23282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a51c0baaa6e432997c9d3552af4f254" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:34.921576 23396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0721a4403a440f383318a3cea9d1442" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:34.925772 25478 leader_election.cc:288] T e71a05ba1d4d42bd95c3f85f9313399e P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 3 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:34.843897 25695 consensus_queue.cc:260] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 67b3aeadfa7a432abe8af4d292e04ab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:34.890139 25477 leader_election.cc:288] T 99bdb93503db4de4a1cfb0161ea0c07c P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 2 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:34.901783 26041 raft_consensus.cc:2726] T df20671a7b874dbb98c0ef28371c1308 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260809 15:46:34.923509 25942 consensus_queue.cc:237] T 0acc47ce21c3422698d1ef7277c3ed63 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } attrs { replace: true } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } }
I20260809 15:46:34.915696 26006 raft_consensus.cc:504] T 2db057fd10e149a9895dff6ef8c9f038 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:34.964308 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "563daf42d7784db195ea99c3cb21b370" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f"
I20260809 15:46:34.758285 23520 rpcz_store.cc:276] 0809 15:46:33.322897 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:33.323012 (+ 115us) service_pool.cc:227] Handling call
0809 15:46:33.763606 (+440594us) spinlock_profiling.cc:243] Waited 8.11 ms on lock 0x7b5800004298. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e613cf11 00007f34e62c0851 00007f34e62c0702 00007f34ef1b0b09 00007f34ee0ec730 00007f34ee1b279b 00007f34ee1b2821 00007f34ee1762b0 00007f34ee1617f9 00007f34ee1612a2 00007f34e7847684
0809 15:46:33.763880 (+ 274us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:34.613184 (+849304us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:34.676297 (+ 63113us) spinlock_profiling.cc:243] Waited 735 ms on lock 0x7b54000f7dd4. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e9220f51 00007f34e921f99e 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772 00007f34e764a94c 00007f34e7649d86
0809 15:46:34.676457 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1708331648}
I20260809 15:46:34.934697 25941 raft_consensus.cc:2726] T 953eea268e17491ca76f3a555860a512 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 15:46:34.940235 23526 raft_consensus.cc:1226] T d1a134ba33ea4076bbaac0ae460c17ba P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Rejecting Update request from peer c0f38c1908414566aa9115bb2cbdd85d for earlier term 1. Current term is 2. Ops: []
I20260809 15:46:34.787029 26096 raft_consensus.cc:2726] T 0c48a3049897452383257aa757834fb5 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 15:46:34.939049 23517 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56766 (request call id 1716) took 1024 ms. Trace:
I20260809 15:46:35.047508 23517 rpcz_store.cc:276] 0809 15:46:33.914858 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:34.937928 (+1023070us) service_pool.cc:227] Handling call
0809 15:46:34.938175 (+ 247us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:34.938762 (+ 587us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:34.939006 (+ 244us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:46:34.917840 25999 raft_consensus.cc:2781] T d2d4f749fcfd4fb3bfb8115af3471598 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Leader election won for term 4
I20260809 15:46:34.918313 23393 raft_consensus.cc:1261] T ba56725e42f74a1baa4c04d9660727f9 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260809 15:46:34.918655 25650 raft_consensus.cc:2781] T 4b30f28311964a0b9149a50495e83d8f P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Leader election won for term 2
I20260809 15:46:34.960660 23404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "763e58d6ba644eea830a6c1664d39c9b" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
W20260809 15:46:34.941641 26092 log.cc:924] Time spent T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 67b3aeadfa7a432abe8af4d292e04ab9: Append to log took a long time: real 0.269s user 0.000s sys 0.001s
I20260809 15:46:34.919495 25900 raft_consensus.cc:2726] T 2db057fd10e149a9895dff6ef8c9f038 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 15:46:34.962635 23394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "297c82ed906d4b48b184985eed3de601" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:34.962855 23399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e71a05ba1d4d42bd95c3f85f9313399e" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:34.962575 23396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ced718c1a5c4da79cb8c1ebd1435b87" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:34.925091 23392 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55320 (request call id 1136) took 1283 ms. Trace:
W20260809 15:46:34.965698 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 316287us
I20260809 15:46:35.084113 23351 leader_election.cc:302] T 4564dfc6d4094c1b9240b367ccecf3b1 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:34.965901 23270 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53562 (request call id 1578) took 1017 ms. Trace:
I20260809 15:46:35.084715 23270 rpcz_store.cc:276] 0809 15:46:33.948429 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:34.471666 (+523237us) service_pool.cc:227] Handling call
0809 15:46:34.965728 (+494062us) spinlock_profiling.cc:243] Waited 371 ms on lock 0x7b54001379d4. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e92242ae 00007f34ee161c27 00007f34e7849c04 00007f34e7849b92 00007f34e7849ac1 00007f34e7849a3c 00007f34e7848cf2 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:46:34.965846 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":852847872}
I20260809 15:46:34.966805 23282 raft_consensus.cc:1226] T f9e7cadcd24d45fcbfa84209225e8f2d P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 LEADER]: Rejecting Update request from peer c0f38c1908414566aa9115bb2cbdd85d for earlier term 2. Current term is 3. Ops: []
I20260809 15:46:34.966894 23273 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53554 (request call id 1190) took 1091 ms. Trace:
I20260809 15:46:35.086118 23273 rpcz_store.cc:276] 0809 15:46:33.875941 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:33.906551 (+ 30610us) service_pool.cc:227] Handling call
0809 15:46:34.966717 (+1060166us) spinlock_profiling.cc:243] Waited 644 ms on lock 0x7b54002f4ed4. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e92242ae 00007f34ee161c27 00007f34e7849c04 00007f34e7849b92 00007f34e7849ac1 00007f34e7849a3c 00007f34e7848cf2 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:46:34.966838 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1480103168}
I20260809 15:46:35.087292 23273 raft_consensus.cc:1226] T d0721a4403a440f383318a3cea9d1442 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Rejecting Update request from peer c0f38c1908414566aa9115bb2cbdd85d for earlier term 3. Current term is 4. Ops: []
I20260809 15:46:35.087805 23273 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53562 (request call id 1561) took 1661 ms. Trace:
I20260809 15:46:35.088124 23273 rpcz_store.cc:276] 0809 15:46:33.425979 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:35.086866 (+1660887us) service_pool.cc:227] Handling call
0809 15:46:35.087130 (+ 264us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:35.087620 (+ 490us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:35.087762 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:46:34.968477 25979 leader_election.cc:288] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 4 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:35.089253 23273 raft_consensus.cc:1226] T 99bdb93503db4de4a1cfb0161ea0c07c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Rejecting Update request from peer c0f38c1908414566aa9115bb2cbdd85d for earlier term 1. Current term is 2. Ops: []
I20260809 15:46:34.945087 25639 raft_consensus.cc:2781] T 294d4bcc1e7a477cb56455581277a60f P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260809 15:46:34.963929 23400 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.769s user 0.011s sys 0.035s
W20260809 15:46:35.092790 23400 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.898s user 0.011s sys 0.035s
I20260809 15:46:34.945233 25852 raft_consensus.cc:482] T e14d118b2eaa42aa8a7cd4025aba7128 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Starting leader election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
I20260809 15:46:35.023105 23526 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56766 (request call id 1717) took 1108 ms. Trace:
I20260809 15:46:34.944725 26082 leader_election.cc:288] T a3622f6e3e3641cea1960e3a31498bb7 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:34.949592 26001 raft_consensus.cc:686] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 61e69695258f48baa984c9d395e8ce3f [term 4 LEADER]: Becoming Leader. State: Replica: 61e69695258f48baa984c9d395e8ce3f, State: Running, Role: LEADER
I20260809 15:46:34.945984 25875 raft_consensus.cc:686] T 4564dfc6d4094c1b9240b367ccecf3b1 P 65c36d164b0540f099d592f38307ee6c [term 3 LEADER]: Becoming Leader. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:46:34.958715 24468 raft_consensus.cc:1226] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 61e69695258f48baa984c9d395e8ce3f [term 4 LEADER]: Rejecting Update request from peer c0f38c1908414566aa9115bb2cbdd85d for earlier term 3. Current term is 4. Ops: [3.5-3.5]
I20260809 15:46:35.048941 23521 raft_consensus.cc:1226] T 953eea268e17491ca76f3a555860a512 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Rejecting Update request from peer 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 2. Current term is 3. Ops: []
I20260809 15:46:35.053170 23402 raft_consensus.cc:1261] T a3622f6e3e3641cea1960e3a31498bb7 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260809 15:46:35.107708 23402 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55320 (request call id 1139) took 1370 ms. Trace:
W20260809 15:46:35.059759 23280 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53562 (request call id 1560) took 1659 ms (client timeout 2158 ms). Trace:
I20260809 15:46:34.928608 25746 consensus_queue.cc:1046] T d3e94b7e50754a81b3d72a07476cb7b9 P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.039s
I20260809 15:46:35.071251 23285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a51c0baaa6e432997c9d3552af4f254" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:35.092643 25639 raft_consensus.cc:482] T 294d4bcc1e7a477cb56455581277a60f P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:46:35.072780 23522 raft_consensus.cc:1226] T 2db057fd10e149a9895dff6ef8c9f038 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Rejecting Update request from peer 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 2. Current term is 3. Ops: []
I20260809 15:46:35.075412 23277 raft_consensus.cc:1261] T e14d118b2eaa42aa8a7cd4025aba7128 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Refusing update from remote peer 61e69695258f48baa984c9d395e8ce3f: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260809 15:46:34.968688 25695 raft_consensus.cc:3037] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:46:34.968335 23395 raft_consensus.cc:1261] T a88421dba80d4d3388aa5eb62e23853b P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260809 15:46:34.973663 24455 raft_consensus.cc:1261] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260809 15:46:35.091223 23351 leader_election.cc:302] T 30c1740ef6874df0a332fbd6fd4b03ce P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:35.050088 24452 raft_consensus.cc:1226] T 3296ef50be254fc29cfc77d22a8f7a23 P 61e69695258f48baa984c9d395e8ce3f [term 2 LEADER]: Rejecting Update request from peer 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 1. Current term is 2. Ops: []
I20260809 15:46:35.093344 23273 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53562 (request call id 1562) took 1661 ms. Trace:
I20260809 15:46:35.129403 23273 rpcz_store.cc:276] 0809 15:46:33.431738 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:35.088879 (+1657141us) service_pool.cc:227] Handling call
0809 15:46:35.089119 (+ 240us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:35.089587 (+ 468us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:35.093292 (+ 3705us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:46:35.103156 25573 leader_election.cc:288] T cef3e7277d7d4760824a3436ed4a847d P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:35.098615 23526 rpcz_store.cc:276] 0809 15:46:33.915134 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:34.939852 (+1024718us) service_pool.cc:227] Handling call
0809 15:46:34.940086 (+ 234us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:35.022880 (+ 82794us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:35.023054 (+ 174us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 15:46:35.100566 23472 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 223687us
I20260809 15:46:35.092176 23270 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d3be1e356b254d22b995be32f1c12f6f"
dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
from {username='slave'} at 127.0.0.1:53562
I20260809 15:46:35.104298 25863 raft_consensus.cc:686] T 05ad7d8fdca64e3da527280ca5be9280 P 65c36d164b0540f099d592f38307ee6c [term 3 LEADER]: Becoming Leader. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:46:35.108037 23402 rpcz_store.cc:276] 0809 15:46:33.737681 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:34.750083 (+1012402us) service_pool.cc:227] Handling call
0809 15:46:35.052656 (+302573us) spinlock_profiling.cc:243] Waited 1.57 ms on lock 0x7b58001d7c98. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e613cf11 00007f34e62c0851 00007f34e62c0702 00007f34ef1b0b09 00007f34ee0ec730 00007f34ee1b279b 00007f34ee1b2821 00007f34ee1762b0 00007f34ee1617f9 00007f34ee1612a2 00007f34e7847684
0809 15:46:35.052850 (+ 194us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:35.107463 (+ 54613us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:35.107651 (+ 188us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3608832}
W20260809 15:46:35.109138 23280 rpcz_store.cc:269] 0809 15:46:33.409251 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:33.695191 (+285940us) service_pool.cc:227] Handling call
0809 15:46:35.059572 (+1364381us) spinlock_profiling.cc:243] Waited 1.13 s on lock 0x7b54003aa7d4. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e92242ae 00007f34ee161c27 00007f34e7849c04 00007f34e7849b92 00007f34e7849ac1 00007f34e7849a3c 00007f34e7848cf2 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:46:35.059700 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2609755392}
I20260809 15:46:35.109274 23404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a823d35f4f348d1b52ba9e6d80b00d2" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
W20260809 15:46:35.105413 23470 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 195536us
I20260809 15:46:35.072101 23283 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53562 (request call id 1552) took 1694 ms. Trace:
I20260809 15:46:35.113255 23399 raft_consensus.cc:1261] T 7660dfa2ef694927b8eda4100a2bcd9b P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Refusing update from remote peer 61e69695258f48baa984c9d395e8ce3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260809 15:46:35.082261 25999 raft_consensus.cc:686] T d2d4f749fcfd4fb3bfb8115af3471598 P 65c36d164b0540f099d592f38307ee6c [term 4 LEADER]: Becoming Leader. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:46:35.116703 23394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e4ffcf83c234ae7910ddef3ec0b4bda" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:35.193069 23470 leader_election.cc:302] T 6ae14a74a26f434db08c652e097385c8 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters: c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:34.950531 26080 consensus_queue.cc:260] T 6ae14a74a26f434db08c652e097385c8 P c0f38c1908414566aa9115bb2cbdd85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:35.116526 23277 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55434 (request call id 126) took 1766 ms. Trace:
I20260809 15:46:35.197139 23277 rpcz_store.cc:276] 0809 15:46:33.350409 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:34.960635 (+1610226us) service_pool.cc:227] Handling call
0809 15:46:35.049353 (+ 88718us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:35.116291 (+ 66938us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:35.116472 (+ 181us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:46:35.093181 25650 raft_consensus.cc:686] T 4b30f28311964a0b9149a50495e83d8f P 65c36d164b0540f099d592f38307ee6c [term 2 LEADER]: Becoming Leader. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:46:35.101274 25852 raft_consensus.cc:3037] T e14d118b2eaa42aa8a7cd4025aba7128 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Advancing to term 5
W20260809 15:46:35.211287 26113 log.cc:924] Time spent T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 61e69695258f48baa984c9d395e8ce3f: Append to log took a long time: real 0.096s user 0.000s sys 0.000s
I20260809 15:46:35.093017 23393 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55320 (request call id 1148) took 1246 ms. Trace:
I20260809 15:46:35.125660 23285 raft_consensus.cc:1226] T 6ae14a74a26f434db08c652e097385c8 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Rejecting Update request from peer c0f38c1908414566aa9115bb2cbdd85d for earlier term 3. Current term is 4. Ops: []
I20260809 15:46:35.126257 23529 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44018 (request call id 124) took 1259 ms. Trace:
I20260809 15:46:34.960865 25953 consensus_queue.cc:260] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P c0f38c1908414566aa9115bb2cbdd85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
W20260809 15:46:35.127133 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 253470us
I20260809 15:46:35.221082 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00a1881267a14ba8b78c81040f17a014" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:35.128854 23351 leader_election.cc:302] T 0a51c0baaa6e432997c9d3552af4f254 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 61e69695258f48baa984c9d395e8ce3f, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:35.094484 23396 raft_consensus.cc:1261] T 0da4d5283c8a46878bd39b85e5c71f66 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Refusing update from remote peer 67b3aeadfa7a432abe8af4d292e04ab9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260809 15:46:35.114027 23286 raft_consensus.cc:2454] T d1a134ba33ea4076bbaac0ae460c17ba P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 2.
I20260809 15:46:35.133777 23270 raft_consensus.cc:482] T d3be1e356b254d22b995be32f1c12f6f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 15:46:35.105489 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "907bacfa79d04a55a68c8dbe60056462" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f"
I20260809 15:46:35.108043 23520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a823d35f4f348d1b52ba9e6d80b00d2" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:35.192057 23283 rpcz_store.cc:276] 0809 15:46:33.377521 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:34.574837 (+1197316us) service_pool.cc:227] Handling call
0809 15:46:35.071928 (+497091us) spinlock_profiling.cc:243] Waited 41.7 ms on lock 0x7b540030e654. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e92242ae 00007f34ee161c27 00007f34e7849c04 00007f34e7849b92 00007f34e7849ac1 00007f34e7849a3c 00007f34e7848cf2 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:46:35.072051 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":95895296}
I20260809 15:46:35.192494 23526 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56766 (request call id 1719) took 1276 ms. Trace:
I20260809 15:46:35.194303 23394 raft_consensus.cc:2379] T 8e4ffcf83c234ae7910ddef3ec0b4bda P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 in current term 4: Already voted for candidate 61e69695258f48baa984c9d395e8ce3f in this term.
I20260809 15:46:35.082409 26117 raft_consensus.cc:2781] T 0acc47ce21c3422698d1ef7277c3ed63 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260809 15:46:35.119788 25639 raft_consensus.cc:3037] T 294d4bcc1e7a477cb56455581277a60f P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Advancing to term 5
I20260809 15:46:35.129528 25772 raft_consensus.cc:2726] T d3e94b7e50754a81b3d72a07476cb7b9 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260809 15:46:35.223006 25793 raft_consensus.cc:2781] T 6ae14a74a26f434db08c652e097385c8 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Leader election won for term 4
I20260809 15:46:35.232510 25953 raft_consensus.cc:3037] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Advancing to term 2
W20260809 15:46:35.235781 26094 log.cc:924] Time spent T d0721a4403a440f383318a3cea9d1442 P 65c36d164b0540f099d592f38307ee6c: Append to log took a long time: real 0.274s user 0.000s sys 0.001s
I20260809 15:46:35.235958 23283 raft_consensus.cc:1226] T d2d4f749fcfd4fb3bfb8115af3471598 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Rejecting Update request from peer c0f38c1908414566aa9115bb2cbdd85d for earlier term 3. Current term is 4. Ops: []
I20260809 15:46:35.236634 23283 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53562 (request call id 1570) took 1506 ms. Trace:
I20260809 15:46:35.236718 23397 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55320 (request call id 1133) took 1765 ms. Trace:
I20260809 15:46:35.224665 23404 raft_consensus.cc:1226] T 28e17dcb5efe45b8b6a8c42da03b4819 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Rejecting Update request from peer 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 2. Current term is 3. Ops: []
I20260809 15:46:35.237239 23397 rpcz_store.cc:276] 0809 15:46:33.471219 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:34.683825 (+1212606us) service_pool.cc:227] Handling call
0809 15:46:34.684099 (+ 274us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 15:46:34.684363 (+ 264us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:46:34.684389 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 15:46:34.684767 (+ 378us) log.cc:842] Serialized 26 byte log entry
0809 15:46:34.949481 (+264714us) raft_consensus.cc:1631] Marking committed up to 1
0809 15:46:34.949517 (+ 36us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:34.949575 (+ 58us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 15:46:35.236473 (+286898us) raft_consensus.cc:1671] finished
0809 15:46:35.236491 (+ 18us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:35.236667 (+ 176us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":176935,"raft.run_cpu_time_us":211,"raft.run_wall_time_us":35767,"thread_start_us":263945,"threads_started":1,"wal-append.queue_time_us":422650}
I20260809 15:46:35.225557 25924 consensus_queue.cc:1035] T 7660dfa2ef694927b8eda4100a2bcd9b P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.016s
W20260809 15:46:35.239941 25929 consensus_meta.cc:287] T 0c29937a80724ddb9d07a27534a146f3 P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent flushing consensus metadata: real 0.658s user 0.008s sys 0.000s
I20260809 15:46:35.240316 25929 raft_consensus.cc:504] T 0c29937a80724ddb9d07a27534a146f3 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:35.212970 23277 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53562 (request call id 1569) took 1777 ms. Trace:
I20260809 15:46:35.135628 25905 consensus_queue.cc:1035] T 73f6f44c259547f1b05a3a434616fa54 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.088s
I20260809 15:46:35.241057 23277 rpcz_store.cc:276] 0809 15:46:33.435474 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:35.197821 (+1762347us) service_pool.cc:227] Handling call
0809 15:46:35.212513 (+ 14692us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:35.212679 (+ 166us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:46:35.212706 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 15:46:35.212740 (+ 34us) raft_consensus.cc:1631] Marking committed up to 3
0809 15:46:35.212767 (+ 27us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:35.212812 (+ 45us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:35.212921 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:46:35.241739 23520 raft_consensus.cc:3037] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:46:35.242374 23277 raft_consensus.cc:1226] T 05ad7d8fdca64e3da527280ca5be9280 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Rejecting Update request from peer c0f38c1908414566aa9115bb2cbdd85d for earlier term 2. Current term is 3. Ops: []
I20260809 15:46:35.242959 23277 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53562 (request call id 1574) took 1509 ms. Trace:
W20260809 15:46:35.225767 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 96991.1us
I20260809 15:46:35.244393 23351 leader_election.cc:302] T ba56725e42f74a1baa4c04d9660727f9 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:35.245591 23351 leader_election.cc:302] T f7715d71dbac447b9351fc5eb36c5e80 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
W20260809 15:46:35.246296 25816 consensus_meta.cc:287] T 07fcabb569414bddbd39df6cedd24e64 P 65c36d164b0540f099d592f38307ee6c: Time spent flushing consensus metadata: real 0.672s user 0.006s sys 0.000s
I20260809 15:46:35.226881 26138 consensus_queue.cc:1035] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.023s
I20260809 15:46:35.248675 26138 consensus_queue.cc:1035] T a3622f6e3e3641cea1960e3a31498bb7 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 15:46:35.249785 25967 raft_consensus.cc:504] T cee02fff2c9c48a19f43eb358a89e7b0 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:35.227077 25885 consensus_queue.cc:1046] T 953eea268e17491ca76f3a555860a512 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.024s
I20260809 15:46:35.251108 23286 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53554 (request call id 1193) took 1302 ms. Trace:
I20260809 15:46:35.227344 26001 consensus_queue.cc:237] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:35.253571 25905 raft_consensus.cc:2726] T ba56725e42f74a1baa4c04d9660727f9 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 15:46:35.254134 25905 raft_consensus.cc:2726] T f7715d71dbac447b9351fc5eb36c5e80 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260809 15:46:35.223351 23270 raft_consensus.cc:3037] T d3be1e356b254d22b995be32f1c12f6f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:35.257085 25694 raft_consensus.cc:504] T 73f6f44c259547f1b05a3a434616fa54 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:35.236908 25999 consensus_queue.cc:237] T d2d4f749fcfd4fb3bfb8115af3471598 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:35.258997 25694 leader_election.cc:288] T 73f6f44c259547f1b05a3a434616fa54 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:35.218083 23529 rpcz_store.cc:276] 0809 15:46:33.866715 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:34.701586 (+834871us) service_pool.cc:227] Handling call
0809 15:46:34.722096 (+ 20510us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:34.722294 (+ 198us) raft_consensus.cc:1520] Early marking committed up to index 6
0809 15:46:35.038986 (+316692us) log.cc:842] Serialized 14 byte log entry
0809 15:46:35.125410 (+ 86424us) log.cc:842] Serialized 14 byte log entry
0809 15:46:35.125761 (+ 351us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 15:46:35.125832 (+ 71us) raft_consensus.cc:1631] Marking committed up to 6
0809 15:46:35.125892 (+ 60us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:35.126033 (+ 141us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:35.126172 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":472508,"raft.queue_time_us":146,"raft.run_cpu_time_us":100,"raft.run_wall_time_us":6131,"spinlock_wait_cycles":136901376,"thread_start_us":371572,"threads_started":2}
I20260809 15:46:35.220521 23285 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53562 (request call id 1563) took 1788 ms. Trace:
W20260809 15:46:35.197741 26093 log.cc:924] Time spent T 472816994d814cb198b69e5248697440 P 61e69695258f48baa984c9d395e8ce3f: Append to log took a long time: real 0.326s user 0.001s sys 0.000s
W20260809 15:46:35.209684 25992 consensus_meta.cc:287] T 0c48a3049897452383257aa757834fb5 P 61e69695258f48baa984c9d395e8ce3f: Time spent flushing consensus metadata: real 0.536s user 0.008s sys 0.000s
I20260809 15:46:35.191104 26101 raft_consensus.cc:3032] T d3e94b7e50754a81b3d72a07476cb7b9 P 61e69695258f48baa984c9d395e8ce3f [term 4 LEADER]: Stepping down as leader of term 4
I20260809 15:46:35.189064 24448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3e94b7e50754a81b3d72a07476cb7b9" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f"
I20260809 15:46:35.211167 24463 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38722 (request call id 352) took 3328 ms. Trace:
W20260809 15:46:35.240707 26005 consensus_meta.cc:287] T 0c29937a80724ddb9d07a27534a146f3 P 65c36d164b0540f099d592f38307ee6c: Time spent flushing consensus metadata: real 0.609s user 0.005s sys 0.002s
I20260809 15:46:35.250777 23519 raft_consensus.cc:1226] T cee02fff2c9c48a19f43eb358a89e7b0 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Rejecting Update request from peer 61e69695258f48baa984c9d395e8ce3f for earlier term 2. Current term is 3. Ops: []
I20260809 15:46:35.228454 26117 raft_consensus.cc:482] T 0acc47ce21c3422698d1ef7277c3ed63 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:46:35.225769 25875 consensus_queue.cc:237] T 4564dfc6d4094c1b9240b367ccecf3b1 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:35.251487 23286 rpcz_store.cc:276] 0809 15:46:33.925746 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:34.444520 (+518774us) service_pool.cc:227] Handling call
0809 15:46:35.227675 (+783155us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:46:35.300536 23529 raft_consensus.cc:1226] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Rejecting Update request from peer c0f38c1908414566aa9115bb2cbdd85d for earlier term 1. Current term is 2. Ops: []
I20260809 15:46:35.216918 23470 leader_election.cc:302] T 213543cdbe814c89a17a963469922c1b P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters: c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:35.301271 23345 leader_election.cc:302] T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c
I20260809 15:46:35.126384 25956 raft_consensus.cc:2726] T 4564dfc6d4094c1b9240b367ccecf3b1 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260809 15:46:35.306936 23529 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56766 (request call id 1720) took 1391 ms. Trace:
I20260809 15:46:35.218354 23399 raft_consensus.cc:1261] T 538a228ddfb54bc6a36528d5ff07597b P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260809 15:46:35.087908 23392 rpcz_store.cc:276] 0809 15:46:33.642170 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:34.719212 (+1077042us) service_pool.cc:227] Handling call
0809 15:46:34.793035 (+ 73823us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:34.850562 (+ 57527us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:46:34.870045 (+ 19483us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 15:46:34.880256 (+ 10211us) raft_consensus.cc:1631] Marking committed up to 2
0809 15:46:34.895067 (+ 14811us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:34.904586 (+ 9519us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:34.921094 (+ 16508us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:46:35.223675 23280 raft_consensus.cc:1226] T d1a134ba33ea4076bbaac0ae460c17ba P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Rejecting Update request from peer c0f38c1908414566aa9115bb2cbdd85d for earlier term 1. Current term is 2. Ops: []
I20260809 15:46:35.265496 23285 rpcz_store.cc:276] 0809 15:46:33.432090 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:35.111382 (+1679292us) service_pool.cc:227] Handling call
0809 15:46:35.125511 (+ 14129us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:35.220297 (+ 94786us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:35.220465 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:46:35.236951 23283 rpcz_store.cc:276] 0809 15:46:33.729994 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:35.227608 (+1497614us) service_pool.cc:227] Handling call
0809 15:46:35.235774 (+ 8166us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:35.236420 (+ 646us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:35.236586 (+ 166us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:46:35.307293 23529 rpcz_store.cc:276] 0809 15:46:33.915915 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:35.273533 (+1357618us) service_pool.cc:227] Handling call
0809 15:46:35.300386 (+ 26853us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:35.306709 (+ 6323us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:35.306875 (+ 166us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:46:35.230849 23274 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53562 (request call id 1579) took 1281 ms. Trace:
I20260809 15:46:35.308563 23274 rpcz_store.cc:276] 0809 15:46:33.948939 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:34.231970 (+283031us) service_pool.cc:227] Handling call
0809 15:46:35.230691 (+998721us) spinlock_profiling.cc:243] Waited 117 ms on lock 0x7b54003aa2d4. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e92242ae 00007f34ee161c27 00007f34e7849c04 00007f34e7849b92 00007f34e7849ac1 00007f34e7849a3c 00007f34e7848cf2 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:46:35.230794 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":270159616}
I20260809 15:46:35.309832 23274 raft_consensus.cc:1226] T cef3e7277d7d4760824a3436ed4a847d P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260809 15:46:35.310359 23274 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53554 (request call id 1183) took 1468 ms. Trace:
I20260809 15:46:35.310624 23274 rpcz_store.cc:276] 0809 15:46:33.841866 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:35.309308 (+1467442us) service_pool.cc:227] Handling call
0809 15:46:35.309629 (+ 321us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 15:46:35.310156 (+ 527us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:35.310312 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:46:35.243229 23277 rpcz_store.cc:276] 0809 15:46:33.733751 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:35.241942 (+1508191us) service_pool.cc:227] Handling call
0809 15:46:35.242231 (+ 289us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:35.242759 (+ 528us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:35.242913 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:46:35.224331 23526 rpcz_store.cc:276] 0809 15:46:33.915664 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:35.189754 (+1274090us) service_pool.cc:227] Handling call
0809 15:46:35.192062 (+ 2308us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:35.192214 (+ 152us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:46:35.192239 (+ 25us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 15:46:35.192274 (+ 35us) raft_consensus.cc:1631] Marking committed up to 1
0809 15:46:35.192299 (+ 25us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:35.192347 (+ 48us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:35.192447 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:46:35.316180 23526 raft_consensus.cc:1261] T 7660dfa2ef694927b8eda4100a2bcd9b P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Refusing update from remote peer 61e69695258f48baa984c9d395e8ce3f: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260809 15:46:35.316675 23526 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44018 (request call id 125) took 1089 ms. Trace:
I20260809 15:46:35.316949 23526 rpcz_store.cc:276] 0809 15:46:34.227709 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:35.315604 (+1087895us) service_pool.cc:227] Handling call
0809 15:46:35.315868 (+ 264us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:35.316478 (+ 610us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:35.316628 (+ 150us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:46:35.230005 25841 raft_consensus.cc:3032] T 4b30f28311964a0b9149a50495e83d8f P c0f38c1908414566aa9115bb2cbdd85d [term 1 LEADER]: Stepping down as leader of term 1
I20260809 15:46:35.269119 23396 raft_consensus.cc:1261] T 3296ef50be254fc29cfc77d22a8f7a23 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Refusing update from remote peer 61e69695258f48baa984c9d395e8ce3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260809 15:46:35.223908 23228 leader_election.cc:302] T 998c6cee72cf40228c6dba064c0db2da P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c
I20260809 15:46:35.261996 26138 consensus_queue.cc:1035] T a88421dba80d4d3388aa5eb62e23853b P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.008s
I20260809 15:46:35.246636 23351 leader_election.cc:302] T df20671a7b874dbb98c0ef28371c1308 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
W20260809 15:46:35.270738 23472 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 43942.4us
W20260809 15:46:35.477243 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 230669us
I20260809 15:46:35.477990 23351 leader_election.cc:302] T c5c2e425a74b4e64ae70b5def750434c P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:35.246672 25816 raft_consensus.cc:504] T 07fcabb569414bddbd39df6cedd24e64 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:46:35.237491 26108 log.cc:924] Time spent T 472816994d814cb198b69e5248697440 P c0f38c1908414566aa9115bb2cbdd85d: Append to log took a long time: real 0.113s user 0.000s sys 0.000s
W20260809 15:46:35.490073 24406 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.20.202.65:35459 timed out after 1.543s (SENT) [suppressed 39 similar messages]
I20260809 15:46:35.251562 25885 raft_consensus.cc:3032] T 953eea268e17491ca76f3a555860a512 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 LEADER]: Stepping down as leader of term 2
I20260809 15:46:35.264200 23270 raft_consensus.cc:504] T d3be1e356b254d22b995be32f1c12f6f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
W20260809 15:46:35.229295 26103 log.cc:924] Time spent T 00000000000000000000000000000000 P a86a2513b8024a869d80303dcbf44163: Append to log took a long time: real 0.100s user 0.000s sys 0.000s
I20260809 15:46:35.274971 23394 raft_consensus.cc:1226] T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Rejecting Update request from peer 61e69695258f48baa984c9d395e8ce3f for earlier term 4. Current term is 5. Ops: [2.5-4.6]
I20260809 15:46:35.277063 24463 rpcz_store.cc:276] 0809 15:46:31.882819 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:33.252479 (+1369660us) service_pool.cc:227] Handling call
0809 15:46:33.821295 (+568816us) spinlock_profiling.cc:243] Waited 2.23 ms on lock 0x7b5800005498. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e613cf11 00007f34e62c0851 00007f34e62c0702 00007f34ef1b0b09 00007f34ee0ec730 00007f34ee1b279b 00007f34ee1b2821 00007f34ee1762b0 00007f34ee1617f9 00007f34ee1612a2 00007f34e7847684
0809 15:46:33.821561 (+ 266us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:34.718387 (+896826us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:35.210866 (+492479us) spinlock_profiling.cc:243] Waited 792 ms on lock 0x7b5400630bd4. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e9220f51 00007f34e921f99e 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772 00007f34e764a94c 00007f34e7649d86
0809 15:46:35.211114 (+ 248us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1825821312}
I20260809 15:46:35.281524 24471 raft_consensus.cc:1226] T e71a05ba1d4d42bd95c3f85f9313399e P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Rejecting Update request from peer 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 2. Current term is 3. Ops: []
I20260809 15:46:35.281984 23276 raft_consensus.cc:1226] T 0c29937a80724ddb9d07a27534a146f3 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Rejecting Update request from peer c0f38c1908414566aa9115bb2cbdd85d for earlier term 3. Current term is 4. Ops: []
I20260809 15:46:35.294972 23402 raft_consensus.cc:1261] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Refusing update from remote peer 61e69695258f48baa984c9d395e8ce3f: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 5. (term mismatch)
I20260809 15:46:35.296268 26117 raft_consensus.cc:3037] T 0acc47ce21c3422698d1ef7277c3ed63 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:46:35.284626 25852 raft_consensus.cc:504] T e14d118b2eaa42aa8a7cd4025aba7128 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:35.260667 26136 raft_consensus.cc:2726] T 0a51c0baaa6e432997c9d3552af4f254 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 15:46:35.224188 25849 consensus_queue.cc:1035] T 998c6cee72cf40228c6dba064c0db2da P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.076s
I20260809 15:46:35.266238 25992 raft_consensus.cc:504] T 0c48a3049897452383257aa757834fb5 P 61e69695258f48baa984c9d395e8ce3f [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
W20260809 15:46:35.301367 23470 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 84540us
I20260809 15:46:35.222575 25959 consensus_queue.cc:1046] T d1a134ba33ea4076bbaac0ae460c17ba P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.072s
I20260809 15:46:35.217676 23393 rpcz_store.cc:276] 0809 15:46:33.837611 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:34.768859 (+931248us) service_pool.cc:227] Handling call
0809 15:46:34.861732 (+ 92873us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:35.074984 (+213252us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:35.084387 (+ 9403us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:46:35.188257 25899 consensus_queue.cc:1046] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.087s
I20260809 15:46:35.467984 23521 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56770 (request call id 1193) took 1743 ms. Trace:
I20260809 15:46:35.573764 23521 rpcz_store.cc:276] 0809 15:46:33.724344 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:34.649675 (+925331us) service_pool.cc:227] Handling call
0809 15:46:34.675458 (+ 25783us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:35.112149 (+436691us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:35.467695 (+355546us) spinlock_profiling.cc:243] Waited 373 ms on lock 0x7b54001df1d4. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e9220f51 00007f34e921f99e 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772 00007f34e764a94c 00007f34e7649d86
0809 15:46:35.467879 (+ 184us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":858545408}
W20260809 15:46:35.479867 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 256022us
I20260809 15:46:35.273175 26101 raft_consensus.cc:729] T d3e94b7e50754a81b3d72a07476cb7b9 P 61e69695258f48baa984c9d395e8ce3f [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 61e69695258f48baa984c9d395e8ce3f, State: Running, Role: LEADER
I20260809 15:46:35.505542 26065 raft_consensus.cc:2726] T df20671a7b874dbb98c0ef28371c1308 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260809 15:46:35.285130 26006 leader_election.cc:288] T 2db057fd10e149a9895dff6ef8c9f038 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:35.231339 26038 consensus_queue.cc:1046] T c5c2e425a74b4e64ae70b5def750434c P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.073s
I20260809 15:46:35.504036 26137 raft_consensus.cc:2781] T 998c6cee72cf40228c6dba064c0db2da P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260809 15:46:35.509537 23529 raft_consensus.cc:1226] T 00a1881267a14ba8b78c81040f17a014 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Rejecting Update request from peer c0f38c1908414566aa9115bb2cbdd85d for earlier term 3. Current term is 4. Ops: []
I20260809 15:46:35.510269 23277 raft_consensus.cc:1226] T ae6d2c08a7a94f58ac8dd698e3fd456c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 3. Current term is 4. Ops: []
I20260809 15:46:35.510648 23270 leader_election.cc:288] T d3be1e356b254d22b995be32f1c12f6f P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 3 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:35.621335 23529 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56766 (request call id 1721) took 1700 ms. Trace:
I20260809 15:46:35.621562 23277 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53554 (request call id 1185) took 1779 ms. Trace:
I20260809 15:46:35.456219 25999 raft_consensus.cc:482] T 3d880de657e544468c82157769622a26 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
I20260809 15:46:35.519965 23402 pending_rounds.cc:76] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P c0f38c1908414566aa9115bb2cbdd85d: Aborting all ops after (but not including) 4
I20260809 15:46:35.621755 23529 rpcz_store.cc:276] 0809 15:46:33.920657 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:35.308263 (+1387606us) service_pool.cc:227] Handling call
0809 15:46:35.509356 (+201093us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:35.621111 (+111755us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:35.621280 (+ 169us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:46:35.622304 23402 pending_rounds.cc:97] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P c0f38c1908414566aa9115bb2cbdd85d: Aborting uncommitted NO_OP operation due to leader change: 3.5
I20260809 15:46:35.622587 23402 raft_consensus.cc:2866] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 15:46:35.538031 23470 leader_election.cc:302] T 99bdb93503db4de4a1cfb0161ea0c07c P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:35.456532 25841 raft_consensus.cc:729] T 4b30f28311964a0b9149a50495e83d8f P c0f38c1908414566aa9115bb2cbdd85d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:46:35.539147 23400 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55320 (request call id 1099) took 3790 ms. Trace:
I20260809 15:46:35.624606 23400 rpcz_store.cc:276] 0809 15:46:31.748749 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:34.184564 (+2435815us) service_pool.cc:227] Handling call
0809 15:46:34.184965 (+ 401us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:34.185139 (+ 174us) raft_consensus.cc:1520] Early marking committed up to index 3
0809 15:46:35.098168 (+913029us) log.cc:842] Serialized 14 byte log entry
0809 15:46:35.538842 (+440674us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 15:46:35.538893 (+ 51us) raft_consensus.cc:1631] Marking committed up to 3
0809 15:46:35.538944 (+ 51us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:35.539003 (+ 59us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:35.539096 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":362456,"prepare.run_cpu_time_us":27,"prepare.run_wall_time_us":27,"raft.queue_time_us":254,"raft.run_cpu_time_us":124,"raft.run_wall_time_us":34666,"thread_start_us":1338841,"threads_started":2}
I20260809 15:46:35.481683 25772 raft_consensus.cc:2781] T 213543cdbe814c89a17a963469922c1b P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Leader election won for term 3
I20260809 15:46:35.231302 25650 consensus_queue.cc:237] T 4b30f28311964a0b9149a50495e83d8f P 65c36d164b0540f099d592f38307ee6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:35.296617 26005 raft_consensus.cc:504] T 0c29937a80724ddb9d07a27534a146f3 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:35.479120 23351 leader_election.cc:302] T 7baa2492df0340c4a5ac241959ce1ac3 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:35.510706 26144 raft_consensus.cc:2726] T c5c2e425a74b4e64ae70b5def750434c P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260809 15:46:35.481377 25478 consensus_queue.cc:1035] T 7660dfa2ef694927b8eda4100a2bcd9b P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.092s
I20260809 15:46:35.285588 25863 consensus_queue.cc:237] T 05ad7d8fdca64e3da527280ca5be9280 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:35.604281 23521 raft_consensus.cc:1226] T c0741ab44d99439abc3aa3bdd27ee160 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Rejecting Update request from peer 61e69695258f48baa984c9d395e8ce3f for earlier term 2. Current term is 3. Ops: []
I20260809 15:46:35.605727 25852 leader_election.cc:288] T e14d118b2eaa42aa8a7cd4025aba7128 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 5 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:35.606673 23522 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56770 (request call id 1189) took 1911 ms. Trace:
I20260809 15:46:35.663141 23522 rpcz_store.cc:276] 0809 15:46:33.695639 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:34.580164 (+884525us) service_pool.cc:227] Handling call
0809 15:46:34.580428 (+ 264us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:35.116028 (+535600us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:35.606440 (+490412us) spinlock_profiling.cc:243] Waited 492 ms on lock 0x7b54000f78d4. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e9220f51 00007f34e921f99e 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772 00007f34e764a94c 00007f34e7649d86
0809 15:46:35.606612 (+ 172us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1131907456}
I20260809 15:46:35.664592 23522 raft_consensus.cc:1261] T 0da4d5283c8a46878bd39b85e5c71f66 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Refusing update from remote peer 67b3aeadfa7a432abe8af4d292e04ab9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260809 15:46:35.507375 25793 raft_consensus.cc:686] T 6ae14a74a26f434db08c652e097385c8 P 65c36d164b0540f099d592f38307ee6c [term 4 LEADER]: Becoming Leader. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:46:35.539597 26142 raft_consensus.cc:2726] T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260809 15:46:35.509007 25885 raft_consensus.cc:729] T 953eea268e17491ca76f3a555860a512 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Running, Role: LEADER
I20260809 15:46:35.621927 23277 rpcz_store.cc:276] 0809 15:46:33.842475 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:35.311614 (+1469139us) service_pool.cc:227] Handling call
0809 15:46:35.510105 (+198491us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:35.621203 (+111098us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:35.621485 (+ 282us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:46:35.488962 26139 raft_consensus.cc:482] T 7660dfa2ef694927b8eda4100a2bcd9b P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:35.628008 23282 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53562 (request call id 1547) took 2251 ms. Trace:
I20260809 15:46:35.630717 23270 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RunLeaderElection from 127.0.0.1:53562 (request call id 1555) took 2239 ms. Trace:
I20260809 15:46:35.580868 25477 consensus_queue.cc:1046] T 0a51c0baaa6e432997c9d3552af4f254 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.077s
I20260809 15:46:35.539857 26148 raft_consensus.cc:2726] T 30c1740ef6874df0a332fbd6fd4b03ce P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260809 15:46:35.574149 23518 raft_consensus.cc:1226] T 07fcabb569414bddbd39df6cedd24e64 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Rejecting Update request from peer 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 3. Current term is 4. Ops: []
I20260809 15:46:35.662078 26053 raft_consensus.cc:2726] T 7baa2492df0340c4a5ac241959ce1ac3 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 15:46:35.719462 26096 raft_consensus.cc:3037] T 99bdb93503db4de4a1cfb0161ea0c07c P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:35.602972 25967 leader_election.cc:288] T cee02fff2c9c48a19f43eb358a89e7b0 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
W20260809 15:46:35.659222 23470 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 121267us
I20260809 15:46:35.746773 23470 leader_election.cc:302] T d1a134ba33ea4076bbaac0ae460c17ba P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters: c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:35.526721 26143 consensus_queue.cc:1035] T 538a228ddfb54bc6a36528d5ff07597b P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.105s
I20260809 15:46:35.620263 24453 raft_consensus.cc:1226] T 0c48a3049897452383257aa757834fb5 P 61e69695258f48baa984c9d395e8ce3f [term 2 FOLLOWER]: Rejecting Update request from peer 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 1. Current term is 2. Ops: []
I20260809 15:46:35.627949 23286 raft_consensus.cc:1261] T a3622f6e3e3641cea1960e3a31498bb7 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260809 15:46:35.577986 25694 consensus_queue.cc:1046] T e71a05ba1d4d42bd95c3f85f9313399e P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.078s
I20260809 15:46:35.619908 24448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3be1e356b254d22b995be32f1c12f6f" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "61e69695258f48baa984c9d395e8ce3f"
I20260809 15:46:35.615959 26137 raft_consensus.cc:482] T 998c6cee72cf40228c6dba064c0db2da P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:46:35.720429 23527 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44018 (request call id 122) took 2077 ms. Trace:
I20260809 15:46:35.628244 26006 raft_consensus.cc:482] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:46:35.721601 23528 raft_consensus.cc:1226] T 0c29937a80724ddb9d07a27534a146f3 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Rejecting Update request from peer c0f38c1908414566aa9115bb2cbdd85d for earlier term 3. Current term is 4. Ops: []
I20260809 15:46:35.723047 23517 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56766 (request call id 1718) took 1807 ms. Trace:
I20260809 15:46:35.506032 26133 consensus_queue.cc:1046] T 6ced718c1a5c4da79cb8c1ebd1435b87 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.150s
I20260809 15:46:35.728318 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e14d118b2eaa42aa8a7cd4025aba7128" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f"
I20260809 15:46:35.728962 23270 rpcz_store.cc:276] 0809 15:46:33.381906 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:35.085510 (+1703604us) service_pool.cc:227] Handling call
0809 15:46:35.621330 (+535820us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:46:35.513316 26001 consensus_queue.cc:1035] T 3296ef50be254fc29cfc77d22a8f7a23 P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.088s
I20260809 15:46:35.728905 23282 rpcz_store.cc:276] 0809 15:46:33.376269 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:34.962050 (+1585781us) service_pool.cc:227] Handling call
0809 15:46:34.962295 (+ 245us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:35.092412 (+130117us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:35.627797 (+535385us) spinlock_profiling.cc:243] Waited 4.36 ms on lock 0x7b5400044554. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e9220f51 00007f34e921f99e 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772 00007f34e764a94c 00007f34e7649d86
0809 15:46:35.627956 (+ 159us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":10021760}
I20260809 15:46:35.732323 25793 consensus_queue.cc:237] T 6ae14a74a26f434db08c652e097385c8 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:35.733544 23529 raft_consensus.cc:1226] T f7715d71dbac447b9351fc5eb36c5e80 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Rejecting Update request from peer 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 2. Current term is 3. Ops: []
I20260809 15:46:35.736665 23277 raft_consensus.cc:1226] T 297c82ed906d4b48b184985eed3de601 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 4. Current term is 5. Ops: []
I20260809 15:46:35.578181 26100 consensus_queue.cc:1046] T d3e94b7e50754a81b3d72a07476cb7b9 P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.077s
I20260809 15:46:35.826514 23270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e14d118b2eaa42aa8a7cd4025aba7128" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:35.826614 23282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07fcabb569414bddbd39df6cedd24e64" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:35.827123 23270 raft_consensus.cc:3037] T e14d118b2eaa42aa8a7cd4025aba7128 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Advancing to term 5
W20260809 15:46:35.738317 23472 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 253718us
I20260809 15:46:35.738410 23521 raft_consensus.cc:1226] T 28e17dcb5efe45b8b6a8c42da03b4819 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Rejecting Update request from peer 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 2. Current term is 3. Ops: []
I20260809 15:46:35.827838 23282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2db057fd10e149a9895dff6ef8c9f038" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:35.828459 23521 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56770 (request call id 1200) took 1102 ms. Trace:
I20260809 15:46:35.828873 23521 rpcz_store.cc:276] 0809 15:46:34.726457 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:35.661727 (+935270us) service_pool.cc:227] Handling call
0809 15:46:35.738255 (+ 76528us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:35.828141 (+ 89886us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:35.828379 (+ 238us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 15:46:35.655063 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 176007us
I20260809 15:46:35.719887 25924 raft_consensus.cc:482] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:46:35.661373 25772 raft_consensus.cc:686] T 213543cdbe814c89a17a963469922c1b P 65c36d164b0540f099d592f38307ee6c [term 3 LEADER]: Becoming Leader. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:46:35.834779 23282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00a1881267a14ba8b78c81040f17a014" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:35.730139 24463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee02fff2c9c48a19f43eb358a89e7b0" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f"
I20260809 15:46:35.757097 23280 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53562 (request call id 1566) took 2323 ms. Trace:
I20260809 15:46:35.656991 25959 consensus_queue.cc:1046] T 00a1881267a14ba8b78c81040f17a014 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.084s
I20260809 15:46:35.644484 25898 consensus_queue.cc:1035] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.090s
W20260809 15:46:35.845973 26160 log.cc:924] Time spent T d2d4f749fcfd4fb3bfb8115af3471598 P 65c36d164b0540f099d592f38307ee6c: Append to log took a long time: real 0.105s user 0.000s sys 0.000s
I20260809 15:46:35.632205 25999 raft_consensus.cc:504] T 3d880de657e544468c82157769622a26 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
W20260809 15:46:35.846535 25639 consensus_meta.cc:287] T 294d4bcc1e7a477cb56455581277a60f P 65c36d164b0540f099d592f38307ee6c: Time spent flushing consensus metadata: real 0.618s user 0.008s sys 0.001s
I20260809 15:46:35.740007 25929 leader_election.cc:288] T 0c29937a80724ddb9d07a27534a146f3 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
W20260809 15:46:35.801615 26159 log.cc:924] Time spent T 8e4ffcf83c234ae7910ddef3ec0b4bda P 61e69695258f48baa984c9d395e8ce3f: Append to log took a long time: real 0.061s user 0.000s sys 0.000s
I20260809 15:46:35.801024 23400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2db057fd10e149a9895dff6ef8c9f038" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:35.801008 26038 raft_consensus.cc:3032] T 0a51c0baaa6e432997c9d3552af4f254 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 LEADER]: Stepping down as leader of term 2
I20260809 15:46:35.762696 25967 consensus_queue.cc:1046] T cef3e7277d7d4760824a3436ed4a847d P 65c36d164b0540f099d592f38307ee6c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.040s
I20260809 15:46:35.755666 25965 raft_consensus.cc:2781] T d1a134ba33ea4076bbaac0ae460c17ba P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Leader election won for term 2
I20260809 15:46:35.817795 23517 rpcz_store.cc:276] 0809 15:46:33.915422 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:35.070997 (+1155575us) service_pool.cc:227] Handling call
0809 15:46:35.071293 (+ 296us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:35.071486 (+ 193us) raft_consensus.cc:1520] Early marking committed up to index 5
0809 15:46:35.265441 (+193955us) log.cc:842] Serialized 14 byte log entry
0809 15:46:35.722759 (+457318us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 15:46:35.722802 (+ 43us) raft_consensus.cc:1631] Marking committed up to 5
0809 15:46:35.722839 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:35.722895 (+ 56us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:35.722996 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3229,"prepare.queue_time_us":157922,"prepare.run_cpu_time_us":31,"prepare.run_wall_time_us":32,"raft.queue_time_us":3565,"raft.run_cpu_time_us":153,"raft.run_wall_time_us":12447,"thread_start_us":596952,"threads_started":2,"wal-append.queue_time_us":492164}
W20260809 15:46:35.821084 24410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 158182us
I20260809 15:46:35.813241 23527 rpcz_store.cc:276] 0809 15:46:33.643500 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:34.437546 (+794046us) service_pool.cc:227] Handling call
0809 15:46:34.477213 (+ 39667us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 15:46:34.601094 (+123881us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:46:34.601121 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 15:46:34.606849 (+ 5728us) log.cc:842] Serialized 26 byte log entry
0809 15:46:34.900853 (+294004us) raft_consensus.cc:1631] Marking committed up to 5
0809 15:46:34.900889 (+ 36us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:34.900953 (+ 64us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 15:46:35.720207 (+819254us) raft_consensus.cc:1671] finished
0809 15:46:35.720228 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:35.720361 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4789,"raft.queue_time_us":20283,"raft.run_cpu_time_us":134,"raft.run_wall_time_us":24729,"thread_start_us":393234,"threads_started":2,"wal-append.queue_time_us":666186}
I20260809 15:46:35.705629 25942 consensus_queue.cc:1046] T 3296ef50be254fc29cfc77d22a8f7a23 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.044s
I20260809 15:46:35.829360 23529 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56770 (request call id 1201) took 1102 ms. Trace:
I20260809 15:46:35.748873 25841 consensus_queue.cc:260] T 4b30f28311964a0b9149a50495e83d8f P c0f38c1908414566aa9115bb2cbdd85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:35.831080 23277 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53554 (request call id 1189) took 1955 ms. Trace:
I20260809 15:46:35.847762 25999 leader_election.cc:288] T 3d880de657e544468c82157769622a26 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
W20260809 15:46:35.833745 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 232587us
I20260809 15:46:35.721989 25980 raft_consensus.cc:482] T 472816994d814cb198b69e5248697440 P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Starting leader election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:46:35.746486 25746 consensus_queue.cc:1046] T c0741ab44d99439abc3aa3bdd27ee160 P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.076s
I20260809 15:46:35.834312 26161 raft_consensus.cc:482] T d0721a4403a440f383318a3cea9d1442 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:46:35.814244 24448 raft_consensus.cc:3037] T d3be1e356b254d22b995be32f1c12f6f P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Advancing to term 3
I20260809 15:46:35.840435 23280 rpcz_store.cc:276] 0809 15:46:33.433189 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:35.194233 (+1761044us) service_pool.cc:227] Handling call
0809 15:46:35.223532 (+ 29299us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:35.307745 (+ 84213us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:35.756915 (+449170us) spinlock_profiling.cc:243] Waited 29 ms on lock 0x7b540008aa50. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e921fa09 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:46:35.757043 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":66758400}
I20260809 15:46:35.839965 23270 raft_consensus.cc:2454] T e14d118b2eaa42aa8a7cd4025aba7128 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 5.
I20260809 15:46:35.881796 23270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c29937a80724ddb9d07a27534a146f3" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:35.823616 26006 raft_consensus.cc:504] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:35.814733 26137 raft_consensus.cc:3037] T 998c6cee72cf40228c6dba064c0db2da P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Advancing to term 6
I20260809 15:46:35.866060 26038 raft_consensus.cc:729] T 0a51c0baaa6e432997c9d3552af4f254 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Running, Role: LEADER
I20260809 15:46:35.825084 23286 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53554 (request call id 1188) took 1972 ms. Trace:
I20260809 15:46:35.821766 25816 leader_election.cc:288] T 07fcabb569414bddbd39df6cedd24e64 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 4 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:35.765048 26158 consensus_queue.cc:1046] T ae6d2c08a7a94f58ac8dd698e3fd456c P 65c36d164b0540f099d592f38307ee6c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.085s
I20260809 15:46:35.864264 23470 leader_election.cc:302] T 00a1881267a14ba8b78c81040f17a014 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d
W20260809 15:46:35.962296 23470 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 98121.9us
I20260809 15:46:35.963281 23470 leader_election.cc:302] T e14d118b2eaa42aa8a7cd4025aba7128 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 61e69695258f48baa984c9d395e8ce3f
I20260809 15:46:35.865551 23399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07fcabb569414bddbd39df6cedd24e64" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:35.849949 25992 leader_election.cc:288] T 0c48a3049897452383257aa757834fb5 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 2 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:35.868494 23282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee02fff2c9c48a19f43eb358a89e7b0" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:35.964206 23282 raft_consensus.cc:3037] T cee02fff2c9c48a19f43eb358a89e7b0 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:35.871405 23277 rpcz_store.cc:276] 0809 15:46:33.875647 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:35.735970 (+1860323us) service_pool.cc:227] Handling call
0809 15:46:35.736495 (+ 525us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:35.830755 (+ 94260us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:35.830990 (+ 235us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:46:35.965310 23277 raft_consensus.cc:1226] T 4b30f28311964a0b9149a50495e83d8f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Rejecting Update request from peer c0f38c1908414566aa9115bb2cbdd85d for earlier term 1. Current term is 2. Ops: []
I20260809 15:46:35.830413 23351 leader_election.cc:302] T 998c6cee72cf40228c6dba064c0db2da P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c
I20260809 15:46:35.870748 23529 rpcz_store.cc:276] 0809 15:46:34.726720 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:35.661743 (+935023us) service_pool.cc:227] Handling call
0809 15:46:35.733366 (+ 71623us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:35.824077 (+ 90711us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:35.829303 (+ 5226us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 15:46:35.819664 25868 consensus_meta.cc:287] T 3924b648461e463b9d7925e5671d68ae P 65c36d164b0540f099d592f38307ee6c: Time spent flushing consensus metadata: real 1.125s user 0.010s sys 0.000s
I20260809 15:46:35.846974 25639 raft_consensus.cc:504] T 294d4bcc1e7a477cb56455581277a60f P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:35.967708 23277 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53562 (request call id 1580) took 1974 ms. Trace:
W20260809 15:46:35.876863 25875 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.577s user 0.021s sys 0.041s
I20260809 15:46:35.886329 23280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d880de657e544468c82157769622a26" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:35.849972 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d880de657e544468c82157769622a26" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:35.819757 25885 consensus_queue.cc:260] T 953eea268e17491ca76f3a555860a512 P 67b3aeadfa7a432abe8af4d292e04ab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:35.987413 23280 raft_consensus.cc:1226] T 73f6f44c259547f1b05a3a434616fa54 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Rejecting Update request from peer c0f38c1908414566aa9115bb2cbdd85d for earlier term 3. Current term is 4. Ops: []
I20260809 15:46:35.987845 23528 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56766 (request call id 1698) took 2492 ms. Trace:
I20260809 15:46:35.987214 23282 raft_consensus.cc:2454] T cee02fff2c9c48a19f43eb358a89e7b0 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 3.
I20260809 15:46:35.990274 23282 raft_consensus.cc:1226] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Rejecting Update request from peer c0f38c1908414566aa9115bb2cbdd85d for earlier term 1. Current term is 2. Ops: []
I20260809 15:46:35.989710 23528 rpcz_store.cc:276] 0809 15:46:33.505097 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:34.080251 (+575154us) service_pool.cc:227] Handling call
0809 15:46:34.224867 (+144616us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:35.821444 (+1596577us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:35.987553 (+166109us) spinlock_profiling.cc:243] Waited 1.5 s on lock 0x7b54003ba1d4. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e9220f51 00007f34e921f99e 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772 00007f34e764a94c 00007f34e7649d86
0809 15:46:35.987791 (+ 238us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3441806208}
I20260809 15:46:35.894090 23527 raft_consensus.cc:1261] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Refusing update from remote peer 61e69695258f48baa984c9d395e8ce3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20260809 15:46:35.847827 25772 consensus_queue.cc:237] T 213543cdbe814c89a17a963469922c1b P 65c36d164b0540f099d592f38307ee6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:35.877889 25478 raft_consensus.cc:482] T 3924b648461e463b9d7925e5671d68ae P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:35.898221 23270 raft_consensus.cc:1261] T ba56725e42f74a1baa4c04d9660727f9 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260809 15:46:36.000258 23270 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53554 (request call id 1192) took 2088 ms. Trace:
I20260809 15:46:36.000597 23270 rpcz_store.cc:276] 0809 15:46:33.912213 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:35.897651 (+1985438us) service_pool.cc:227] Handling call
0809 15:46:35.897926 (+ 275us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:36.000032 (+102106us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:36.000205 (+ 173us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:46:36.001647 23270 raft_consensus.cc:1226] T d3be1e356b254d22b995be32f1c12f6f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Rejecting Update request from peer c0f38c1908414566aa9115bb2cbdd85d for earlier term 1. Current term is 3. Ops: []
I20260809 15:46:36.002178 23270 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53562 (request call id 1583) took 1911 ms. Trace:
I20260809 15:46:36.002480 23270 rpcz_store.cc:276] 0809 15:46:34.090898 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:36.001251 (+1910353us) service_pool.cc:227] Handling call
0809 15:46:36.001514 (+ 263us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:36.001976 (+ 462us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:36.002136 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 15:46:35.898365 26163 log.cc:924] Time spent T 472816994d814cb198b69e5248697440 P 65c36d164b0540f099d592f38307ee6c: Append to log took a long time: real 0.067s user 0.000s sys 0.000s
I20260809 15:46:35.878005 25883 raft_consensus.cc:3032] T e71a05ba1d4d42bd95c3f85f9313399e P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 LEADER]: Stepping down as leader of term 2
I20260809 15:46:35.859253 26138 consensus_queue.cc:1035] T ba56725e42f74a1baa4c04d9660727f9 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.035s
I20260809 15:46:35.988538 25913 raft_consensus.cc:3032] T 05ad7d8fdca64e3da527280ca5be9280 P c0f38c1908414566aa9115bb2cbdd85d [term 2 LEADER]: Stepping down as leader of term 2
I20260809 15:46:36.000277 23280 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53562 (request call id 1581) took 1909 ms. Trace:
I20260809 15:46:36.002360 25639 leader_election.cc:288] T 294d4bcc1e7a477cb56455581277a60f P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 5 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:36.005697 25913 raft_consensus.cc:729] T 05ad7d8fdca64e3da527280ca5be9280 P c0f38c1908414566aa9115bb2cbdd85d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:46:35.839432 25924 raft_consensus.cc:504] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:36.006834 25913 consensus_queue.cc:260] T 05ad7d8fdca64e3da527280ca5be9280 P c0f38c1908414566aa9115bb2cbdd85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:36.007573 24463 raft_consensus.cc:3037] T 0acc47ce21c3422698d1ef7277c3ed63 P 61e69695258f48baa984c9d395e8ce3f [term 2 LEARNER]: Advancing to term 3
I20260809 15:46:35.877976 25980 raft_consensus.cc:3037] T 472816994d814cb198b69e5248697440 P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Advancing to term 4
W20260809 15:46:35.966341 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 136011us
W20260809 15:46:36.010324 26166 log.cc:924] Time spent T 7660dfa2ef694927b8eda4100a2bcd9b P c0f38c1908414566aa9115bb2cbdd85d: Append to log took a long time: real 0.196s user 0.001s sys 0.000s
I20260809 15:46:36.010864 23351 leader_election.cc:302] T 2424fd9bd7f14e508fd106cbfda786a4 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
W20260809 15:46:36.011775 26156 log.cc:924] Time spent T 30c1740ef6874df0a332fbd6fd4b03ce P c0f38c1908414566aa9115bb2cbdd85d: Append to log took a long time: real 0.190s user 0.000s sys 0.000s
I20260809 15:46:36.012012 23351 leader_election.cc:302] T cef3e7277d7d4760824a3436ed4a847d P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:35.965744 25899 raft_consensus.cc:482] T 0da4d5283c8a46878bd39b85e5c71f66 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 15:46:35.986632 25875 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.685s user 0.022s sys 0.041s
I20260809 15:46:36.013127 25885 raft_consensus.cc:3037] T 953eea268e17491ca76f3a555860a512 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:36.008507 25924 leader_election.cc:288] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:36.016012 23270 raft_consensus.cc:1226] T cee02fff2c9c48a19f43eb358a89e7b0 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Rejecting Update request from peer 61e69695258f48baa984c9d395e8ce3f for earlier term 2. Current term is 3. Ops: []
I20260809 15:46:35.888759 25965 raft_consensus.cc:686] T d1a134ba33ea4076bbaac0ae460c17ba P 65c36d164b0540f099d592f38307ee6c [term 2 LEADER]: Becoming Leader. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:46:35.885118 23404 raft_consensus.cc:1261] T 8e4ffcf83c234ae7910ddef3ec0b4bda P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Refusing update from remote peer 61e69695258f48baa984c9d395e8ce3f: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260809 15:46:36.025653 23404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3aa6a9a52804df0ba9b0ca3c82f38c3" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:35.969136 23277 rpcz_store.cc:276] 0809 15:46:33.993555 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:35.964912 (+1971357us) service_pool.cc:227] Handling call
0809 15:46:35.965167 (+ 255us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:35.965642 (+ 475us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:35.967655 (+ 2013us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:46:35.979918 25868 raft_consensus.cc:504] T 3924b648461e463b9d7925e5671d68ae P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:35.981887 23526 raft_consensus.cc:1226] T 294d4bcc1e7a477cb56455581277a60f P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Rejecting Update request from peer 61e69695258f48baa984c9d395e8ce3f for earlier term 4. Current term is 5. Ops: []
I20260809 15:46:36.035326 23274 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53554 (request call id 1184) took 2193 ms. Trace:
I20260809 15:46:35.739874 26139 raft_consensus.cc:504] T 7660dfa2ef694927b8eda4100a2bcd9b P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:35.997830 23282 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53562 (request call id 1582) took 1907 ms. Trace:
W20260809 15:46:35.986948 23520 consensus_meta.cc:287] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 65c36d164b0540f099d592f38307ee6c: Time spent flushing consensus metadata: real 0.745s user 0.008s sys 0.000s
I20260809 15:46:36.049461 23520 raft_consensus.cc:2454] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 4.
I20260809 15:46:36.003545 26053 consensus_queue.cc:1046] T 0c29937a80724ddb9d07a27534a146f3 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 15:46:36.051826 23522 raft_consensus.cc:1226] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Rejecting Update request from peer c0f38c1908414566aa9115bb2cbdd85d for earlier term 3. Current term is 4. Ops: [3.5-3.5]
I20260809 15:46:36.051427 25746 consensus_queue.cc:1035] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260809 15:46:36.008306 25913 raft_consensus.cc:3037] T 05ad7d8fdca64e3da527280ca5be9280 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:36.049140 23282 rpcz_store.cc:276] 0809 15:46:34.090661 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:35.989861 (+1899200us) service_pool.cc:227] Handling call
0809 15:46:35.990136 (+ 275us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:35.997633 (+ 7497us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:35.997785 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:46:36.054812 23282 raft_consensus.cc:1226] T c0741ab44d99439abc3aa3bdd27ee160 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Rejecting Update request from peer 61e69695258f48baa984c9d395e8ce3f for earlier term 2. Current term is 3. Ops: []
I20260809 15:46:36.003803 25883 raft_consensus.cc:729] T e71a05ba1d4d42bd95c3f85f9313399e P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Running, Role: LEADER
I20260809 15:46:36.004535 23280 rpcz_store.cc:276] 0809 15:46:34.090383 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:35.986988 (+1896605us) service_pool.cc:227] Handling call
0809 15:46:35.987244 (+ 256us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:36.000057 (+ 12813us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:36.000225 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:46:36.055334 23282 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55434 (request call id 137) took 1339 ms. Trace:
I20260809 15:46:36.055692 23282 rpcz_store.cc:276] 0809 15:46:34.715968 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:36.054412 (+1338444us) service_pool.cc:227] Handling call
0809 15:46:36.054667 (+ 255us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:36.055124 (+ 457us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:36.055285 (+ 161us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:46:36.056255 23280 raft_consensus.cc:1261] T 294d4bcc1e7a477cb56455581277a60f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Refusing update from remote peer 61e69695258f48baa984c9d395e8ce3f: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260809 15:46:36.012703 25899 raft_consensus.cc:504] T 0da4d5283c8a46878bd39b85e5c71f66 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:35.968518 23529 raft_consensus.cc:1226] T d2d4f749fcfd4fb3bfb8115af3471598 P 65c36d164b0540f099d592f38307ee6c [term 4 LEADER]: Rejecting Update request from peer c0f38c1908414566aa9115bb2cbdd85d for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260809 15:46:36.056819 23280 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55434 (request call id 139) took 1339 ms. Trace:
I20260809 15:46:36.057859 23529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3aa6a9a52804df0ba9b0ca3c82f38c3" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:35.966770 25694 raft_consensus.cc:482] T a88421dba80d4d3388aa5eb62e23853b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:36.059577 25694 raft_consensus.cc:504] T a88421dba80d4d3388aa5eb62e23853b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:36.062198 25694 leader_election.cc:288] T a88421dba80d4d3388aa5eb62e23853b P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:35.967036 25841 raft_consensus.cc:3037] T 4b30f28311964a0b9149a50495e83d8f P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:36.066079 23280 rpcz_store.cc:276] 0809 15:46:34.717608 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:36.055684 (+1338076us) service_pool.cc:227] Handling call
0809 15:46:36.055964 (+ 280us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:36.056597 (+ 633us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:36.056768 (+ 171us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:46:35.889729 26161 raft_consensus.cc:504] T d0721a4403a440f383318a3cea9d1442 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:36.067384 23280 raft_consensus.cc:1261] T 538a228ddfb54bc6a36528d5ff07597b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260809 15:46:36.069442 24410 leader_election.cc:302] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:36.070647 25979 raft_consensus.cc:2726] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 15:46:36.076076 25883 consensus_queue.cc:260] T e71a05ba1d4d42bd95c3f85f9313399e P 67b3aeadfa7a432abe8af4d292e04ab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
W20260809 15:46:36.004738 26168 log.cc:924] Time spent T 5712f6b6fffc4cc5a8ddfbe63e4097df P 61e69695258f48baa984c9d395e8ce3f: Append to log took a long time: real 0.136s user 0.000s sys 0.000s
I20260809 15:46:35.968230 25816 raft_consensus.cc:2726] T 00a1881267a14ba8b78c81040f17a014 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260809 15:46:36.025655 23270 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55434 (request call id 135) took 1310 ms. Trace:
I20260809 15:46:36.035684 23274 rpcz_store.cc:276] 0809 15:46:33.842185 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:35.311242 (+1469057us) service_pool.cc:227] Handling call
0809 15:46:35.320050 (+ 8808us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:35.320221 (+ 171us) raft_consensus.cc:1520] Early marking committed up to index 5
0809 15:46:35.630969 (+310748us) log.cc:842] Serialized 14 byte log entry
0809 15:46:36.034997 (+404028us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 15:46:36.035057 (+ 60us) raft_consensus.cc:1631] Marking committed up to 5
0809 15:46:36.035098 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:36.035156 (+ 58us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:36.035259 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":10612,"prepare.queue_time_us":437212,"prepare.run_cpu_time_us":2971,"prepare.run_wall_time_us":17195,"thread_start_us":509234,"threads_started":2}
W20260809 15:46:36.044477 23470 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 76951.8us
I20260809 15:46:36.082042 23270 rpcz_store.cc:276] 0809 15:46:34.715435 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:36.003103 (+1287668us) service_pool.cc:227] Handling call
0809 15:46:36.015859 (+ 12756us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:36.016889 (+ 1030us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:36.025603 (+ 8714us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:46:36.083066 23470 leader_election.cc:302] T 3d880de657e544468c82157769622a26 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 61e69695258f48baa984c9d395e8ce3f, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:35.985338 25967 raft_consensus.cc:2781] T e14d118b2eaa42aa8a7cd4025aba7128 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Leader election won for term 5
I20260809 15:46:36.046761 23277 raft_consensus.cc:1226] T 563daf42d7784db195ea99c3cb21b370 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Rejecting Update request from peer 61e69695258f48baa984c9d395e8ce3f for earlier term 2. Current term is 3. Ops: []
I20260809 15:46:36.056800 23282 raft_consensus.cc:1226] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Rejecting Update request from peer 61e69695258f48baa984c9d395e8ce3f for earlier term 2. Current term is 3. Ops: []
I20260809 15:46:36.083760 23529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c48a3049897452383257aa757834fb5" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:36.083936 23277 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55434 (request call id 136) took 1368 ms. Trace:
I20260809 15:46:36.084349 23529 raft_consensus.cc:3037] T 0c48a3049897452383257aa757834fb5 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:36.084300 23277 rpcz_store.cc:276] 0809 15:46:34.715733 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:36.046305 (+1330572us) service_pool.cc:227] Handling call
0809 15:46:36.046598 (+ 293us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:36.083720 (+ 37122us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:36.083889 (+ 169us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:46:36.083256 23270 raft_consensus.cc:1261] T a88421dba80d4d3388aa5eb62e23853b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260809 15:46:36.086033 23274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "294d4bcc1e7a477cb56455581277a60f" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:36.086573 23274 raft_consensus.cc:3037] T 294d4bcc1e7a477cb56455581277a60f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Advancing to term 5
I20260809 15:46:36.086920 23282 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55434 (request call id 140) took 1368 ms. Trace:
I20260809 15:46:36.087239 23282 rpcz_store.cc:276] 0809 15:46:34.718956 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:36.056352 (+1337396us) service_pool.cc:227] Handling call
0809 15:46:36.056632 (+ 280us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:36.086720 (+ 30088us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:36.086875 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:46:35.907231 23286 rpcz_store.cc:276] 0809 15:46:33.849338 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:35.490296 (+1640958us) service_pool.cc:227] Handling call
0809 15:46:35.540668 (+ 50372us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:35.763018 (+222350us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:35.821963 (+ 58945us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:46:36.102437 23470 leader_election.cc:302] T cee02fff2c9c48a19f43eb358a89e7b0 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 61e69695258f48baa984c9d395e8ce3f
I20260809 15:46:36.107487 26143 raft_consensus.cc:2726] T 3d880de657e544468c82157769622a26 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 15:46:36.047319 25924 consensus_queue.cc:1035] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.061s
I20260809 15:46:36.110628 23404 raft_consensus.cc:1226] T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Rejecting Update request from peer 61e69695258f48baa984c9d395e8ce3f for earlier term 4. Current term is 5. Ops: [2.5-4.6]
I20260809 15:46:36.046309 25959 raft_consensus.cc:2781] T 998c6cee72cf40228c6dba064c0db2da P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260809 15:46:36.127889 23274 raft_consensus.cc:2454] T 294d4bcc1e7a477cb56455581277a60f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 5.
I20260809 15:46:36.024639 25478 raft_consensus.cc:504] T 3924b648461e463b9d7925e5671d68ae P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:36.121474 25875 consensus_queue.cc:1035] T ba56725e42f74a1baa4c04d9660727f9 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.005s
I20260809 15:46:36.113708 25967 raft_consensus.cc:686] T e14d118b2eaa42aa8a7cd4025aba7128 P 65c36d164b0540f099d592f38307ee6c [term 5 LEADER]: Becoming Leader. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:46:36.158269 25967 consensus_queue.cc:237] T e14d118b2eaa42aa8a7cd4025aba7128 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:36.159056 25478 leader_election.cc:288] T 3924b648461e463b9d7925e5671d68ae P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:36.109066 25965 consensus_queue.cc:237] T d1a134ba33ea4076bbaac0ae460c17ba P 65c36d164b0540f099d592f38307ee6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:36.120079 26158 raft_consensus.cc:2781] T cee02fff2c9c48a19f43eb358a89e7b0 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Leader election won for term 3
I20260809 15:46:36.028636 26038 consensus_queue.cc:260] T 0a51c0baaa6e432997c9d3552af4f254 P 67b3aeadfa7a432abe8af4d292e04ab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:36.146621 24451 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38722 (request call id 365) took 1977 ms. Trace:
I20260809 15:46:36.148133 25899 leader_election.cc:288] T 0da4d5283c8a46878bd39b85e5c71f66 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:36.149140 24452 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38722 (request call id 366) took 1467 ms. Trace:
I20260809 15:46:36.167796 24451 rpcz_store.cc:276] 0809 15:46:34.168877 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:34.201157 (+ 32280us) service_pool.cc:227] Handling call
0809 15:46:36.146374 (+1945217us) spinlock_profiling.cc:243] Waited 912 ms on lock 0x7b5400545454. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e92242ae 00007f34ee161c27 00007f34e7849c04 00007f34e7849b92 00007f34e7849ac1 00007f34e7849a3c 00007f34e7848cf2 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:46:36.146535 (+ 161us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2098123136}
I20260809 15:46:36.168334 24452 rpcz_store.cc:276] 0809 15:46:34.682130 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:34.689266 (+ 7136us) service_pool.cc:227] Handling call
0809 15:46:34.719089 (+ 29823us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:35.134548 (+415459us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:36.148880 (+1014332us) spinlock_profiling.cc:243] Waited 242 ms on lock 0x7b54005a1754. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e9220f51 00007f34e921f99e 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772 00007f34e764a94c 00007f34e7649d86
0809 15:46:36.149079 (+ 199us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":555939072}
I20260809 15:46:36.150314 23285 raft_consensus.cc:1226] T 763e58d6ba644eea830a6c1664d39c9b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 2. Current term is 3. Ops: []
I20260809 15:46:36.169287 25899 raft_consensus.cc:3037] T cef3e7277d7d4760824a3436ed4a847d P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:46:36.169768 23285 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53554 (request call id 1176) took 2329 ms. Trace:
I20260809 15:46:36.170104 23285 rpcz_store.cc:276] 0809 15:46:33.839982 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:35.307876 (+1467894us) service_pool.cc:227] Handling call
0809 15:46:36.149988 (+842112us) spinlock_profiling.cc:243] Waited 7.01 ms on lock 0x7b5800003398. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e613cf11 00007f34e62c0851 00007f34e62c0702 00007f34ef1b0b09 00007f34ee0ec730 00007f34ee1b279b 00007f34ee1b2821 00007f34ee1762b0 00007f34ee1617f9 00007f34ee1612a2 00007f34e7847684
0809 15:46:36.150160 (+ 172us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:36.169536 (+ 19376us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:36.169718 (+ 182us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":16115072}
I20260809 15:46:36.080647 26173 consensus_queue.cc:1046] T 297c82ed906d4b48b184985eed3de601 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.062s
I20260809 15:46:36.124728 26143 consensus_queue.cc:1035] T 538a228ddfb54bc6a36528d5ff07597b P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.031s
I20260809 15:46:36.155965 24455 raft_consensus.cc:1226] T e71a05ba1d4d42bd95c3f85f9313399e P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Rejecting Update request from peer 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 2. Current term is 3. Ops: []
I20260809 15:46:36.089208 23520 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44018 (request call id 128) took 1123 ms. Trace:
I20260809 15:46:36.177747 23520 rpcz_store.cc:276] 0809 15:46:34.965530 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:35.050906 (+ 85376us) service_pool.cc:227] Handling call
0809 15:46:36.083671 (+1032765us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:46:36.181643 23520 raft_consensus.cc:1226] T e14d118b2eaa42aa8a7cd4025aba7128 P 65c36d164b0540f099d592f38307ee6c [term 5 LEADER]: Rejecting Update request from peer 61e69695258f48baa984c9d395e8ce3f for earlier term 4. Current term is 5. Ops: []
I20260809 15:46:36.127713 26118 raft_consensus.cc:3032] T f9e7cadcd24d45fcbfa84209225e8f2d P c0f38c1908414566aa9115bb2cbdd85d [term 2 LEADER]: Stepping down as leader of term 2
I20260809 15:46:36.271654 24475 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38734 (request call id 213) took 1006 ms. Trace:
I20260809 15:46:36.126219 25746 consensus_queue.cc:1046] T d3e94b7e50754a81b3d72a07476cb7b9 P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.079s
I20260809 15:46:36.150699 26144 raft_consensus.cc:2726] T 2424fd9bd7f14e508fd106cbfda786a4 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20260809 15:46:36.205557 26117 consensus_meta.cc:287] T 0acc47ce21c3422698d1ef7277c3ed63 P c0f38c1908414566aa9115bb2cbdd85d: Time spent flushing consensus metadata: real 0.639s user 0.007s sys 0.002s
I20260809 15:46:36.190371 25899 raft_consensus.cc:2726] T cef3e7277d7d4760824a3436ed4a847d P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260809 15:46:36.298728 24406 leader_election.cc:334] T d3be1e356b254d22b995be32f1c12f6f P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459): Timed out: RequestConsensusVote RPC to 127.20.202.65:35459 timed out after 1.577s (SENT)
I20260809 15:46:36.317760 24406 leader_election.cc:302] T d3be1e356b254d22b995be32f1c12f6f P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f; no voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:36.192800 26158 raft_consensus.cc:686] T cee02fff2c9c48a19f43eb358a89e7b0 P 65c36d164b0540f099d592f38307ee6c [term 3 LEADER]: Becoming Leader. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:46:36.277175 24475 rpcz_store.cc:276] 0809 15:46:35.265006 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:35.266066 (+ 1060us) service_pool.cc:227] Handling call
0809 15:46:35.266386 (+ 320us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 15:46:35.266758 (+ 372us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:46:35.266783 (+ 25us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 15:46:35.285181 (+ 18398us) log.cc:842] Serialized 26 byte log entry
0809 15:46:35.619102 (+333921us) raft_consensus.cc:1631] Marking committed up to 5
0809 15:46:35.619140 (+ 38us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:35.619206 (+ 66us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 15:46:36.208724 (+589518us) raft_consensus.cc:1671] finished
0809 15:46:36.208743 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:36.271603 (+ 62860us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":70,"raft.run_cpu_time_us":108,"raft.run_wall_time_us":1349,"thread_start_us":128497,"threads_started":1,"wal-append.queue_time_us":517228}
I20260809 15:46:36.340680 26139 leader_election.cc:288] T 7660dfa2ef694927b8eda4100a2bcd9b P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:36.277859 26161 leader_election.cc:288] T d0721a4403a440f383318a3cea9d1442 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:36.203967 25816 consensus_queue.cc:1035] T a88421dba80d4d3388aa5eb62e23853b P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.127s
W20260809 15:46:36.346374 26180 log.cc:924] Time spent T 4564dfc6d4094c1b9240b367ccecf3b1 P 65c36d164b0540f099d592f38307ee6c: Append to log took a long time: real 0.151s user 0.000s sys 0.001s
I20260809 15:46:36.346908 25931 raft_consensus.cc:3032] T d2d4f749fcfd4fb3bfb8115af3471598 P c0f38c1908414566aa9115bb2cbdd85d [term 3 LEADER]: Stepping down as leader of term 3
I20260809 15:46:36.347443 25931 raft_consensus.cc:729] T d2d4f749fcfd4fb3bfb8115af3471598 P c0f38c1908414566aa9115bb2cbdd85d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:46:36.347826 25868 leader_election.cc:288] T 3924b648461e463b9d7925e5671d68ae P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 5 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:36.349169 25959 raft_consensus.cc:482] T 998c6cee72cf40228c6dba064c0db2da P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Starting leader election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:36.349762 25959 raft_consensus.cc:3037] T 998c6cee72cf40228c6dba064c0db2da P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Advancing to term 6
I20260809 15:46:36.352144 23518 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56770 (request call id 1186) took 3022 ms. Trace:
I20260809 15:46:36.352713 23518 rpcz_store.cc:276] 0809 15:46:33.330058 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:33.917788 (+587730us) service_pool.cc:227] Handling call
0809 15:46:33.918243 (+ 455us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:35.742885 (+1824642us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:36.351786 (+608901us) spinlock_profiling.cc:243] Waited 1.61 s on lock 0x7b5400341054. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e9220f51 00007f34e921f99e 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772 00007f34e764a94c 00007f34e7649d86
0809 15:46:36.352064 (+ 278us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3697991040}
I20260809 15:46:36.355994 25883 raft_consensus.cc:3037] T e71a05ba1d4d42bd95c3f85f9313399e P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:36.358207 25956 raft_consensus.cc:3032] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d [term 3 LEADER]: Stepping down as leader of term 3
I20260809 15:46:36.358585 25956 raft_consensus.cc:729] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:46:36.316903 26138 raft_consensus.cc:482] T 28e17dcb5efe45b8b6a8c42da03b4819 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:36.359469 25956 consensus_queue.cc:260] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:36.360517 26138 raft_consensus.cc:504] T 28e17dcb5efe45b8b6a8c42da03b4819 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:36.361330 25956 raft_consensus.cc:3037] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:46:36.363473 24455 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3aa6a9a52804df0ba9b0ca3c82f38c3" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:36.364401 26133 raft_consensus.cc:482] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:36.317303 26006 leader_election.cc:288] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
W20260809 15:46:34.964735 25501 consensus_queue.cc:1159] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 67b3aeadfa7a432abe8af4d292e04ab9 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "61e69695258f48baa984c9d395e8ce3f" responder_term: 4 status { last_received { term: 2 index: 5 } last_committed_idx: 5 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 3. Current term is 4. Ops: [3.6-3.6]" } } last_received_current_leader { term: 0 index: 0 } }
I20260809 15:46:36.338671 26144 consensus_queue.cc:1046] T d2d4f749fcfd4fb3bfb8115af3471598 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.031s
I20260809 15:46:36.368322 26144 consensus_queue.cc:1225] T d2d4f749fcfd4fb3bfb8115af3471598 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Peer 65c36d164b0540f099d592f38307ee6c log is divergent from this leader: its last log entry 4.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260809 15:46:36.203840 25694 consensus_queue.cc:1046] T 3296ef50be254fc29cfc77d22a8f7a23 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.130s
I20260809 15:46:36.369751 25694 consensus_queue.cc:1225] T 3296ef50be254fc29cfc77d22a8f7a23 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Peer 61e69695258f48baa984c9d395e8ce3f log is divergent from this leader: its last log entry 2.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260809 15:46:36.364414 24455 raft_consensus.cc:2454] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 1.
I20260809 15:46:36.365154 26133 raft_consensus.cc:504] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:36.375660 26133 leader_election.cc:288] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
W20260809 15:46:36.377359 26096 consensus_meta.cc:287] T 99bdb93503db4de4a1cfb0161ea0c07c P 65c36d164b0540f099d592f38307ee6c: Time spent flushing consensus metadata: real 0.629s user 0.008s sys 0.001s
I20260809 15:46:36.377976 26096 raft_consensus.cc:2726] T 99bdb93503db4de4a1cfb0161ea0c07c P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 15:46:36.378666 23528 raft_consensus.cc:1226] T 99bdb93503db4de4a1cfb0161ea0c07c P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Rejecting Update request from peer c0f38c1908414566aa9115bb2cbdd85d for earlier term 1. Current term is 2. Ops: []
I20260809 15:46:36.332306 26117 raft_consensus.cc:504] T 0acc47ce21c3422698d1ef7277c3ed63 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:36.385175 26117 leader_election.cc:288] T 0acc47ce21c3422698d1ef7277c3ed63 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 4 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:36.385048 26001 raft_consensus.cc:3032] T c0741ab44d99439abc3aa3bdd27ee160 P 61e69695258f48baa984c9d395e8ce3f [term 2 LEADER]: Stepping down as leader of term 2
I20260809 15:46:36.386818 26001 raft_consensus.cc:729] T c0741ab44d99439abc3aa3bdd27ee160 P 61e69695258f48baa984c9d395e8ce3f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 61e69695258f48baa984c9d395e8ce3f, State: Running, Role: LEADER
I20260809 15:46:36.387732 26136 raft_consensus.cc:3032] T 6ced718c1a5c4da79cb8c1ebd1435b87 P c0f38c1908414566aa9115bb2cbdd85d [term 1 LEADER]: Stepping down as leader of term 1
I20260809 15:46:36.388113 26136 raft_consensus.cc:729] T 6ced718c1a5c4da79cb8c1ebd1435b87 P c0f38c1908414566aa9115bb2cbdd85d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:46:36.388749 26001 consensus_queue.cc:260] T c0741ab44d99439abc3aa3bdd27ee160 P 61e69695258f48baa984c9d395e8ce3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:36.389236 26136 consensus_queue.cc:260] T 6ced718c1a5c4da79cb8c1ebd1435b87 P c0f38c1908414566aa9115bb2cbdd85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:36.390338 26001 raft_consensus.cc:3037] T c0741ab44d99439abc3aa3bdd27ee160 P 61e69695258f48baa984c9d395e8ce3f [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:36.390656 26136 raft_consensus.cc:3037] T 6ced718c1a5c4da79cb8c1ebd1435b87 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:36.391705 25695 raft_consensus.cc:3032] T c5c2e425a74b4e64ae70b5def750434c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 LEADER]: Stepping down as leader of term 2
I20260809 15:46:36.392140 25695 raft_consensus.cc:729] T c5c2e425a74b4e64ae70b5def750434c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Running, Role: LEADER
I20260809 15:46:36.393654 24451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f9d6565aa4c45a3b3f01210d0eb71f7" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:36.393218 25695 consensus_queue.cc:260] T c5c2e425a74b4e64ae70b5def750434c P 67b3aeadfa7a432abe8af4d292e04ab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:36.394348 24451 raft_consensus.cc:2454] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 61e69695258f48baa984c9d395e8ce3f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 2.
I20260809 15:46:36.276952 26118 raft_consensus.cc:729] T f9e7cadcd24d45fcbfa84209225e8f2d P c0f38c1908414566aa9115bb2cbdd85d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:46:36.405184 23529 raft_consensus.cc:2454] T 0c48a3049897452383257aa757834fb5 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 2.
I20260809 15:46:36.406219 23468 leader_election.cc:302] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c; no voters:
W20260809 15:46:36.408349 26167 log.cc:924] Time spent T 0acc47ce21c3422698d1ef7277c3ed63 P 67b3aeadfa7a432abe8af4d292e04ab9: Append to log took a long time: real 0.572s user 0.000s sys 0.000s
I20260809 15:46:36.411283 26065 raft_consensus.cc:3032] T 00a1881267a14ba8b78c81040f17a014 P c0f38c1908414566aa9115bb2cbdd85d [term 3 LEADER]: Stepping down as leader of term 3
I20260809 15:46:36.411645 26133 raft_consensus.cc:482] T ae6d2c08a7a94f58ac8dd698e3fd456c P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:36.412369 26133 raft_consensus.cc:504] T ae6d2c08a7a94f58ac8dd698e3fd456c P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:36.412907 24468 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38724 (request call id 426) took 1843 ms. Trace:
I20260809 15:46:36.413410 24468 rpcz_store.cc:276] 0809 15:46:34.569446 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:34.577324 (+ 7878us) service_pool.cc:227] Handling call
0809 15:46:34.612988 (+ 35664us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 15:46:35.116152 (+503164us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:36.412490 (+1296338us) spinlock_profiling.cc:243] Waited 299 ms on lock 0x7b54005906d4. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e9220f51 00007f34e921f99e 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772 00007f34e764a94c 00007f34e7649d86
0809 15:46:36.412829 (+ 339us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":688016640}
I20260809 15:46:36.418795 23345 leader_election.cc:302] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:46:36.423298 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3924b648461e463b9d7925e5671d68ae" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f"
I20260809 15:46:36.423702 26148 raft_consensus.cc:2781] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260809 15:46:36.424149 24468 raft_consensus.cc:3037] T 3924b648461e463b9d7925e5671d68ae P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Advancing to term 5
I20260809 15:46:36.424244 26148 raft_consensus.cc:482] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:46:36.424782 26148 raft_consensus.cc:3037] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:36.414970 26133 leader_election.cc:288] T ae6d2c08a7a94f58ac8dd698e3fd456c P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:36.378150 26005 leader_election.cc:288] T 0c29937a80724ddb9d07a27534a146f3 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 4 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:36.403321 25695 raft_consensus.cc:3037] T c5c2e425a74b4e64ae70b5def750434c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:36.431996 23273 raft_consensus.cc:1226] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 LEADER]: Rejecting Update request from peer c0f38c1908414566aa9115bb2cbdd85d for earlier term 1. Current term is 2. Ops: []
I20260809 15:46:36.417881 26065 raft_consensus.cc:729] T 00a1881267a14ba8b78c81040f17a014 P c0f38c1908414566aa9115bb2cbdd85d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:46:36.439409 23273 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53562 (request call id 1564) took 3007 ms. Trace:
I20260809 15:46:36.439759 23273 rpcz_store.cc:276] 0809 15:46:33.432419 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:35.130126 (+1697707us) service_pool.cc:227] Handling call
0809 15:46:35.130372 (+ 246us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:36.431840 (+1301468us) spinlock_profiling.cc:243] Waited 67 ms on lock 0x7b5400210d20. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e91b0730 00007f34e9222300 00007f34e9223371 00007f34e921fedf 00007f34e921f99e 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc
0809 15:46:36.439159 (+ 7319us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:36.439360 (+ 201us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":154091904}
I20260809 15:46:36.426558 25852 raft_consensus.cc:2781] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260809 15:46:36.454855 26065 consensus_queue.cc:260] T 00a1881267a14ba8b78c81040f17a014 P c0f38c1908414566aa9115bb2cbdd85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:36.456707 26065 raft_consensus.cc:3037] T 00a1881267a14ba8b78c81040f17a014 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:46:36.443580 25852 raft_consensus.cc:482] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Starting leader election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
W20260809 15:46:36.471997 26104 log.cc:924] Time spent T f9e7cadcd24d45fcbfa84209225e8f2d P 67b3aeadfa7a432abe8af4d292e04ab9: Append to log took a long time: real 1.266s user 0.015s sys 0.006s
I20260809 15:46:36.471958 25852 raft_consensus.cc:3037] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:36.475180 25942 raft_consensus.cc:482] T 28e17dcb5efe45b8b6a8c42da03b4819 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:36.475924 25942 raft_consensus.cc:504] T 28e17dcb5efe45b8b6a8c42da03b4819 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:36.482288 24463 raft_consensus.cc:1261] T 0acc47ce21c3422698d1ef7277c3ed63 P 61e69695258f48baa984c9d395e8ce3f [term 3 LEARNER]: Refusing update from remote peer 67b3aeadfa7a432abe8af4d292e04ab9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260809 15:46:36.481777 26101 consensus_queue.cc:260] T d3e94b7e50754a81b3d72a07476cb7b9 P 61e69695258f48baa984c9d395e8ce3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:36.492429 26101 raft_consensus.cc:3037] T d3e94b7e50754a81b3d72a07476cb7b9 P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Advancing to term 5
I20260809 15:46:36.483952 23276 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53562 (request call id 1553) took 3106 ms. Trace:
I20260809 15:46:36.491156 25639 raft_consensus.cc:3032] T ae6d2c08a7a94f58ac8dd698e3fd456c P 65c36d164b0540f099d592f38307ee6c [term 3 LEADER]: Stepping down as leader of term 3
I20260809 15:46:36.498364 23280 raft_consensus.cc:1261] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 6. (term mismatch)
I20260809 15:46:36.508134 23280 pending_rounds.cc:76] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 67b3aeadfa7a432abe8af4d292e04ab9: Aborting all ops after (but not including) 5
I20260809 15:46:36.508378 23280 pending_rounds.cc:97] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 67b3aeadfa7a432abe8af4d292e04ab9: Aborting uncommitted NO_OP operation due to leader change: 3.6
I20260809 15:46:36.508596 23280 raft_consensus.cc:2866] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 15:46:36.499667 23522 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56766 (request call id 1726) took 1579 ms. Trace:
I20260809 15:46:36.509900 23522 rpcz_store.cc:276] 0809 15:46:34.920330 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:35.665251 (+744921us) service_pool.cc:227] Handling call
0809 15:46:36.051604 (+386353us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 15:46:36.052240 (+ 636us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:36.499491 (+447251us) spinlock_profiling.cc:243] Waited 386 ms on lock 0x7b54001077d0. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e921fa09 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:46:36.499613 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":887806848}
I20260809 15:46:36.500901 23396 raft_consensus.cc:1226] T cef3e7277d7d4760824a3436ed4a847d P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 3. Current term is 4. Ops: []
I20260809 15:46:36.486054 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "294d4bcc1e7a477cb56455581277a60f" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f"
W20260809 15:46:36.425930 25900 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.768s user 0.017s sys 0.022s
I20260809 15:46:36.516934 26177 raft_consensus.cc:3032] T 3296ef50be254fc29cfc77d22a8f7a23 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 LEADER]: Stepping down as leader of term 1
I20260809 15:46:36.517999 25639 raft_consensus.cc:729] T ae6d2c08a7a94f58ac8dd698e3fd456c P 65c36d164b0540f099d592f38307ee6c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:46:36.464083 26118 consensus_queue.cc:260] T f9e7cadcd24d45fcbfa84209225e8f2d P c0f38c1908414566aa9115bb2cbdd85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:36.532503 23519 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44018 (request call id 123) took 2666 ms. Trace:
I20260809 15:46:36.523978 25639 consensus_queue.cc:260] T ae6d2c08a7a94f58ac8dd698e3fd456c P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:36.429733 26138 leader_election.cc:288] T 28e17dcb5efe45b8b6a8c42da03b4819 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:36.524991 23521 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56766 (request call id 1727) took 1430 ms. Trace:
I20260809 15:46:36.564845 23521 rpcz_store.cc:276] 0809 15:46:35.095071 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:35.860791 (+765720us) service_pool.cc:227] Handling call
0809 15:46:35.879682 (+ 18891us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:35.961568 (+ 81886us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:46:35.961598 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 15:46:35.961634 (+ 36us) raft_consensus.cc:1631] Marking committed up to 4
0809 15:46:35.961661 (+ 27us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:36.524783 (+563122us) spinlock_profiling.cc:243] Waited 81.7 ms on lock 0x7b54001fc054. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e9220f51 00007f34e921f99e 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772 00007f34e764a94c 00007f34e7649d86
0809 15:46:36.524827 (+ 44us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:36.524937 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":187928960}
I20260809 15:46:36.518321 23276 rpcz_store.cc:276] 0809 15:46:33.377749 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:35.050769 (+1673020us) service_pool.cc:227] Handling call
0809 15:46:35.111711 (+ 60942us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:35.568955 (+457244us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:36.483628 (+914673us) spinlock_profiling.cc:243] Waited 162 ms on lock 0x7b54001540d4. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e9220f51 00007f34e921f99e 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772 00007f34e764a94c 00007f34e7649d86
0809 15:46:36.483873 (+ 245us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":372945664}
I20260809 15:46:36.537313 23519 rpcz_store.cc:276] 0809 15:46:33.866404 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:34.688470 (+822066us) service_pool.cc:227] Handling call
0809 15:46:34.688703 (+ 233us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:35.285476 (+596773us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:36.532284 (+1246808us) spinlock_profiling.cc:243] Waited 562 ms on lock 0x7b54000f8cd4. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e9220f51 00007f34e921f99e 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772 00007f34e764a94c 00007f34e7649d86
0809 15:46:36.532450 (+ 166us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1292311936}
I20260809 15:46:36.555248 25905 raft_consensus.cc:3032] T d1a134ba33ea4076bbaac0ae460c17ba P c0f38c1908414566aa9115bb2cbdd85d [term 1 LEADER]: Stepping down as leader of term 1
I20260809 15:46:36.526409 26026 consensus_queue.cc:1035] T 0acc47ce21c3422698d1ef7277c3ed63 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.030s
I20260809 15:46:36.520684 26177 raft_consensus.cc:729] T 3296ef50be254fc29cfc77d22a8f7a23 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Running, Role: LEADER
I20260809 15:46:36.525874 26118 raft_consensus.cc:3037] T f9e7cadcd24d45fcbfa84209225e8f2d P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:36.530448 25931 consensus_queue.cc:260] T d2d4f749fcfd4fb3bfb8115af3471598 P c0f38c1908414566aa9115bb2cbdd85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:36.561358 23526 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44018 (request call id 126) took 1983 ms. Trace:
I20260809 15:46:36.601608 23526 rpcz_store.cc:276] 0809 15:46:34.578365 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:35.317607 (+739242us) service_pool.cc:227] Handling call
0809 15:46:35.317858 (+ 251us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:36.047435 (+729577us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:36.561140 (+513705us) spinlock_profiling.cc:243] Waited 664 ms on lock 0x7b5400076cd4. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e9220f51 00007f34e921f99e 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772 00007f34e764a94c 00007f34e7649d86
0809 15:46:36.561303 (+ 163us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1526732032}
I20260809 15:46:36.563588 23396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c29937a80724ddb9d07a27534a146f3" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:36.603468 23396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3924b648461e463b9d7925e5671d68ae" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:36.604022 23396 raft_consensus.cc:3037] T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Advancing to term 5
I20260809 15:46:36.487169 25980 raft_consensus.cc:504] T 472816994d814cb198b69e5248697440 P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:36.559329 26080 raft_consensus.cc:3037] T 6ae14a74a26f434db08c652e097385c8 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:46:36.445907 24448 raft_consensus.cc:2454] T d3be1e356b254d22b995be32f1c12f6f P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 3.
I20260809 15:46:36.595335 23397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3aa6a9a52804df0ba9b0ca3c82f38c3" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:36.595688 23472 leader_election.cc:302] T 2db057fd10e149a9895dff6ef8c9f038 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:36.614478 23472 leader_election.cc:302] T 07fcabb569414bddbd39df6cedd24e64 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:36.533335 25942 leader_election.cc:288] T 28e17dcb5efe45b8b6a8c42da03b4819 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
W20260809 15:46:36.520538 25900 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.865s user 0.018s sys 0.023s
I20260809 15:46:36.363580 25959 raft_consensus.cc:504] T 998c6cee72cf40228c6dba064c0db2da P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:36.616817 26177 consensus_queue.cc:260] T 3296ef50be254fc29cfc77d22a8f7a23 P 67b3aeadfa7a432abe8af4d292e04ab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
W20260809 15:46:36.599316 26137 consensus_meta.cc:287] T 998c6cee72cf40228c6dba064c0db2da P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent flushing consensus metadata: real 0.701s user 0.007s sys 0.004s
I20260809 15:46:36.575186 25905 raft_consensus.cc:729] T d1a134ba33ea4076bbaac0ae460c17ba P c0f38c1908414566aa9115bb2cbdd85d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:46:36.633581 25905 consensus_queue.cc:260] T d1a134ba33ea4076bbaac0ae460c17ba P c0f38c1908414566aa9115bb2cbdd85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:36.578409 25639 raft_consensus.cc:3037] T ae6d2c08a7a94f58ac8dd698e3fd456c P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Advancing to term 4
W20260809 15:46:36.620417 24406 leader_election.cc:334] T 3d880de657e544468c82157769622a26 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459): Timed out: RequestConsensusVote RPC to 127.20.202.65:35459 timed out after 1.524s (SENT)
I20260809 15:46:36.641681 24406 leader_election.cc:302] T 3d880de657e544468c82157769622a26 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:36.642709 25478 raft_consensus.cc:2726] T 3d880de657e544468c82157769622a26 P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 15:46:36.630584 25959 leader_election.cc:288] T 998c6cee72cf40228c6dba064c0db2da P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 6 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:36.631618 26177 raft_consensus.cc:3037] T 3296ef50be254fc29cfc77d22a8f7a23 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:36.520277 26158 consensus_queue.cc:237] T cee02fff2c9c48a19f43eb358a89e7b0 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:36.632187 26137 raft_consensus.cc:504] T 998c6cee72cf40228c6dba064c0db2da P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:36.667376 26137 leader_election.cc:288] T 998c6cee72cf40228c6dba064c0db2da P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 6 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:36.672600 25980 leader_election.cc:288] T 472816994d814cb198b69e5248697440 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 4 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:36.635118 25905 raft_consensus.cc:3037] T d1a134ba33ea4076bbaac0ae460c17ba P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:36.679152 23283 raft_consensus.cc:1226] T 998c6cee72cf40228c6dba064c0db2da P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 5. Current term is 6. Ops: []
I20260809 15:46:36.679752 23394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0721a4403a440f383318a3cea9d1442" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:36.680974 23394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7660dfa2ef694927b8eda4100a2bcd9b" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:36.626387 26161 raft_consensus.cc:2726] T 07fcabb569414bddbd39df6cedd24e64 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260809 15:46:36.685631 25875 consensus_queue.cc:1046] T cef3e7277d7d4760824a3436ed4a847d P 65c36d164b0540f099d592f38307ee6c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 15:46:36.687968 23397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e17dcb5efe45b8b6a8c42da03b4819" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:36.616051 25746 raft_consensus.cc:2726] T d3be1e356b254d22b995be32f1c12f6f P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 15:46:36.622717 25868 raft_consensus.cc:2726] T 2db057fd10e149a9895dff6ef8c9f038 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260809 15:46:36.623301 24452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7660dfa2ef694927b8eda4100a2bcd9b" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:36.628432 24448 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38722 (request call id 375) took 1112 ms. Trace:
I20260809 15:46:36.691987 24471 raft_consensus.cc:1226] T d3be1e356b254d22b995be32f1c12f6f P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Rejecting Update request from peer c0f38c1908414566aa9115bb2cbdd85d for earlier term 1. Current term is 3. Ops: []
I20260809 15:46:36.695055 24453 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38722 (request call id 372) took 1563 ms. Trace:
I20260809 15:46:36.695624 24453 rpcz_store.cc:276] 0809 15:46:35.127973 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:35.130084 (+ 2111us) service_pool.cc:227] Handling call
0809 15:46:35.227388 (+ 97304us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:35.763220 (+535832us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:36.690901 (+927681us) spinlock_profiling.cc:243] Waited 335 ms on lock 0x7b5400521a54. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e9220f51 00007f34e921f99e 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772 00007f34e764a94c 00007f34e7649d86
0809 15:46:36.691175 (+ 274us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":769932160}
I20260809 15:46:36.696277 25852 raft_consensus.cc:504] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:36.688565 23397 raft_consensus.cc:2454] T 28e17dcb5efe45b8b6a8c42da03b4819 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 3.
W20260809 15:46:36.706542 23228 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.747s user 0.023s sys 0.027s
W20260809 15:46:36.706885 23228 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.748s user 0.023s sys 0.027s
W20260809 15:46:36.707201 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 820165us
W20260809 15:46:36.725581 24406 leader_election.cc:334] T 0a51c0baaa6e432997c9d3552af4f254 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459): Timed out: RequestConsensusVote RPC to 127.20.202.65:35459 timed out after 1.610s (SENT)
I20260809 15:46:36.698771 26177 consensus_queue.cc:1046] T 0c48a3049897452383257aa757834fb5 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.007s
I20260809 15:46:36.741487 23396 raft_consensus.cc:2454] T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 5.
W20260809 15:46:36.735723 25885 consensus_meta.cc:287] T 953eea268e17491ca76f3a555860a512 P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent flushing consensus metadata: real 0.722s user 0.006s sys 0.006s
I20260809 15:46:36.754453 23228 leader_election.cc:302] T d3be1e356b254d22b995be32f1c12f6f P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:36.755600 23283 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53554 (request call id 1179) took 2914 ms. Trace:
I20260809 15:46:36.755957 23283 rpcz_store.cc:276] 0809 15:46:33.840812 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:35.308118 (+1467306us) service_pool.cc:227] Handling call
0809 15:46:35.660180 (+352062us) spinlock_profiling.cc:243] Waited 6.54 ms on lock 0x7b5800003398. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e613cf11 00007f34e62c0851 00007f34e62c0702 00007f34ef1b0b09 00007f34ee0ec730 00007f34ee1b279b 00007f34ee1b2821 00007f34ee1762b0 00007f34ee1617f9 00007f34ee1612a2 00007f34e7847684
0809 15:46:35.660370 (+ 190us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:36.679531 (+1019161us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:36.754935 (+ 75404us) spinlock_profiling.cc:243] Waited 1.02 s on lock 0x7b54000415d4. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e9220f51 00007f34e921f99e 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772 00007f34e764a94c 00007f34e7649d86
0809 15:46:36.755524 (+ 589us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2357645056}
W20260809 15:46:36.758937 23399 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.758s user 0.014s sys 0.020s
W20260809 15:46:36.759297 23399 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.758s user 0.014s sys 0.020s
I20260809 15:46:36.701491 25959 consensus_queue.cc:1046] T d3be1e356b254d22b995be32f1c12f6f P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.040s
W20260809 15:46:36.769496 24406 leader_election.cc:334] T 16ca412a0dae4feebe85dd01bf1a9173 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459): Timed out: RequestConsensusVote RPC to 127.20.202.65:35459 timed out after 1.642s (SENT)
I20260809 15:46:36.781962 24406 leader_election.cc:302] T 16ca412a0dae4feebe85dd01bf1a9173 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:36.701804 24448 rpcz_store.cc:276] 0809 15:46:35.513151 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:35.519666 (+ 6515us) service_pool.cc:227] Handling call
0809 15:46:36.625631 (+1105965us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:46:36.664844 26148 raft_consensus.cc:504] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:36.741567 26026 consensus_queue.cc:1046] T f7715d71dbac447b9351fc5eb36c5e80 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.054s
W20260809 15:46:36.819578 26100 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.767s user 0.014s sys 0.025s
W20260809 15:46:36.822527 26100 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.770s user 0.014s sys 0.025s
I20260809 15:46:36.795907 25980 raft_consensus.cc:2726] T 16ca412a0dae4feebe85dd01bf1a9173 P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260809 15:46:36.795603 25868 raft_consensus.cc:482] T c0741ab44d99439abc3aa3bdd27ee160 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 15:46:36.824481 23517 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.837s user 0.009s sys 0.017s
W20260809 15:46:36.832499 23517 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.845s user 0.009s sys 0.017s
W20260809 15:46:36.824790 23400 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.842s user 0.033s sys 0.025s
W20260809 15:46:36.833258 23400 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.850s user 0.033s sys 0.025s
W20260809 15:46:36.817862 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 63492.8us
W20260809 15:46:36.825119 23226 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.866s user 0.008s sys 0.046s
W20260809 15:46:36.839063 23226 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.880s user 0.009s sys 0.047s
I20260809 15:46:36.817788 26177 raft_consensus.cc:3032] T 0c48a3049897452383257aa757834fb5 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 LEADER]: Stepping down as leader of term 1
W20260809 15:46:36.825007 25793 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.821s user 0.010s sys 0.018s
I20260809 15:46:36.819096 25942 raft_consensus.cc:2726] T d3be1e356b254d22b995be32f1c12f6f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260809 15:46:36.838434 25883 raft_consensus.cc:482] T 16ca412a0dae4feebe85dd01bf1a9173 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:36.844591 23228 leader_election.cc:302] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:36.838616 26190 raft_consensus.cc:482] T f9e7cadcd24d45fcbfa84209225e8f2d P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:36.850131 25883 raft_consensus.cc:504] T 16ca412a0dae4feebe85dd01bf1a9173 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:36.840468 25852 leader_election.cc:288] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 2 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
W20260809 15:46:36.842692 25793 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.839s user 0.010s sys 0.019s
I20260809 15:46:36.841349 26177 raft_consensus.cc:729] T 0c48a3049897452383257aa757834fb5 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Running, Role: LEADER
I20260809 15:46:36.861121 23283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3be1e356b254d22b995be32f1c12f6f" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:36.861831 23283 raft_consensus.cc:2354] T d3be1e356b254d22b995be32f1c12f6f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 61e69695258f48baa984c9d395e8ce3f for earlier term 2. Current term is 3.
I20260809 15:46:36.862142 26148 leader_election.cc:288] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:36.883634 23276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d880de657e544468c82157769622a26" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:36.892217 24448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f9d6565aa4c45a3b3f01210d0eb71f7" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f"
I20260809 15:46:36.892916 24448 raft_consensus.cc:3037] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 61e69695258f48baa984c9d395e8ce3f [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:36.893369 26038 raft_consensus.cc:3037] T 0a51c0baaa6e432997c9d3552af4f254 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:36.910053 23276 raft_consensus.cc:1261] T 3296ef50be254fc29cfc77d22a8f7a23 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Refusing update from remote peer 61e69695258f48baa984c9d395e8ce3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260809 15:46:36.911414 23280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a51c0baaa6e432997c9d3552af4f254" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:36.922906 23273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16ca412a0dae4feebe85dd01bf1a9173" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:36.923728 23273 raft_consensus.cc:2396] T 16ca412a0dae4feebe85dd01bf1a9173 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 61e69695258f48baa984c9d395e8ce3f for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260809 15:46:36.924306 23283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e71a05ba1d4d42bd95c3f85f9313399e" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:36.925160 23273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c48a3049897452383257aa757834fb5" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:36.926502 23276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f9d6565aa4c45a3b3f01210d0eb71f7" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:36.926783 23285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c48a3049897452383257aa757834fb5" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:36.840075 25868 raft_consensus.cc:504] T c0741ab44d99439abc3aa3bdd27ee160 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:36.929998 23396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99bdb93503db4de4a1cfb0161ea0c07c" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:36.932706 23396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3622f6e3e3641cea1960e3a31498bb7" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
W20260809 15:46:36.933593 24468 consensus_meta.cc:287] T 3924b648461e463b9d7925e5671d68ae P 61e69695258f48baa984c9d395e8ce3f: Time spent flushing consensus metadata: real 0.509s user 0.006s sys 0.002s
I20260809 15:46:36.933939 24468 raft_consensus.cc:2454] T 3924b648461e463b9d7925e5671d68ae P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 5.
I20260809 15:46:36.935145 23396 raft_consensus.cc:1226] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Rejecting Update request from peer 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 2. Current term is 3. Ops: []
I20260809 15:46:36.937566 23396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "907bacfa79d04a55a68c8dbe60056462" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:36.940181 23396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "563daf42d7784db195ea99c3cb21b370" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:36.945896 23396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cef3e7277d7d4760824a3436ed4a847d" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:36.946700 23396 raft_consensus.cc:2396] T cef3e7277d7d4760824a3436ed4a847d P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260809 15:46:36.947568 26177 consensus_queue.cc:260] T 0c48a3049897452383257aa757834fb5 P 67b3aeadfa7a432abe8af4d292e04ab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:36.948782 26177 raft_consensus.cc:3037] T 0c48a3049897452383257aa757834fb5 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:36.879668 26192 raft_consensus.cc:2726] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260809 15:46:36.951923 23285 raft_consensus.cc:1261] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Refusing update from remote peer 61e69695258f48baa984c9d395e8ce3f: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260809 15:46:36.879195 26190 raft_consensus.cc:504] T f9e7cadcd24d45fcbfa84209225e8f2d P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:36.955068 26190 leader_election.cc:288] T f9e7cadcd24d45fcbfa84209225e8f2d P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:36.956413 26040 raft_consensus.cc:482] T 2424fd9bd7f14e508fd106cbfda786a4 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:36.956895 26040 raft_consensus.cc:504] T 2424fd9bd7f14e508fd106cbfda786a4 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:36.958832 26040 leader_election.cc:288] T 2424fd9bd7f14e508fd106cbfda786a4 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:36.863492 25929 raft_consensus.cc:482] T 213543cdbe814c89a17a963469922c1b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:36.959597 25929 raft_consensus.cc:504] T 213543cdbe814c89a17a963469922c1b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:36.959859 23396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73f6f44c259547f1b05a3a434616fa54" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:36.961479 25929 leader_election.cc:288] T 213543cdbe814c89a17a963469922c1b P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:36.963145 26137 raft_consensus.cc:3032] T f7715d71dbac447b9351fc5eb36c5e80 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 LEADER]: Stepping down as leader of term 2
I20260809 15:46:36.963421 23396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3be1e356b254d22b995be32f1c12f6f" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:36.852465 26026 consensus_queue.cc:1035] T 0da4d5283c8a46878bd39b85e5c71f66 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.076s
I20260809 15:46:36.964224 23396 raft_consensus.cc:3032] T d3be1e356b254d22b995be32f1c12f6f P c0f38c1908414566aa9115bb2cbdd85d [term 1 LEADER]: Stepping down as leader of term 1
I20260809 15:46:36.964717 23396 raft_consensus.cc:729] T d3be1e356b254d22b995be32f1c12f6f P c0f38c1908414566aa9115bb2cbdd85d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:46:36.965566 23396 consensus_queue.cc:260] T d3be1e356b254d22b995be32f1c12f6f P c0f38c1908414566aa9115bb2cbdd85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:36.966747 23396 raft_consensus.cc:3037] T d3be1e356b254d22b995be32f1c12f6f P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Advancing to term 3
I20260809 15:46:36.969494 25868 leader_election.cc:288] T c0741ab44d99439abc3aa3bdd27ee160 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
W20260809 15:46:36.973521 25695 consensus_meta.cc:287] T c5c2e425a74b4e64ae70b5def750434c P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent flushing consensus metadata: real 0.536s user 0.006s sys 0.000s
I20260809 15:46:36.973898 23397 raft_consensus.cc:1226] T 953eea268e17491ca76f3a555860a512 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Rejecting Update request from peer 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 2. Current term is 3. Ops: []
I20260809 15:46:36.974104 25883 leader_election.cc:288] T 16ca412a0dae4feebe85dd01bf1a9173 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:36.964128 25980 raft_consensus.cc:482] T d3e94b7e50754a81b3d72a07476cb7b9 P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:36.906883 26148 raft_consensus.cc:482] T 2db057fd10e149a9895dff6ef8c9f038 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
I20260809 15:46:36.998294 23397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998c6cee72cf40228c6dba064c0db2da" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:36.999125 26148 raft_consensus.cc:504] T 2db057fd10e149a9895dff6ef8c9f038 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:37.000083 23397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e17dcb5efe45b8b6a8c42da03b4819" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:37.000662 23397 raft_consensus.cc:2454] T 28e17dcb5efe45b8b6a8c42da03b4819 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 3.
I20260809 15:46:37.001566 23397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a88421dba80d4d3388aa5eb62e23853b" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:37.001904 26148 leader_election.cc:288] T 2db057fd10e149a9895dff6ef8c9f038 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:36.963554 26137 raft_consensus.cc:729] T f7715d71dbac447b9351fc5eb36c5e80 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Running, Role: LEADER
I20260809 15:46:36.975575 23394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c29937a80724ddb9d07a27534a146f3" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:37.003404 26137 consensus_queue.cc:260] T f7715d71dbac447b9351fc5eb36c5e80 P 67b3aeadfa7a432abe8af4d292e04ab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:37.004562 23394 raft_consensus.cc:1261] T f9e7cadcd24d45fcbfa84209225e8f2d P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Refusing update from remote peer 67b3aeadfa7a432abe8af4d292e04ab9: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260809 15:46:37.009195 23397 raft_consensus.cc:1226] T 0acc47ce21c3422698d1ef7277c3ed63 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Rejecting Update request from peer 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 3. Current term is 4. Ops: []
I20260809 15:46:36.998659 25980 raft_consensus.cc:504] T d3e94b7e50754a81b3d72a07476cb7b9 P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
W20260809 15:46:36.990654 26136 consensus_meta.cc:287] T 6ced718c1a5c4da79cb8c1ebd1435b87 P c0f38c1908414566aa9115bb2cbdd85d: Time spent flushing consensus metadata: real 0.599s user 0.007s sys 0.007s
I20260809 15:46:37.004683 26137 raft_consensus.cc:3037] T f7715d71dbac447b9351fc5eb36c5e80 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Advancing to term 3
W20260809 15:46:37.023456 23527 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.897s user 0.004s sys 0.026s
W20260809 15:46:37.023738 23527 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.898s user 0.005s sys 0.026s
I20260809 15:46:37.026000 23396 raft_consensus.cc:2454] T d3be1e356b254d22b995be32f1c12f6f P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 3.
W20260809 15:46:37.030531 23470 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.879s user 0.007s sys 0.016s
W20260809 15:46:37.032539 23470 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.902s user 0.007s sys 0.016s
W20260809 15:46:37.034510 23470 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 904319us
I20260809 15:46:37.035888 23283 raft_consensus.cc:2454] T e71a05ba1d4d42bd95c3f85f9313399e P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 3.
W20260809 15:46:37.019906 26053 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.945s user 0.025s sys 0.006s
W20260809 15:46:37.018709 25650 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.002s user 0.016s sys 0.015s
I20260809 15:46:37.018548 26185 raft_consensus.cc:482] T 953eea268e17491ca76f3a555860a512 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 15:46:37.041271 26198 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 3 current term: 3 is higher.
W20260809 15:46:37.021240 25979 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.912s user 0.016s sys 0.015s
W20260809 15:46:37.044788 23404 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.889s user 0.006s sys 0.022s
I20260809 15:46:37.030462 25695 raft_consensus.cc:482] T 538a228ddfb54bc6a36528d5ff07597b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:37.045327 23470 leader_election.cc:302] T 294d4bcc1e7a477cb56455581277a60f P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 61e69695258f48baa984c9d395e8ce3f
I20260809 15:46:37.037307 25868 raft_consensus.cc:482] T f7715d71dbac447b9351fc5eb36c5e80 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 15:46:37.054205 23404 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.898s user 0.006s sys 0.022s
I20260809 15:46:37.054466 23228 leader_election.cc:302] T 30c1740ef6874df0a332fbd6fd4b03ce P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
W20260809 15:46:37.047096 23469 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.767s user 0.016s sys 0.018s
W20260809 15:46:37.104802 23351 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.090s user 0.000s sys 0.036s
W20260809 15:46:37.105257 23351 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.090s user 0.000s sys 0.036s
W20260809 15:46:37.106024 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 1.094e+06us
W20260809 15:46:37.062000 25979 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.953s user 0.016s sys 0.018s
I20260809 15:46:37.107347 26136 consensus_queue.cc:1046] T d0721a4403a440f383318a3cea9d1442 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 15:46:37.108361 26148 raft_consensus.cc:3032] T d0721a4403a440f383318a3cea9d1442 P c0f38c1908414566aa9115bb2cbdd85d [term 3 LEADER]: Stepping down as leader of term 3
W20260809 15:46:37.109967 23469 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.832s user 0.016s sys 0.022s
W20260809 15:46:37.047868 26053 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.963s user 0.025s sys 0.006s
W20260809 15:46:37.055199 26184 thread.cc:636] apply [worker] (thread pool) Time spent creating pthread: real 0.933s user 0.000s sys 0.028s
W20260809 15:46:37.105067 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 50672.9us
I20260809 15:46:37.044379 25883 raft_consensus.cc:482] T 05ad7d8fdca64e3da527280ca5be9280 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:37.115902 23351 leader_election.cc:302] T 907bacfa79d04a55a68c8dbe60056462 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 61e69695258f48baa984c9d395e8ce3f, 67b3aeadfa7a432abe8af4d292e04ab9
W20260809 15:46:37.061930 25650 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.046s user 0.017s sys 0.016s
W20260809 15:46:37.116344 23353 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.037s user 0.025s sys 0.027s
I20260809 15:46:37.159665 23522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3622f6e3e3641cea1960e3a31498bb7" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:37.119015 23519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99bdb93503db4de4a1cfb0161ea0c07c" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
W20260809 15:46:37.168605 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 52856.5us
I20260809 15:46:37.169307 23351 leader_election.cc:302] T a3622f6e3e3641cea1960e3a31498bb7 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:37.109361 25793 raft_consensus.cc:2781] T 294d4bcc1e7a477cb56455581277a60f P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Leader election won for term 5
I20260809 15:46:37.181028 23351 leader_election.cc:302] T 28e17dcb5efe45b8b6a8c42da03b4819 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c
W20260809 15:46:37.111804 25849 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.694s user 0.008s sys 0.030s
I20260809 15:46:37.158172 23404 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52856 (request call id 125) took 1040 ms. Trace:
I20260809 15:46:37.158928 23228 leader_election.cc:302] T ae6d2c08a7a94f58ac8dd698e3fd456c P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:37.185446 23522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a88421dba80d4d3388aa5eb62e23853b" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:37.184465 23404 rpcz_store.cc:276] 0809 15:46:36.117821 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:36.119067 (+ 1246us) service_pool.cc:227] Handling call
0809 15:46:36.131738 (+ 12671us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:36.144417 (+ 12679us) raft_consensus.cc:1520] Early marking committed up to index 6
0809 15:46:37.112132 (+967715us) log.cc:842] Serialized 14 byte log entry
0809 15:46:37.112543 (+ 411us) log.cc:842] Serialized 14 byte log entry
0809 15:46:37.157864 (+ 45321us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 15:46:37.157909 (+ 45us) raft_consensus.cc:1631] Marking committed up to 6
0809 15:46:37.157957 (+ 48us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:37.158016 (+ 59us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:37.158119 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":10718,"raft.queue_time_us":206,"raft.run_wall_time_us":46354,"spinlock_wait_cycles":35290880,"thread_start_us":955871,"threads_started":1}
I20260809 15:46:37.053838 25942 raft_consensus.cc:482] T d1a134ba33ea4076bbaac0ae460c17ba P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:37.117985 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "763e58d6ba644eea830a6c1664d39c9b" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
W20260809 15:46:37.168823 23470 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 53631.7us
W20260809 15:46:37.168038 23353 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.089s user 0.025s sys 0.027s
I20260809 15:46:37.185820 25883 raft_consensus.cc:504] T 05ad7d8fdca64e3da527280ca5be9280 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:37.192245 23528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73f6f44c259547f1b05a3a434616fa54" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
W20260809 15:46:37.208034 23353 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 1.12939e+06us
W20260809 15:46:37.170665 25965 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.004s user 0.025s sys 0.028s
I20260809 15:46:37.046443 26190 consensus_queue.cc:1035] T a3622f6e3e3641cea1960e3a31498bb7 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.117s
I20260809 15:46:37.178604 23529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cef3e7277d7d4760824a3436ed4a847d" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:37.231882 23274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e17dcb5efe45b8b6a8c42da03b4819" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
W20260809 15:46:37.183363 25941 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.746s user 0.013s sys 0.031s
W20260809 15:46:37.232969 25941 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.796s user 0.013s sys 0.031s
I20260809 15:46:37.233191 26191 consensus_queue.cc:1046] T d2d4f749fcfd4fb3bfb8115af3471598 P c0f38c1908414566aa9115bb2cbdd85d [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.046s
I20260809 15:46:37.234414 26191 raft_consensus.cc:482] T e71a05ba1d4d42bd95c3f85f9313399e P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
W20260809 15:46:37.178292 24410 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.051s user 0.016s sys 0.026s
I20260809 15:46:37.113436 26148 raft_consensus.cc:729] T d0721a4403a440f383318a3cea9d1442 P c0f38c1908414566aa9115bb2cbdd85d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:46:37.112757 26006 raft_consensus.cc:482] T 0da4d5283c8a46878bd39b85e5c71f66 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 15:46:37.235391 26158 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.588s user 0.029s sys 0.020s
I20260809 15:46:37.234987 26191 raft_consensus.cc:504] T e71a05ba1d4d42bd95c3f85f9313399e P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:37.235751 26006 raft_consensus.cc:504] T 0da4d5283c8a46878bd39b85e5c71f66 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:37.237659 26191 leader_election.cc:288] T e71a05ba1d4d42bd95c3f85f9313399e P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:37.238250 26006 leader_election.cc:288] T 0da4d5283c8a46878bd39b85e5c71f66 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:37.113265 25929 raft_consensus.cc:2726] T 30c1740ef6874df0a332fbd6fd4b03ce P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260809 15:46:37.055181 26026 raft_consensus.cc:482] T 00a1881267a14ba8b78c81040f17a014 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
W20260809 15:46:37.235841 26158 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.589s user 0.029s sys 0.020s
I20260809 15:46:37.239390 26026 raft_consensus.cc:504] T 00a1881267a14ba8b78c81040f17a014 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:37.240602 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e71a05ba1d4d42bd95c3f85f9313399e" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:37.241626 24468 raft_consensus.cc:2379] T e71a05ba1d4d42bd95c3f85f9313399e P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d in current term 3: Already voted for candidate 61e69695258f48baa984c9d395e8ce3f in this term.
I20260809 15:46:37.242019 26026 leader_election.cc:288] T 00a1881267a14ba8b78c81040f17a014 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
W20260809 15:46:37.122072 25863 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.917s user 0.006s sys 0.035s
W20260809 15:46:37.242733 25863 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.051s user 0.006s sys 0.038s
W20260809 15:46:37.157765 26184 thread.cc:602] apply [worker] (thread pool) Time spent starting thread: real 1.036s user 0.000s sys 0.031s
W20260809 15:46:37.244765 24410 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.117s user 0.017s sys 0.026s
I20260809 15:46:37.116829 26136 consensus_queue.cc:1046] T 99bdb93503db4de4a1cfb0161ea0c07c P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.076s
I20260809 15:46:37.191819 23522 raft_consensus.cc:1261] T f9e7cadcd24d45fcbfa84209225e8f2d P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Refusing update from remote peer 67b3aeadfa7a432abe8af4d292e04ab9: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
W20260809 15:46:37.184352 25849 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.767s user 0.009s sys 0.030s
I20260809 15:46:37.218890 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e17dcb5efe45b8b6a8c42da03b4819" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:37.163126 26096 raft_consensus.cc:482] T 472816994d814cb198b69e5248697440 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:46:37.264146 23529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16ca412a0dae4feebe85dd01bf1a9173" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:37.263901 26096 raft_consensus.cc:504] T 472816994d814cb198b69e5248697440 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:37.161274 23518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c29937a80724ddb9d07a27534a146f3" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:37.264614 24448 raft_consensus.cc:2454] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 3.
I20260809 15:46:37.265468 23529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2db057fd10e149a9895dff6ef8c9f038" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:37.266175 23529 raft_consensus.cc:2379] T 2db057fd10e149a9895dff6ef8c9f038 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d in current term 3: Already voted for candidate 65c36d164b0540f099d592f38307ee6c in this term.
I20260809 15:46:37.266502 26096 leader_election.cc:288] T 472816994d814cb198b69e5248697440 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
W20260809 15:46:37.220294 25965 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.054s user 0.025s sys 0.029s
I20260809 15:46:37.193660 25956 raft_consensus.cc:2781] T 28e17dcb5efe45b8b6a8c42da03b4819 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader election won for term 3
I20260809 15:46:37.267726 23518 raft_consensus.cc:2379] T 0c29937a80724ddb9d07a27534a146f3 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 in current term 4: Already voted for candidate 65c36d164b0540f099d592f38307ee6c in this term.
I20260809 15:46:37.173367 23519 raft_consensus.cc:2454] T 99bdb93503db4de4a1cfb0161ea0c07c P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 2.
I20260809 15:46:37.221833 23283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0721a4403a440f383318a3cea9d1442" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:37.261981 23528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998c6cee72cf40228c6dba064c0db2da" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:37.268970 23283 raft_consensus.cc:2379] T d0721a4403a440f383318a3cea9d1442 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c in current term 4: Already voted for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in this term.
I20260809 15:46:37.104486 25695 raft_consensus.cc:504] T 538a228ddfb54bc6a36528d5ff07597b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:37.191459 23228 leader_election.cc:302] T c0741ab44d99439abc3aa3bdd27ee160 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c
I20260809 15:46:37.183193 26118 raft_consensus.cc:3037] T 907bacfa79d04a55a68c8dbe60056462 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:37.270539 23526 raft_consensus.cc:2454] T 28e17dcb5efe45b8b6a8c42da03b4819 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 3.
I20260809 15:46:37.057803 26185 raft_consensus.cc:504] T 953eea268e17491ca76f3a555860a512 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:37.232462 23274 raft_consensus.cc:2454] T 28e17dcb5efe45b8b6a8c42da03b4819 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 3.
W20260809 15:46:37.258121 26080 consensus_meta.cc:287] T 6ae14a74a26f434db08c652e097385c8 P c0f38c1908414566aa9115bb2cbdd85d: Time spent flushing consensus metadata: real 0.634s user 0.008s sys 0.003s
I20260809 15:46:37.261919 25863 raft_consensus.cc:482] T 953eea268e17491ca76f3a555860a512 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:37.262347 23522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998c6cee72cf40228c6dba064c0db2da" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:37.154026 25639 consensus_queue.cc:1035] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.065s
W20260809 15:46:37.245091 24410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 1.1175e+06us
I20260809 15:46:37.285462 23528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0da4d5283c8a46878bd39b85e5c71f66" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:37.288625 23522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3924b648461e463b9d7925e5671d68ae" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:37.289361 23522 raft_consensus.cc:2354] T 3924b648461e463b9d7925e5671d68ae P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 61e69695258f48baa984c9d395e8ce3f for earlier term 4. Current term is 5.
I20260809 15:46:37.290258 23522 raft_consensus.cc:1226] T c0741ab44d99439abc3aa3bdd27ee160 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Rejecting Update request from peer 61e69695258f48baa984c9d395e8ce3f for earlier term 2. Current term is 3. Ops: []
I20260809 15:46:37.269738 23521 raft_consensus.cc:1226] T 0c48a3049897452383257aa757834fb5 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Rejecting Update request from peer 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 1. Current term is 2. Ops: []
I20260809 15:46:37.267001 23529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae6d2c08a7a94f58ac8dd698e3fd456c" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:37.267802 23270 raft_consensus.cc:1226] T cef3e7277d7d4760824a3436ed4a847d P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260809 15:46:37.271169 23519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0acc47ce21c3422698d1ef7277c3ed63" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:37.291463 23519 raft_consensus.cc:3037] T 0acc47ce21c3422698d1ef7277c3ed63 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:46:37.291006 23529 raft_consensus.cc:2454] T ae6d2c08a7a94f58ac8dd698e3fd456c P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 4.
I20260809 15:46:37.292649 23353 leader_election.cc:302] T ae6d2c08a7a94f58ac8dd698e3fd456c P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:46:37.286018 23528 raft_consensus.cc:2454] T 0da4d5283c8a46878bd39b85e5c71f66 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 3.
W20260809 15:46:37.278970 23392 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.707s user 0.011s sys 0.035s
W20260809 15:46:37.272856 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 81479.2us
I20260809 15:46:37.307834 23228 leader_election.cc:302] T 297c82ed906d4b48b184985eed3de601 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
W20260809 15:46:37.223348 26166 log.cc:924] Time spent T 7660dfa2ef694927b8eda4100a2bcd9b P c0f38c1908414566aa9115bb2cbdd85d: Append to log took a long time: real 0.055s user 0.000s sys 0.000s
W20260809 15:46:37.273631 23402 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.571s user 0.012s sys 0.035s
I20260809 15:46:37.118008 25868 raft_consensus.cc:504] T f7715d71dbac447b9351fc5eb36c5e80 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:37.124152 25999 consensus_queue.cc:1046] T 763e58d6ba644eea830a6c1664d39c9b P 65c36d164b0540f099d592f38307ee6c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.092s
I20260809 15:46:37.193152 25793 raft_consensus.cc:686] T 294d4bcc1e7a477cb56455581277a60f P 65c36d164b0540f099d592f38307ee6c [term 5 LEADER]: Becoming Leader. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:46:37.216343 26194 raft_consensus.cc:482] T df20671a7b874dbb98c0ef28371c1308 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 15:46:37.321980 23520 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.717s user 0.018s sys 0.044s
I20260809 15:46:37.216487 25885 raft_consensus.cc:2726] T ae6d2c08a7a94f58ac8dd698e3fd456c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20260809 15:46:37.246544 26161 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.559s user 0.019s sys 0.026s
I20260809 15:46:37.277469 23274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0da4d5283c8a46878bd39b85e5c71f66" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:37.278318 23470 leader_election.cc:302] T 28e17dcb5efe45b8b6a8c42da03b4819 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
W20260809 15:46:37.459793 23470 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 181557us
W20260809 15:46:37.215251 23395 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.007s user 0.019s sys 0.028s
I20260809 15:46:37.182956 25650 consensus_queue.cc:1046] T 998c6cee72cf40228c6dba064c0db2da P 65c36d164b0540f099d592f38307ee6c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.086s
I20260809 15:46:37.277578 23282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0741ab44d99439abc3aa3bdd27ee160" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:37.283089 23526 raft_consensus.cc:1226] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Rejecting Update request from peer c0f38c1908414566aa9115bb2cbdd85d for earlier term 1. Current term is 2. Ops: []
W20260809 15:46:37.282370 24406 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 495440us
I20260809 15:46:37.593787 23527 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44018 (request call id 131) took 1555 ms. Trace:
I20260809 15:46:37.594380 23527 rpcz_store.cc:276] 0809 15:46:36.038006 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:36.038120 (+ 114us) service_pool.cc:227] Handling call
0809 15:46:36.107449 (+ 69329us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 15:46:36.125450 (+ 18001us) raft_consensus.cc:1520] Early marking committed up to index 4
0809 15:46:36.125483 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 15:46:36.125852 (+ 369us) log.cc:842] Serialized 26 byte log entry
0809 15:46:37.024063 (+898211us) raft_consensus.cc:1631] Marking committed up to 4
0809 15:46:37.024102 (+ 39us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:37.164299 (+140197us) spinlock_profiling.cc:243] Waited 15.4 ms on lock 0x7b54000f87d4. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e9220f51 00007f34e921f99e 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772 00007f34e764a94c 00007f34e7649d86
0809 15:46:37.164348 (+ 49us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 15:46:37.592738 (+428390us) raft_consensus.cc:1671] finished
0809 15:46:37.592779 (+ 41us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:37.593553 (+ 774us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2173,"raft.queue_time_us":3652,"raft.run_cpu_time_us":135,"raft.run_wall_time_us":14257,"spinlock_wait_cycles":35510016,"thread_start_us":897943,"threads_started":1,"wal-append.queue_time_us":1188486}
I20260809 15:46:37.584772 23270 raft_consensus.cc:1261] T d2d4f749fcfd4fb3bfb8115af3471598 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20260809 15:46:37.598875 23468 leader_election.cc:302] T 3924b648461e463b9d7925e5671d68ae P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c; no voters:
I20260809 15:46:37.284164 23283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9e7cadcd24d45fcbfa84209225e8f2d" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
W20260809 15:46:37.300372 25772 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.690s user 0.002s sys 0.022s
W20260809 15:46:37.601631 25772 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.996s user 0.003s sys 0.022s
I20260809 15:46:37.271473 23345 leader_election.cc:302] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d; no voters:
W20260809 15:46:37.603792 23345 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.68:34689, user_credentials={real_user=slave}} blocked reactor thread for 332215us
I20260809 15:46:37.586862 23274 raft_consensus.cc:1261] T 6ae14a74a26f434db08c652e097385c8 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20260809 15:46:37.604863 25695 leader_election.cc:288] T 538a228ddfb54bc6a36528d5ff07597b P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:37.605670 25695 raft_consensus.cc:482] T d2d4f749fcfd4fb3bfb8115af3471598 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:37.301981 23353 leader_election.cc:302] T 0da4d5283c8a46878bd39b85e5c71f66 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
W20260809 15:46:37.607372 23353 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 305312us
W20260809 15:46:37.464136 23395 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.259s user 0.020s sys 0.028s
I20260809 15:46:37.432045 23394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472816994d814cb198b69e5248697440" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:37.610553 23395 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55320 (request call id 1156) took 1852 ms. Trace:
I20260809 15:46:37.587543 23286 raft_consensus.cc:1226] T ae6d2c08a7a94f58ac8dd698e3fd456c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 3. Current term is 4. Ops: []
I20260809 15:46:37.377229 23404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3924b648461e463b9d7925e5671d68ae" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:37.612159 23404 raft_consensus.cc:2354] T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 61e69695258f48baa984c9d395e8ce3f for earlier term 4. Current term is 5.
I20260809 15:46:37.312284 23519 raft_consensus.cc:2454] T 0acc47ce21c3422698d1ef7277c3ed63 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 4.
I20260809 15:46:37.431244 23521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472816994d814cb198b69e5248697440" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
W20260809 15:46:37.307343 23392 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.738s user 0.011s sys 0.035s
W20260809 15:46:37.444957 23402 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.743s user 0.037s sys 0.070s
I20260809 15:46:37.308974 23228 leader_election.cc:302] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters: c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:37.305186 24410 leader_election.cc:302] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c; no voters: c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:37.620718 24448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16ca412a0dae4feebe85dd01bf1a9173" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
W20260809 15:46:37.621407 24410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 316185us
W20260809 15:46:37.457180 23520 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.857s user 0.019s sys 0.045s
I20260809 15:46:37.624584 25746 raft_consensus.cc:2781] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Leader election won for term 4
I20260809 15:46:37.297351 25863 raft_consensus.cc:504] T 953eea268e17491ca76f3a555860a512 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:37.229103 25942 raft_consensus.cc:504] T d1a134ba33ea4076bbaac0ae460c17ba P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:37.532037 25965 consensus_queue.cc:1046] T cef3e7277d7d4760824a3436ed4a847d P 65c36d164b0540f099d592f38307ee6c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.014s
I20260809 15:46:37.541819 25980 leader_election.cc:288] T d3e94b7e50754a81b3d72a07476cb7b9 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:37.714457 25883 leader_election.cc:288] T 05ad7d8fdca64e3da527280ca5be9280 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:37.714993 26206 raft_consensus.cc:2781] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader election won for term 3
I20260809 15:46:37.716630 26209 raft_consensus.cc:482] T f7715d71dbac447b9351fc5eb36c5e80 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:37.717422 26209 raft_consensus.cc:504] T f7715d71dbac447b9351fc5eb36c5e80 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:37.279395 26026 raft_consensus.cc:2726] T c0741ab44d99439abc3aa3bdd27ee160 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20260809 15:46:37.454789 26038 consensus_meta.cc:287] T 0a51c0baaa6e432997c9d3552af4f254 P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent flushing consensus metadata: real 0.561s user 0.009s sys 0.006s
I20260809 15:46:37.523361 26053 raft_consensus.cc:2781] T ae6d2c08a7a94f58ac8dd698e3fd456c P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260809 15:46:37.553066 26118 raft_consensus.cc:2726] T 907bacfa79d04a55a68c8dbe60056462 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260809 15:46:37.465663 26161 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.778s user 0.019s sys 0.026s
I20260809 15:46:37.735642 23280 raft_consensus.cc:2454] T 0a51c0baaa6e432997c9d3552af4f254 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 3.
I20260809 15:46:37.735798 26053 raft_consensus.cc:482] T ae6d2c08a7a94f58ac8dd698e3fd456c P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Starting leader election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:37.736375 26053 raft_consensus.cc:3037] T ae6d2c08a7a94f58ac8dd698e3fd456c P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Advancing to term 4
W20260809 15:46:37.278903 26173 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.962s user 0.018s sys 0.023s
W20260809 15:46:37.737133 26173 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.527s user 0.019s sys 0.023s
W20260809 15:46:37.533480 26177 consensus_meta.cc:287] T 0c48a3049897452383257aa757834fb5 P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent flushing consensus metadata: real 0.584s user 0.014s sys 0.000s
I20260809 15:46:37.364199 26191 consensus_queue.cc:1046] T 99bdb93503db4de4a1cfb0161ea0c07c P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.126s
W20260809 15:46:37.737869 25941 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.501s user 0.006s sys 0.007s
I20260809 15:46:37.462141 26194 raft_consensus.cc:504] T df20671a7b874dbb98c0ef28371c1308 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:37.741562 26257 raft_consensus.cc:2781] T 0da4d5283c8a46878bd39b85e5c71f66 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260809 15:46:37.523427 26196 log.cc:924] Time spent T 5ad8a3b25d7b49e5a2142d5709edd0c7 P c0f38c1908414566aa9115bb2cbdd85d: Append to log took a long time: real 0.466s user 0.001s sys 0.001s
W20260809 15:46:37.742468 26137 consensus_meta.cc:287] T f7715d71dbac447b9351fc5eb36c5e80 P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent flushing consensus metadata: real 0.723s user 0.008s sys 0.002s
I20260809 15:46:37.526170 26198 consensus_queue.cc:1046] T 73f6f44c259547f1b05a3a434616fa54 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.014s
I20260809 15:46:37.743096 26118 raft_consensus.cc:482] T 4bcac4b24e7a4448abdc934bba5a624e P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
I20260809 15:46:37.743144 23397 raft_consensus.cc:1226] T 907bacfa79d04a55a68c8dbe60056462 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Rejecting Update request from peer 61e69695258f48baa984c9d395e8ce3f for earlier term 2. Current term is 3. Ops: []
I20260809 15:46:37.744210 26198 raft_consensus.cc:482] T 7660dfa2ef694927b8eda4100a2bcd9b P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
W20260809 15:46:37.527410 26199 log.cc:924] Time spent T 73f6f44c259547f1b05a3a434616fa54 P 65c36d164b0540f099d592f38307ee6c: Append to log took a long time: real 0.281s user 0.001s sys 0.000s
I20260809 15:46:37.744112 25650 raft_consensus.cc:482] T 2424fd9bd7f14e508fd106cbfda786a4 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:37.746235 23273 raft_consensus.cc:2454] T 0c48a3049897452383257aa757834fb5 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 2.
W20260809 15:46:37.526449 26204 log.cc:924] Time spent T 6ae14a74a26f434db08c652e097385c8 P 65c36d164b0540f099d592f38307ee6c: Append to log took a long time: real 0.224s user 0.000s sys 0.000s
I20260809 15:46:37.735556 23402 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55320 (request call id 1165) took 1170 ms. Trace:
I20260809 15:46:37.505205 26231 raft_consensus.cc:2781] T 28e17dcb5efe45b8b6a8c42da03b4819 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260809 15:46:37.639909 25956 raft_consensus.cc:686] T 28e17dcb5efe45b8b6a8c42da03b4819 P c0f38c1908414566aa9115bb2cbdd85d [term 3 LEADER]: Becoming Leader. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:46:37.748339 25899 raft_consensus.cc:3032] T 0c29937a80724ddb9d07a27534a146f3 P c0f38c1908414566aa9115bb2cbdd85d [term 3 LEADER]: Stepping down as leader of term 3
W20260809 15:46:37.733166 23468 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.68:34689, user_credentials={real_user=slave}} blocked reactor thread for 134690us
I20260809 15:46:37.748840 25899 raft_consensus.cc:729] T 0c29937a80724ddb9d07a27534a146f3 P c0f38c1908414566aa9115bb2cbdd85d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:46:37.749993 25956 consensus_queue.cc:237] T 28e17dcb5efe45b8b6a8c42da03b4819 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:46:37.640138 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 207948us
I20260809 15:46:37.751818 25942 leader_election.cc:288] T d1a134ba33ea4076bbaac0ae460c17ba P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:37.623159 24410 leader_election.cc:302] T 0c48a3049897452383257aa757834fb5 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c; no voters:
I20260809 15:46:37.606415 25695 raft_consensus.cc:504] T d2d4f749fcfd4fb3bfb8115af3471598 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:37.690346 25959 raft_consensus.cc:2726] T a3622f6e3e3641cea1960e3a31498bb7 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260809 15:46:37.691110 25929 raft_consensus.cc:482] T 4564dfc6d4094c1b9240b367ccecf3b1 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:37.691824 25793 consensus_queue.cc:237] T 294d4bcc1e7a477cb56455581277a60f P 65c36d164b0540f099d592f38307ee6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:37.695894 25885 raft_consensus.cc:2781] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 15:46:37.610983 23395 rpcz_store.cc:276] 0809 15:46:35.758124 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:35.760555 (+ 2431us) service_pool.cc:227] Handling call
0809 15:46:35.861666 (+101111us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 15:46:35.905073 (+ 43407us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:46:35.916329 (+ 11256us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 15:46:36.121239 (+204910us) log.cc:842] Serialized 26 byte log entry
0809 15:46:37.609767 (+1488528us) raft_consensus.cc:1631] Marking committed up to 1
0809 15:46:37.609878 (+ 111us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:37.610006 (+ 128us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 15:46:37.610074 (+ 68us) raft_consensus.cc:1671] finished
0809 15:46:37.610096 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:37.610290 (+ 194us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6161,"raft.queue_time_us":6762,"raft.run_cpu_time_us":123,"raft.run_wall_time_us":13224,"thread_start_us":1416678,"threads_started":1,"wal-append.queue_time_us":1400000}
I20260809 15:46:37.696784 25868 leader_election.cc:288] T f7715d71dbac447b9351fc5eb36c5e80 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:37.724541 24468 raft_consensus.cc:1226] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Rejecting Update request from peer 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 2. Current term is 3. Ops: []
I20260809 15:46:37.405107 26192 raft_consensus.cc:2726] T 297c82ed906d4b48b184985eed3de601 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260809 15:46:37.742053 26257 raft_consensus.cc:482] T 0da4d5283c8a46878bd39b85e5c71f66 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:46:37.820479 25746 raft_consensus.cc:686] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 61e69695258f48baa984c9d395e8ce3f [term 4 LEADER]: Becoming Leader. State: Replica: 61e69695258f48baa984c9d395e8ce3f, State: Running, Role: LEADER
W20260809 15:46:37.542459 26221 log.cc:924] Time spent T 4b30f28311964a0b9149a50495e83d8f P 65c36d164b0540f099d592f38307ee6c: Append to log took a long time: real 0.086s user 0.000s sys 0.000s
I20260809 15:46:37.539198 26222 consensus_queue.cc:1046] T 0c29937a80724ddb9d07a27534a146f3 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.017s
I20260809 15:46:37.842722 26283 raft_consensus.cc:482] T c5c2e425a74b4e64ae70b5def750434c P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:37.843856 26222 raft_consensus.cc:482] T 05ad7d8fdca64e3da527280ca5be9280 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:37.844615 26222 raft_consensus.cc:504] T 05ad7d8fdca64e3da527280ca5be9280 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:37.745182 26198 raft_consensus.cc:504] T 7660dfa2ef694927b8eda4100a2bcd9b P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:37.747051 23402 rpcz_store.cc:276] 0809 15:46:36.571480 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:36.571594 (+ 114us) service_pool.cc:227] Handling call
0809 15:46:36.601713 (+ 30119us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 15:46:36.602926 (+ 1213us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:46:36.602952 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 15:46:36.612113 (+ 9161us) log.cc:842] Serialized 26 byte log entry
0809 15:46:37.617467 (+1005354us) raft_consensus.cc:1631] Marking committed up to 1
0809 15:46:37.617560 (+ 93us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:37.617674 (+ 114us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 15:46:37.731257 (+113583us) raft_consensus.cc:1671] finished
0809 15:46:37.731327 (+ 70us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:37.735387 (+ 4060us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":792,"raft.queue_time_us":1074,"raft.run_cpu_time_us":172,"raft.run_wall_time_us":16663,"thread_start_us":914935,"threads_started":1,"wal-append.queue_time_us":1042233}
W20260809 15:46:37.787108 24453 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.784s user 0.034s sys 0.095s
W20260809 15:46:37.848026 24453 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.949s user 0.035s sys 0.095s
I20260809 15:46:37.849220 24453 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38722 (request call id 379) took 1149 ms. Trace:
I20260809 15:46:37.849614 24453 rpcz_store.cc:276] 0809 15:46:36.699805 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:36.702502 (+ 2697us) service_pool.cc:227] Handling call
0809 15:46:36.755097 (+ 52595us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 15:46:36.807477 (+ 52380us) raft_consensus.cc:1520] Early marking committed up to index 1
0809 15:46:36.819966 (+ 12489us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 15:46:36.851960 (+ 31994us) log.cc:842] Serialized 26 byte log entry
0809 15:46:37.848612 (+996652us) raft_consensus.cc:1631] Marking committed up to 1
0809 15:46:37.848729 (+ 117us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:37.848831 (+ 102us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 15:46:37.848929 (+ 98us) raft_consensus.cc:1671] finished
0809 15:46:37.848945 (+ 16us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:37.849140 (+ 195us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":2942,"raft.run_cpu_time_us":118,"raft.run_wall_time_us":3612,"thread_start_us":949834,"threads_started":1,"wal-append.queue_time_us":971044}
I20260809 15:46:37.851644 26209 leader_election.cc:288] T f7715d71dbac447b9351fc5eb36c5e80 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:37.851822 23392 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55320 (request call id 1164) took 1287 ms. Trace:
I20260809 15:46:37.852267 23392 rpcz_store.cc:276] 0809 15:46:36.564309 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:36.566878 (+ 2569us) service_pool.cc:227] Handling call
0809 15:46:36.567502 (+ 624us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 15:46:36.568018 (+ 516us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:46:36.568090 (+ 72us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 15:46:36.568627 (+ 537us) log.cc:842] Serialized 26 byte log entry
0809 15:46:37.616440 (+1047813us) raft_consensus.cc:1631] Marking committed up to 2
0809 15:46:37.616553 (+ 113us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:37.616694 (+ 141us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 15:46:37.851431 (+234737us) raft_consensus.cc:1671] finished
0809 15:46:37.851451 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:37.851653 (+ 202us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1941,"raft.run_cpu_time_us":212,"raft.run_wall_time_us":22641,"thread_start_us":1046983,"threads_started":1,"wal-append.queue_time_us":1218519}
I20260809 15:46:37.744676 26158 raft_consensus.cc:2781] T 3924b648461e463b9d7925e5671d68ae P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Leader election won for term 5
I20260809 15:46:37.742195 26194 leader_election.cc:288] T df20671a7b874dbb98c0ef28371c1308 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
W20260809 15:46:37.819646 23472 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.056s user 0.030s sys 0.068s
W20260809 15:46:37.819620 25979 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.578s user 0.016s sys 0.041s
W20260809 15:46:37.854456 25979 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.747s user 0.017s sys 0.041s
I20260809 15:46:37.748894 26231 raft_consensus.cc:482] T 28e17dcb5efe45b8b6a8c42da03b4819 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:46:37.829720 25931 raft_consensus.cc:3037] T d2d4f749fcfd4fb3bfb8115af3471598 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:46:37.855516 25979 raft_consensus.cc:2781] T 0c48a3049897452383257aa757834fb5 P 61e69695258f48baa984c9d395e8ce3f [term 2 FOLLOWER]: Leader election won for term 2
I20260809 15:46:37.856254 23274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0acc47ce21c3422698d1ef7277c3ed63" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:37.857111 25941 raft_consensus.cc:3032] T cef3e7277d7d4760824a3436ed4a847d P 65c36d164b0540f099d592f38307ee6c [term 3 LEADER]: Stepping down as leader of term 3
I20260809 15:46:37.857553 25941 raft_consensus.cc:729] T cef3e7277d7d4760824a3436ed4a847d P 65c36d164b0540f099d592f38307ee6c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:46:37.859105 23399 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52856 (request call id 121) took 1950 ms. Trace:
I20260809 15:46:37.856178 25979 raft_consensus.cc:686] T 0c48a3049897452383257aa757834fb5 P 61e69695258f48baa984c9d395e8ce3f [term 2 LEADER]: Becoming Leader. State: Replica: 61e69695258f48baa984c9d395e8ce3f, State: Running, Role: LEADER
I20260809 15:46:37.858779 25941 consensus_queue.cc:260] T cef3e7277d7d4760824a3436ed4a847d P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:37.860514 25979 consensus_queue.cc:237] T 0c48a3049897452383257aa757834fb5 P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:37.745945 25650 raft_consensus.cc:504] T 2424fd9bd7f14e508fd106cbfda786a4 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:37.862584 26096 raft_consensus.cc:482] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:37.862216 26118 raft_consensus.cc:504] T 4bcac4b24e7a4448abdc934bba5a624e P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:37.855584 26231 raft_consensus.cc:3037] T 28e17dcb5efe45b8b6a8c42da03b4819 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:46:37.863250 26096 raft_consensus.cc:504] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:37.751852 25905 raft_consensus.cc:482] T 7baa2492df0340c4a5ac241959ce1ac3 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
I20260809 15:46:37.865947 25905 raft_consensus.cc:504] T 7baa2492df0340c4a5ac241959ce1ac3 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:37.867995 25905 leader_election.cc:288] T 7baa2492df0340c4a5ac241959ce1ac3 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:37.824615 25885 raft_consensus.cc:686] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 LEADER]: Becoming Leader. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Running, Role: LEADER
W20260809 15:46:37.834810 24410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 211900us
I20260809 15:46:37.833585 26257 raft_consensus.cc:3037] T 0da4d5283c8a46878bd39b85e5c71f66 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:46:37.868847 26148 consensus_queue.cc:260] T d0721a4403a440f383318a3cea9d1442 P c0f38c1908414566aa9115bb2cbdd85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:37.869560 25885 consensus_queue.cc:237] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:37.870671 26148 raft_consensus.cc:3037] T d0721a4403a440f383318a3cea9d1442 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:46:37.871905 23286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "213543cdbe814c89a17a963469922c1b" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:37.872939 23286 raft_consensus.cc:2379] T 213543cdbe814c89a17a963469922c1b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d in current term 3: Already voted for candidate 65c36d164b0540f099d592f38307ee6c in this term.
I20260809 15:46:37.873281 26179 consensus_queue.cc:1046] T cee02fff2c9c48a19f43eb358a89e7b0 P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260809 15:46:37.873900 23517 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56766 (request call id 1728) took 2613 ms. Trace:
I20260809 15:46:37.841320 26210 consensus_queue.cc:1046] T e14d118b2eaa42aa8a7cd4025aba7128 P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.120s
I20260809 15:46:37.874310 23517 rpcz_store.cc:276] 0809 15:46:35.260891 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:35.869365 (+608474us) service_pool.cc:227] Handling call
0809 15:46:35.869653 (+ 288us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 15:46:35.883310 (+ 13657us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:46:35.883343 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 15:46:35.883739 (+ 396us) log.cc:842] Serialized 26 byte log entry
0809 15:46:36.832854 (+949115us) raft_consensus.cc:1631] Marking committed up to 1
0809 15:46:36.833021 (+ 167us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:36.833087 (+ 66us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 15:46:37.873566 (+1040479us) raft_consensus.cc:1671] finished
0809 15:46:37.873603 (+ 37us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:37.873756 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":13282,"raft.queue_time_us":14325,"raft.run_cpu_time_us":158,"raft.run_wall_time_us":17364,"thread_start_us":845713,"threads_started":1,"wal-append.queue_time_us":1162351}
I20260809 15:46:37.875560 26210 consensus_queue.cc:1225] T e14d118b2eaa42aa8a7cd4025aba7128 P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Peer 65c36d164b0540f099d592f38307ee6c log is divergent from this leader: its last log entry 5.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260809 15:46:37.876303 26210 raft_consensus.cc:3032] T cee02fff2c9c48a19f43eb358a89e7b0 P 61e69695258f48baa984c9d395e8ce3f [term 2 LEADER]: Stepping down as leader of term 2
I20260809 15:46:37.842097 23520 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44018 (request call id 135) took 1668 ms. Trace:
I20260809 15:46:37.876506 23282 raft_consensus.cc:2454] T c0741ab44d99439abc3aa3bdd27ee160 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 3.
I20260809 15:46:37.876945 26210 raft_consensus.cc:729] T cee02fff2c9c48a19f43eb358a89e7b0 P 61e69695258f48baa984c9d395e8ce3f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 61e69695258f48baa984c9d395e8ce3f, State: Running, Role: LEADER
I20260809 15:46:37.828694 26065 raft_consensus.cc:3032] T 73f6f44c259547f1b05a3a434616fa54 P c0f38c1908414566aa9115bb2cbdd85d [term 3 LEADER]: Stepping down as leader of term 3
I20260809 15:46:37.879972 26065 raft_consensus.cc:729] T 73f6f44c259547f1b05a3a434616fa54 P c0f38c1908414566aa9115bb2cbdd85d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:46:37.878196 26210 consensus_queue.cc:260] T cee02fff2c9c48a19f43eb358a89e7b0 P 61e69695258f48baa984c9d395e8ce3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
W20260809 15:46:37.854288 23472 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.230s user 0.030s sys 0.069s
I20260809 15:46:37.881377 26065 consensus_queue.cc:260] T 73f6f44c259547f1b05a3a434616fa54 P c0f38c1908414566aa9115bb2cbdd85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:46:37.882261 23472 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 1.25756e+06us
I20260809 15:46:37.883993 23472 leader_election.cc:302] T 0c29937a80724ddb9d07a27534a146f3 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:37.884289 21289 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 15:46:37.884032 25650 leader_election.cc:288] T 2424fd9bd7f14e508fd106cbfda786a4 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:37.884886 26257 raft_consensus.cc:504] T 0da4d5283c8a46878bd39b85e5c71f66 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:37.885329 26198 leader_election.cc:288] T 7660dfa2ef694927b8eda4100a2bcd9b P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:37.887288 26257 leader_election.cc:288] T 0da4d5283c8a46878bd39b85e5c71f66 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 4 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:37.836918 25929 raft_consensus.cc:504] T 4564dfc6d4094c1b9240b367ccecf3b1 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:37.878238 25999 raft_consensus.cc:3032] T 998c6cee72cf40228c6dba064c0db2da P 65c36d164b0540f099d592f38307ee6c [term 5 LEADER]: Stepping down as leader of term 5
I20260809 15:46:37.888789 25999 raft_consensus.cc:729] T 998c6cee72cf40228c6dba064c0db2da P 65c36d164b0540f099d592f38307ee6c [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:46:37.890074 25929 leader_election.cc:288] T 4564dfc6d4094c1b9240b367ccecf3b1 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:37.889781 25999 consensus_queue.cc:260] T 998c6cee72cf40228c6dba064c0db2da P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:37.890957 25999 raft_consensus.cc:3037] T 998c6cee72cf40228c6dba064c0db2da P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Advancing to term 6
I20260809 15:46:37.876796 23520 rpcz_store.cc:276] 0809 15:46:36.065974 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:36.182697 (+116723us) service_pool.cc:227] Handling call
0809 15:46:36.183233 (+ 536us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 15:46:36.331648 (+148415us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:46:36.332871 (+ 1223us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 15:46:36.515649 (+182778us) log.cc:842] Serialized 26 byte log entry
0809 15:46:37.623873 (+1108224us) raft_consensus.cc:1631] Marking committed up to 4
0809 15:46:37.623987 (+ 114us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:37.624063 (+ 76us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 15:46:37.719476 (+ 95413us) raft_consensus.cc:1671] finished
0809 15:46:37.720287 (+ 811us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:37.733355 (+ 13068us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":87833,"raft.queue_time_us":109997,"raft.run_cpu_time_us":193,"raft.run_wall_time_us":31847,"thread_start_us":1025738,"threads_started":1,"wal-append.queue_time_us":1163071}
I20260809 15:46:37.864652 26118 leader_election.cc:288] T 4bcac4b24e7a4448abdc934bba5a624e P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:37.843607 26283 raft_consensus.cc:504] T c5c2e425a74b4e64ae70b5def750434c P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:37.746809 25863 leader_election.cc:288] T 953eea268e17491ca76f3a555860a512 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:37.894984 25695 leader_election.cc:288] T d2d4f749fcfd4fb3bfb8115af3471598 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:37.876185 23400 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52856 (request call id 120) took 2000 ms. Trace:
I20260809 15:46:37.892840 26158 raft_consensus.cc:686] T 3924b648461e463b9d7925e5671d68ae P 65c36d164b0540f099d592f38307ee6c [term 5 LEADER]: Becoming Leader. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:46:37.859483 23399 rpcz_store.cc:276] 0809 15:46:35.908534 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:35.969218 (+ 60684us) service_pool.cc:227] Handling call
0809 15:46:36.000951 (+ 31733us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 15:46:36.759801 (+758850us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:46:36.759832 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 15:46:36.831384 (+ 71552us) log.cc:842] Serialized 26 byte log entry
0809 15:46:37.308757 (+477373us) raft_consensus.cc:1631] Marking committed up to 4
0809 15:46:37.308787 (+ 30us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:37.308847 (+ 60us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 15:46:37.823773 (+514926us) raft_consensus.cc:1671] finished
0809 15:46:37.826139 (+ 2366us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:37.858950 (+ 32811us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":116798,"raft.run_cpu_time_us":173,"raft.run_wall_time_us":10757,"thread_start_us":1217682,"threads_started":2,"wal-append.queue_time_us":877883}
I20260809 15:46:37.896662 25967 raft_consensus.cc:3032] T 297c82ed906d4b48b184985eed3de601 P 65c36d164b0540f099d592f38307ee6c [term 4 LEADER]: Stepping down as leader of term 4
I20260809 15:46:37.883044 26065 raft_consensus.cc:3037] T 73f6f44c259547f1b05a3a434616fa54 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Advancing to term 4
W20260809 15:46:37.892014 23353 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 167251us
I20260809 15:46:37.837227 26206 raft_consensus.cc:686] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P c0f38c1908414566aa9115bb2cbdd85d [term 3 LEADER]: Becoming Leader. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:46:37.912286 24410 leader_election.cc:398] T 3924b648461e463b9d7925e5671d68ae P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 4 pre-election: Vote denied by peer 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553) with higher term. Message: Invalid argument: T 3924b648461e463b9d7925e5671d68ae P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 61e69695258f48baa984c9d395e8ce3f for earlier term 4. Current term is 5.
I20260809 15:46:37.912613 24410 leader_election.cc:401] T 3924b648461e463b9d7925e5671d68ae P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
I20260809 15:46:37.919227 23353 leader_election.cc:302] T 0acc47ce21c3422698d1ef7277c3ed63 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
W20260809 15:46:37.922485 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 613431us
I20260809 15:46:37.923364 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0da4d5283c8a46878bd39b85e5c71f66" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:37.923846 23521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bcac4b24e7a4448abdc934bba5a624e" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:37.924135 23527 raft_consensus.cc:3037] T 0da4d5283c8a46878bd39b85e5c71f66 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:46:37.924005 25746 consensus_queue.cc:237] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:37.894886 26228 consensus_queue.cc:1035] T d2d4f749fcfd4fb3bfb8115af3471598 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.030s
I20260809 15:46:37.925359 23519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7baa2492df0340c4a5ac241959ce1ac3" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:37.926079 23519 raft_consensus.cc:2454] T 7baa2492df0340c4a5ac241959ce1ac3 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 1.
I20260809 15:46:37.927152 25650 raft_consensus.cc:2726] T 0c29937a80724ddb9d07a27534a146f3 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260809 15:46:37.927474 23274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "297c82ed906d4b48b184985eed3de601" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:37.928604 23274 raft_consensus.cc:2379] T 297c82ed906d4b48b184985eed3de601 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d in current term 5: Already voted for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in this term.
I20260809 15:46:37.928745 23517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7660dfa2ef694927b8eda4100a2bcd9b" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:37.929517 23517 raft_consensus.cc:2454] T 7660dfa2ef694927b8eda4100a2bcd9b P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 4.
I20260809 15:46:37.929667 23274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "763e58d6ba644eea830a6c1664d39c9b" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
W20260809 15:46:37.928750 24406 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 126978us
I20260809 15:46:37.930536 23274 raft_consensus.cc:2379] T 763e58d6ba644eea830a6c1664d39c9b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d in current term 3: Already voted for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in this term.
I20260809 15:46:37.932090 23274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07fcabb569414bddbd39df6cedd24e64" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:37.927531 23526 raft_consensus.cc:1226] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Rejecting Update request from peer c0f38c1908414566aa9115bb2cbdd85d for earlier term 1. Current term is 2. Ops: []
I20260809 15:46:37.907531 25967 raft_consensus.cc:729] T 297c82ed906d4b48b184985eed3de601 P 65c36d164b0540f099d592f38307ee6c [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:46:37.923774 23228 leader_election.cc:302] T d0721a4403a440f383318a3cea9d1442 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:37.936937 23280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "953eea268e17491ca76f3a555860a512" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:37.954115 23517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05ad7d8fdca64e3da527280ca5be9280" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:37.931314 23276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f9d6565aa4c45a3b3f01210d0eb71f7" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:37.959204 23276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3296ef50be254fc29cfc77d22a8f7a23" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:37.879698 25980 consensus_queue.cc:1035] T e14d118b2eaa42aa8a7cd4025aba7128 P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.057s
I20260809 15:46:37.961900 23353 leader_election.cc:302] T 7baa2492df0340c4a5ac241959ce1ac3 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:46:37.900658 23400 rpcz_store.cc:276] 0809 15:46:35.875849 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:35.905521 (+ 29672us) service_pool.cc:227] Handling call
0809 15:46:35.969271 (+ 63750us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 15:46:35.969759 (+ 488us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:46:35.969786 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 15:46:35.982764 (+ 12978us) log.cc:842] Serialized 26 byte log entry
0809 15:46:36.833634 (+850870us) raft_consensus.cc:1631] Marking committed up to 4
0809 15:46:36.833672 (+ 38us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:36.833741 (+ 69us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 15:46:37.875750 (+1042009us) raft_consensus.cc:1671] finished
0809 15:46:37.875794 (+ 44us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:37.875998 (+ 204us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":186,"raft.queue_time_us":18894,"raft.run_cpu_time_us":110,"raft.run_wall_time_us":5783,"thread_start_us":850563,"threads_started":1,"wal-append.queue_time_us":1054574}
I20260809 15:46:37.963991 23276 raft_consensus.cc:2454] T 3296ef50be254fc29cfc77d22a8f7a23 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 2.
I20260809 15:46:37.964555 23273 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55434 (request call id 148) took 1039 ms. Trace:
I20260809 15:46:37.920816 26118 raft_consensus.cc:2781] T 0acc47ce21c3422698d1ef7277c3ed63 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Leader election won for term 4
I20260809 15:46:37.965113 26161 raft_consensus.cc:482] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
I20260809 15:46:37.965767 26161 raft_consensus.cc:504] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:37.965852 25967 consensus_queue.cc:260] T 297c82ed906d4b48b184985eed3de601 P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:37.967880 25967 raft_consensus.cc:3037] T 297c82ed906d4b48b184985eed3de601 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Advancing to term 5
I20260809 15:46:37.969534 26206 consensus_queue.cc:237] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:37.970747 23280 raft_consensus.cc:2454] T 953eea268e17491ca76f3a555860a512 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 3.
I20260809 15:46:37.971395 26040 raft_consensus.cc:482] T ba56725e42f74a1baa4c04d9660727f9 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:37.971607 23280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f9d6565aa4c45a3b3f01210d0eb71f7" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:37.971887 26279 consensus_queue.cc:1046] T 294d4bcc1e7a477cb56455581277a60f P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260809 15:46:37.972294 26158 consensus_queue.cc:237] T 3924b648461e463b9d7925e5671d68ae P 65c36d164b0540f099d592f38307ee6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:37.973762 24453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bcac4b24e7a4448abdc934bba5a624e" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:37.974668 25941 raft_consensus.cc:3037] T cef3e7277d7d4760824a3436ed4a847d P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:46:37.975121 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7660dfa2ef694927b8eda4100a2bcd9b" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:37.977198 23345 leader_election.cc:302] T 4bcac4b24e7a4448abdc934bba5a624e P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c
I20260809 15:46:37.978500 26118 raft_consensus.cc:686] T 0acc47ce21c3422698d1ef7277c3ed63 P c0f38c1908414566aa9115bb2cbdd85d [term 4 LEADER]: Becoming Leader. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:46:37.979861 26118 consensus_queue.cc:237] T 0acc47ce21c3422698d1ef7277c3ed63 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:46:37.960597 24410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 47261.5us
I20260809 15:46:37.971716 26096 leader_election.cc:288] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:37.972751 23280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae6d2c08a7a94f58ac8dd698e3fd456c" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:37.964880 23273 rpcz_store.cc:276] 0809 15:46:36.924782 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:36.924903 (+ 121us) service_pool.cc:227] Handling call
0809 15:46:37.964367 (+1039464us) spinlock_profiling.cc:243] Waited 820 ms on lock 0x7b5400089b54. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e92242ae 00007f34ee161c27 00007f34e7849c04 00007f34e7849b92 00007f34e7849ac1 00007f34e7849a3c 00007f34e7848cf2 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:46:37.964500 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1886270720}
I20260809 15:46:37.967396 23283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "953eea268e17491ca76f3a555860a512" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:37.936847 26222 leader_election.cc:288] T 05ad7d8fdca64e3da527280ca5be9280 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:38.020367 23283 raft_consensus.cc:2362] T 953eea268e17491ca76f3a555860a512 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 3. Re-sending same reply.
I20260809 15:46:38.022534 23353 leader_election.cc:302] T 7660dfa2ef694927b8eda4100a2bcd9b P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters: 61e69695258f48baa984c9d395e8ce3f
I20260809 15:46:38.021750 23527 raft_consensus.cc:2454] T 0da4d5283c8a46878bd39b85e5c71f66 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 4.
I20260809 15:46:38.024690 23274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998c6cee72cf40228c6dba064c0db2da" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:38.028295 23283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e71a05ba1d4d42bd95c3f85f9313399e" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:38.028209 23273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0da4d5283c8a46878bd39b85e5c71f66" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:37.978471 26259 raft_consensus.cc:2726] T 4bcac4b24e7a4448abdc934bba5a624e P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 15:46:38.021468 25956 raft_consensus.cc:2781] T 7baa2492df0340c4a5ac241959ce1ac3 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260809 15:46:37.985714 26283 leader_election.cc:288] T c5c2e425a74b4e64ae70b5def750434c P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:38.033774 26040 raft_consensus.cc:504] T ba56725e42f74a1baa4c04d9660727f9 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:38.038062 23117 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:37326 (request call id 148) took 9771 ms. Trace:
I20260809 15:46:38.039091 23273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7715d71dbac447b9351fc5eb36c5e80" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:38.039320 26222 raft_consensus.cc:2781] T 7660dfa2ef694927b8eda4100a2bcd9b P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260809 15:46:38.045274 24406 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 113884us
W20260809 15:46:38.040138 23470 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 78447.8us
W20260809 15:46:38.040575 23223 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.68:34689, user_credentials={real_user=slave}} blocked reactor thread for 462574us
I20260809 15:46:38.040351 25899 consensus_queue.cc:260] T 0c29937a80724ddb9d07a27534a146f3 P c0f38c1908414566aa9115bb2cbdd85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:37.966346 25650 consensus_queue.cc:1035] T 6ae14a74a26f434db08c652e097385c8 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.064s
I20260809 15:46:38.019758 23286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e4ffcf83c234ae7910ddef3ec0b4bda" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:38.022203 25885 raft_consensus.cc:2726] T d0721a4403a440f383318a3cea9d1442 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260809 15:46:38.033396 23280 raft_consensus.cc:2379] T ae6d2c08a7a94f58ac8dd698e3fd456c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d in current term 4: Already voted for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in this term.
I20260809 15:46:37.964979 23276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2db057fd10e149a9895dff6ef8c9f038" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:38.044757 23273 raft_consensus.cc:2454] T f7715d71dbac447b9351fc5eb36c5e80 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 3.
I20260809 15:46:38.046022 23274 raft_consensus.cc:2379] T 998c6cee72cf40228c6dba064c0db2da P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d in current term 6: Already voted for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in this term.
I20260809 15:46:38.032348 25979 raft_consensus.cc:2726] T 3924b648461e463b9d7925e5671d68ae P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553) with higher term. Message: Invalid argument: T 3924b648461e463b9d7925e5671d68ae P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 61e69695258f48baa984c9d395e8ce3f for earlier term 4. Current term is 5.
I20260809 15:46:38.048573 23283 raft_consensus.cc:2379] T e71a05ba1d4d42bd95c3f85f9313399e P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d in current term 3: Already voted for candidate 61e69695258f48baa984c9d395e8ce3f in this term.
W20260809 15:46:38.048784 23472 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 164987us
I20260809 15:46:38.049819 23353 leader_election.cc:302] T 0da4d5283c8a46878bd39b85e5c71f66 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:46:38.043558 25956 raft_consensus.cc:482] T 7baa2492df0340c4a5ac241959ce1ac3 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
I20260809 15:46:38.045688 26222 raft_consensus.cc:482] T 7660dfa2ef694927b8eda4100a2bcd9b P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Starting leader election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
I20260809 15:46:38.074554 23393 raft_consensus.cc:1261] T d2d4f749fcfd4fb3bfb8115af3471598 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 2. (term mismatch)
I20260809 15:46:38.074916 23393 pending_rounds.cc:76] T d2d4f749fcfd4fb3bfb8115af3471598 P c0f38c1908414566aa9115bb2cbdd85d: Aborting all ops after (but not including) 1
I20260809 15:46:38.075160 23393 pending_rounds.cc:97] T d2d4f749fcfd4fb3bfb8115af3471598 P c0f38c1908414566aa9115bb2cbdd85d: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20260809 15:46:38.075551 23393 raft_consensus.cc:2866] T d2d4f749fcfd4fb3bfb8115af3471598 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 15:46:38.077211 23285 raft_consensus.cc:1226] T d0721a4403a440f383318a3cea9d1442 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Rejecting Update request from peer c0f38c1908414566aa9115bb2cbdd85d for earlier term 3. Current term is 4. Ops: []
I20260809 15:46:38.080185 26275 consensus_queue.cc:1046] T 2db057fd10e149a9895dff6ef8c9f038 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260809 15:46:38.081720 26275 consensus_queue.cc:1035] T 0da4d5283c8a46878bd39b85e5c71f66 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 15:46:38.083462 25929 raft_consensus.cc:3032] T 2db057fd10e149a9895dff6ef8c9f038 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 LEADER]: Stepping down as leader of term 2
I20260809 15:46:38.083897 25929 raft_consensus.cc:729] T 2db057fd10e149a9895dff6ef8c9f038 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Running, Role: LEADER
I20260809 15:46:38.084424 25899 raft_consensus.cc:3037] T 0c29937a80724ddb9d07a27534a146f3 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:46:38.064978 25956 raft_consensus.cc:3037] T 7baa2492df0340c4a5ac241959ce1ac3 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:38.087663 26026 consensus_queue.cc:1046] T 07fcabb569414bddbd39df6cedd24e64 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260809 15:46:38.089154 26194 raft_consensus.cc:3032] T 07fcabb569414bddbd39df6cedd24e64 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 LEADER]: Stepping down as leader of term 3
I20260809 15:46:38.089573 26194 raft_consensus.cc:729] T 07fcabb569414bddbd39df6cedd24e64 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Running, Role: LEADER
I20260809 15:46:38.084689 23277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0acc47ce21c3422698d1ef7277c3ed63" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:38.090468 26194 consensus_queue.cc:260] T 07fcabb569414bddbd39df6cedd24e64 P 67b3aeadfa7a432abe8af4d292e04ab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:38.091822 26194 raft_consensus.cc:3037] T 07fcabb569414bddbd39df6cedd24e64 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:46:38.092796 23470 leader_election.cc:302] T f7715d71dbac447b9351fc5eb36c5e80 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:46:38.094070 23470 leader_election.cc:302] T c0741ab44d99439abc3aa3bdd27ee160 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:46:38.094735 25793 raft_consensus.cc:2781] T f7715d71dbac447b9351fc5eb36c5e80 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260809 15:46:38.095182 25793 raft_consensus.cc:482] T f7715d71dbac447b9351fc5eb36c5e80 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:46:38.095531 25793 raft_consensus.cc:3037] T f7715d71dbac447b9351fc5eb36c5e80 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:46:38.095750 26139 raft_consensus.cc:2781] T c0741ab44d99439abc3aa3bdd27ee160 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260809 15:46:38.096112 26139 raft_consensus.cc:482] T c0741ab44d99439abc3aa3bdd27ee160 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:46:38.096446 26139 raft_consensus.cc:3037] T c0741ab44d99439abc3aa3bdd27ee160 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:46:38.084895 23472 leader_election.cc:302] T d0721a4403a440f383318a3cea9d1442 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:38.074911 26185 leader_election.cc:288] T 953eea268e17491ca76f3a555860a512 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:38.055083 23286 raft_consensus.cc:2354] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d for earlier term 3. Current term is 4.
W20260809 15:46:38.100786 23353 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 51043.6us
I20260809 15:46:38.056576 26161 leader_election.cc:288] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:38.103932 23282 raft_consensus.cc:1226] T 763e58d6ba644eea830a6c1664d39c9b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 2. Current term is 3. Ops: []
I20260809 15:46:38.105038 23520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7715d71dbac447b9351fc5eb36c5e80" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:38.105456 23282 raft_consensus.cc:1226] T cef3e7277d7d4760824a3436ed4a847d P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260809 15:46:38.106236 23270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2424fd9bd7f14e508fd106cbfda786a4" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 5 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:38.106463 23282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ced718c1a5c4da79cb8c1ebd1435b87" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:38.106804 24410 leader_election.cc:302] T 472816994d814cb198b69e5248697440 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:38.084969 25929 consensus_queue.cc:260] T 2db057fd10e149a9895dff6ef8c9f038 P 67b3aeadfa7a432abe8af4d292e04ab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:38.112598 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5c2e425a74b4e64ae70b5def750434c" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:38.106899 23270 raft_consensus.cc:2454] T 2424fd9bd7f14e508fd106cbfda786a4 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 4.
I20260809 15:46:38.077335 23393 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55320 (request call id 1166) took 1505 ms. Trace:
I20260809 15:46:38.066118 26222 raft_consensus.cc:3037] T 7660dfa2ef694927b8eda4100a2bcd9b P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Advancing to term 5
I20260809 15:46:38.108351 25931 raft_consensus.cc:2781] T 0da4d5283c8a46878bd39b85e5c71f66 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Leader election won for term 4
I20260809 15:46:38.117846 23277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "953eea268e17491ca76f3a555860a512" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:38.120149 23285 raft_consensus.cc:1226] T 998c6cee72cf40228c6dba064c0db2da P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20260809 15:46:38.121343 24410 leader_election.cc:398] T 3924b648461e463b9d7925e5671d68ae P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 4 pre-election: Vote denied by peer c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263) with higher term. Message: Invalid argument: T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 61e69695258f48baa984c9d395e8ce3f for earlier term 4. Current term is 5.
I20260809 15:46:38.122648 26301 raft_consensus.cc:2726] T 472816994d814cb198b69e5248697440 P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260809 15:46:38.122781 26284 raft_consensus.cc:482] T 6ced718c1a5c4da79cb8c1ebd1435b87 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:38.125550 23472 leader_election.cc:302] T 7660dfa2ef694927b8eda4100a2bcd9b P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:38.126166 25999 raft_consensus.cc:3037] T d0721a4403a440f383318a3cea9d1442 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:46:38.127429 25863 consensus_queue.cc:1046] T 763e58d6ba644eea830a6c1664d39c9b P 65c36d164b0540f099d592f38307ee6c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 15:46:38.119163 23277 raft_consensus.cc:2454] T 953eea268e17491ca76f3a555860a512 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 3.
I20260809 15:46:38.131129 23270 raft_consensus.cc:1226] T 297c82ed906d4b48b184985eed3de601 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 4. Current term is 5. Ops: []
I20260809 15:46:38.135294 26096 raft_consensus.cc:2726] T 7660dfa2ef694927b8eda4100a2bcd9b P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260809 15:46:38.135761 26290 log.cc:924] Time spent T 294d4bcc1e7a477cb56455581277a60f P 65c36d164b0540f099d592f38307ee6c: Append to log took a long time: real 0.094s user 0.000s sys 0.001s
I20260809 15:46:38.135978 26284 raft_consensus.cc:504] T 6ced718c1a5c4da79cb8c1ebd1435b87 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:38.137313 26286 raft_consensus.cc:3032] T e14d118b2eaa42aa8a7cd4025aba7128 P 61e69695258f48baa984c9d395e8ce3f [term 4 LEADER]: Stepping down as leader of term 4
I20260809 15:46:38.137706 26286 raft_consensus.cc:729] T e14d118b2eaa42aa8a7cd4025aba7128 P 61e69695258f48baa984c9d395e8ce3f [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 61e69695258f48baa984c9d395e8ce3f, State: Running, Role: LEADER
I20260809 15:46:38.138088 26284 leader_election.cc:288] T 6ced718c1a5c4da79cb8c1ebd1435b87 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:38.139508 23526 raft_consensus.cc:1226] T 0c29937a80724ddb9d07a27534a146f3 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Rejecting Update request from peer c0f38c1908414566aa9115bb2cbdd85d for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260809 15:46:38.139075 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ced718c1a5c4da79cb8c1ebd1435b87" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:38.140462 23527 raft_consensus.cc:2454] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 2.
I20260809 15:46:38.141559 26053 raft_consensus.cc:504] T ae6d2c08a7a94f58ac8dd698e3fd456c P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:38.051105 26259 consensus_queue.cc:1046] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.074s
I20260809 15:46:38.115067 23393 rpcz_store.cc:276] 0809 15:46:36.571910 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:36.590629 (+ 18719us) service_pool.cc:227] Handling call
0809 15:46:36.684082 (+ 93453us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:38.076246 (+1392164us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:38.077079 (+ 833us) spinlock_profiling.cc:243] Waited 1.38 s on lock 0x7b54001cf7d4. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e9220f51 00007f34e921f99e 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772 00007f34e764a94c 00007f34e7649d86
0809 15:46:38.077286 (+ 207us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3171512192}
I20260809 15:46:38.149449 26148 raft_consensus.cc:482] T ba56725e42f74a1baa4c04d9660727f9 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:38.151695 26148 raft_consensus.cc:504] T ba56725e42f74a1baa4c04d9660727f9 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:38.153520 23228 leader_election.cc:302] T 763e58d6ba644eea830a6c1664d39c9b P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:38.162624 23351 leader_election.cc:302] T 213543cdbe814c89a17a963469922c1b P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:38.163288 26278 raft_consensus.cc:482] T 538a228ddfb54bc6a36528d5ff07597b P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:38.149533 25931 raft_consensus.cc:686] T 0da4d5283c8a46878bd39b85e5c71f66 P c0f38c1908414566aa9115bb2cbdd85d [term 4 LEADER]: Becoming Leader. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:46:38.158469 23353 leader_election.cc:302] T 6ced718c1a5c4da79cb8c1ebd1435b87 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:46:38.138654 26286 consensus_queue.cc:260] T e14d118b2eaa42aa8a7cd4025aba7128 P 61e69695258f48baa984c9d395e8ce3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:38.123510 26301 consensus_queue.cc:1046] T cee02fff2c9c48a19f43eb358a89e7b0 P 61e69695258f48baa984c9d395e8ce3f [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.045s
I20260809 15:46:38.176951 23470 leader_election.cc:302] T 2424fd9bd7f14e508fd106cbfda786a4 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:46:38.166540 25959 raft_consensus.cc:482] T a88421dba80d4d3388aa5eb62e23853b P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:38.169407 25929 raft_consensus.cc:3037] T 2db057fd10e149a9895dff6ef8c9f038 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:38.180378 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae6d2c08a7a94f58ac8dd698e3fd456c" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:38.168247 26284 raft_consensus.cc:482] T 563daf42d7784db195ea99c3cb21b370 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
I20260809 15:46:38.179891 23470 leader_election.cc:302] T 953eea268e17491ca76f3a555860a512 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:46:38.169456 26053 leader_election.cc:288] T ae6d2c08a7a94f58ac8dd698e3fd456c P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 4 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:38.143133 23116 catalog_manager.cc:5495] T d0721a4403a440f383318a3cea9d1442 P c0f38c1908414566aa9115bb2cbdd85d reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "c0f38c1908414566aa9115bb2cbdd85d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } health_report { overall_health: UNKNOWN } } }
I20260809 15:46:38.044513 23117 rpcz_store.cc:276] 0809 15:46:28.268422 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:28.282428 (+ 14006us) service_pool.cc:227] Handling call
0809 15:46:38.037817 (+9755389us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0809 15:46:34.596065 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0809 15:46:34.596550 (+ 485us) write_op.cc:432] Acquiring schema lock in shared mode
0809 15:46:34.596580 (+ 30us) write_op.cc:435] Acquired schema lock
0809 15:46:34.596595 (+ 15us) tablet.cc:660] Decoding operations
0809 15:46:34.596910 (+ 315us) write_op.cc:620] Acquiring the partition lock for write op
0809 15:46:34.600573 (+ 3663us) write_op.cc:641] Partition lock acquired for write op
0809 15:46:34.602771 (+ 2198us) tablet.cc:683] Acquiring locks for 1 operations
0809 15:46:34.612940 (+ 10169us) tablet.cc:699] Row locks acquired
0809 15:46:34.612959 (+ 19us) write_op.cc:260] PREPARE: finished
0809 15:46:34.614872 (+ 1913us) write_op.cc:270] Start()
0809 15:46:34.614991 (+ 119us) write_op.cc:276] Timestamp: P: 1786290394614827 usec, L: 0
0809 15:46:34.615011 (+ 20us) op_driver.cc:331] REPLICATION: starting
0809 15:46:34.624227 (+ 9216us) log.cc:842] Serialized 536 byte log entry
0809 15:46:36.119739 (+1495512us) op_driver.cc:444] REPLICATION: finished
0809 15:46:37.490903 (+1371164us) write_op.cc:301] APPLY: starting
0809 15:46:37.505263 (+ 14360us) tablet.cc:1365] starting BulkCheckPresence
0809 15:46:37.532310 (+ 27047us) tablet.cc:1368] finished BulkCheckPresence
0809 15:46:37.543122 (+ 10812us) tablet.cc:1370] starting ApplyRowOperation cycle
0809 15:46:37.545066 (+ 1944us) tablet.cc:1381] finished ApplyRowOperation cycle
0809 15:46:37.545223 (+ 157us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0809 15:46:37.545266 (+ 43us) write_op.cc:312] APPLY: finished
0809 15:46:37.548021 (+ 2755us) log.cc:842] Serialized 23 byte log entry
0809 15:46:37.927259 (+379238us) write_op.cc:489] Releasing partition, row and schema locks
0809 15:46:37.927448 (+ 189us) write_op.cc:454] Released schema lock
0809 15:46:37.927525 (+ 77us) write_op.cc:341] FINISH: Updating metrics
0809 15:46:37.928033 (+ 508us) write_op.cc:489] Releasing partition, row and schema locks
0809 15:46:37.928073 (+ 40us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":1646605,"reported_tablets":3,"child_traces":[["op",{"apply.queue_time_us":1283718,"apply.run_cpu_time_us":12537,"apply.run_wall_time_us":471784,"num_ops":1,"prepare.queue_time_us":482868,"prepare.run_cpu_time_us":33886,"prepare.run_wall_time_us":368781,"raft.queue_time_us":361207,"raft.run_cpu_time_us":31889,"raft.run_wall_time_us":1139690,"replication_time_us":1504573,"thread_start_us":2239281,"threads_started":5,"wal-append.queue_time_us":769323,"wal-append.run_cpu_time_us":67537,"wal-append.run_wall_time_us":1782018}]]}
I20260809 15:46:38.135799 26293 consensus_queue.cc:1035] T 294d4bcc1e7a477cb56455581277a60f P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.044s
I20260809 15:46:38.174315 26148 leader_election.cc:288] T ba56725e42f74a1baa4c04d9660727f9 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:38.183187 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba56725e42f74a1baa4c04d9660727f9" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:38.176124 25885 raft_consensus.cc:2726] T 763e58d6ba644eea830a6c1664d39c9b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260809 15:46:38.191481 23282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7715d71dbac447b9351fc5eb36c5e80" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:38.198628 23282 raft_consensus.cc:2454] T f7715d71dbac447b9351fc5eb36c5e80 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 3.
I20260809 15:46:38.199585 26158 raft_consensus.cc:2781] T 953eea268e17491ca76f3a555860a512 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260809 15:46:38.200114 26158 raft_consensus.cc:482] T 953eea268e17491ca76f3a555860a512 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:46:38.200565 26158 raft_consensus.cc:3037] T 953eea268e17491ca76f3a555860a512 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:46:38.202821 23351 leader_election.cc:302] T 297c82ed906d4b48b184985eed3de601 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:38.180608 25941 raft_consensus.cc:2781] T 2424fd9bd7f14e508fd106cbfda786a4 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260809 15:46:38.211724 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 76925.1us
I20260809 15:46:38.237063 23282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba56725e42f74a1baa4c04d9660727f9" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:38.247406 26285 raft_consensus.cc:2726] T 213543cdbe814c89a17a963469922c1b P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260809 15:46:38.248023 23286 raft_consensus.cc:1261] T e14d118b2eaa42aa8a7cd4025aba7128 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260809 15:46:38.249094 23286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0da4d5283c8a46878bd39b85e5c71f66" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:38.246654 26284 raft_consensus.cc:504] T 563daf42d7784db195ea99c3cb21b370 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:38.253727 23280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05ad7d8fdca64e3da527280ca5be9280" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:38.234650 24468 raft_consensus.cc:1226] T 472816994d814cb198b69e5248697440 P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Rejecting Update request from peer c0f38c1908414566aa9115bb2cbdd85d for earlier term 3. Current term is 4. Ops: []
I20260809 15:46:38.212396 25941 raft_consensus.cc:482] T 2424fd9bd7f14e508fd106cbfda786a4 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:46:38.173535 26278 raft_consensus.cc:504] T 538a228ddfb54bc6a36528d5ff07597b P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:38.258335 23282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ced718c1a5c4da79cb8c1ebd1435b87" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:38.249723 26053 raft_consensus.cc:3037] T 297c82ed906d4b48b184985eed3de601 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Advancing to term 5
I20260809 15:46:38.263832 26144 raft_consensus.cc:2781] T 6ced718c1a5c4da79cb8c1ebd1435b87 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260809 15:46:38.264313 26144 raft_consensus.cc:482] T 6ced718c1a5c4da79cb8c1ebd1435b87 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:46:38.264647 26144 raft_consensus.cc:3037] T 6ced718c1a5c4da79cb8c1ebd1435b87 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:38.265048 23400 raft_consensus.cc:1261] T 4564dfc6d4094c1b9240b367ccecf3b1 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260809 15:46:38.265534 25863 raft_consensus.cc:482] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:38.266229 26278 leader_election.cc:288] T 538a228ddfb54bc6a36528d5ff07597b P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:38.266059 25863 raft_consensus.cc:504] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:38.268173 25863 leader_election.cc:288] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:38.268869 23395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0da4d5283c8a46878bd39b85e5c71f66" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:38.269327 25875 consensus_queue.cc:1035] T d2d4f749fcfd4fb3bfb8115af3471598 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.003s
I20260809 15:46:38.272581 23397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9e7cadcd24d45fcbfa84209225e8f2d" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:38.256675 25941 raft_consensus.cc:3037] T 2424fd9bd7f14e508fd106cbfda786a4 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Advancing to term 5
I20260809 15:46:38.273262 23397 raft_consensus.cc:2454] T f9e7cadcd24d45fcbfa84209225e8f2d P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 3.
I20260809 15:46:38.274011 23228 leader_election.cc:302] T 99bdb93503db4de4a1cfb0161ea0c07c P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:46:38.275399 26177 raft_consensus.cc:482] T c5c2e425a74b4e64ae70b5def750434c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:38.276010 25956 raft_consensus.cc:504] T 7baa2492df0340c4a5ac241959ce1ac3 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:38.276270 25885 consensus_queue.cc:1035] T f9e7cadcd24d45fcbfa84209225e8f2d P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 15:46:38.277622 25885 raft_consensus.cc:2781] T 99bdb93503db4de4a1cfb0161ea0c07c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 15:46:38.278049 25885 raft_consensus.cc:686] T 99bdb93503db4de4a1cfb0161ea0c07c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 LEADER]: Becoming Leader. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Running, Role: LEADER
I20260809 15:46:38.279827 25931 consensus_queue.cc:237] T 0da4d5283c8a46878bd39b85e5c71f66 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:38.281452 25999 raft_consensus.cc:2726] T d0721a4403a440f383318a3cea9d1442 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260809 15:46:38.282218 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 79525.6us
I20260809 15:46:38.271461 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3aa6a9a52804df0ba9b0ca3c82f38c3" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:38.283157 23280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5c2e425a74b4e64ae70b5def750434c" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
W20260809 15:46:38.267153 26292 log.cc:924] Time spent T 0c48a3049897452383257aa757834fb5 P 61e69695258f48baa984c9d395e8ce3f: Append to log took a long time: real 0.132s user 0.000s sys 0.001s
I20260809 15:46:38.284775 23527 raft_consensus.cc:2454] T ae6d2c08a7a94f58ac8dd698e3fd456c P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 4.
I20260809 15:46:38.275163 23228 leader_election.cc:302] T 28e17dcb5efe45b8b6a8c42da03b4819 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:46:38.264559 26284 leader_election.cc:288] T 563daf42d7784db195ea99c3cb21b370 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:38.263094 25650 raft_consensus.cc:482] T df20671a7b874dbb98c0ef28371c1308 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:38.236081 25959 raft_consensus.cc:504] T a88421dba80d4d3388aa5eb62e23853b P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:38.297340 23353 leader_election.cc:302] T ae6d2c08a7a94f58ac8dd698e3fd456c P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
W20260809 15:46:38.294869 24406 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 182734us
I20260809 15:46:38.293574 23282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7baa2492df0340c4a5ac241959ce1ac3" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:38.277971 25956 leader_election.cc:288] T 7baa2492df0340c4a5ac241959ce1ac3 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 2 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:38.296603 23517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "538a228ddfb54bc6a36528d5ff07597b" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:38.303622 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7baa2492df0340c4a5ac241959ce1ac3" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:38.304216 23526 raft_consensus.cc:3037] T 7baa2492df0340c4a5ac241959ce1ac3 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:38.304862 23351 leader_election.cc:302] T 763e58d6ba644eea830a6c1664d39c9b P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:38.258951 25967 consensus_queue.cc:1035] T e14d118b2eaa42aa8a7cd4025aba7128 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.034s
I20260809 15:46:38.279066 25885 consensus_queue.cc:237] T 99bdb93503db4de4a1cfb0161ea0c07c P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:38.290072 26231 raft_consensus.cc:504] T 28e17dcb5efe45b8b6a8c42da03b4819 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:38.275977 26177 raft_consensus.cc:504] T c5c2e425a74b4e64ae70b5def750434c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:38.315557 25956 raft_consensus.cc:2781] T ae6d2c08a7a94f58ac8dd698e3fd456c P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Leader election won for term 4
I20260809 15:46:38.314204 26194 raft_consensus.cc:2781] T 28e17dcb5efe45b8b6a8c42da03b4819 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260809 15:46:38.336275 26333 raft_consensus.cc:482] T c0741ab44d99439abc3aa3bdd27ee160 P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:38.338176 26222 raft_consensus.cc:504] T 7660dfa2ef694927b8eda4100a2bcd9b P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
W20260809 15:46:38.336367 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 53555.6us
I20260809 15:46:38.361946 26333 raft_consensus.cc:504] T c0741ab44d99439abc3aa3bdd27ee160 P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:38.341346 26194 raft_consensus.cc:482] T 28e17dcb5efe45b8b6a8c42da03b4819 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:46:38.363147 26222 leader_election.cc:288] T 7660dfa2ef694927b8eda4100a2bcd9b P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 election: Requested vote from peers 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:38.363610 26194 raft_consensus.cc:3037] T 28e17dcb5efe45b8b6a8c42da03b4819 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:46:38.364183 26333 leader_election.cc:288] T c0741ab44d99439abc3aa3bdd27ee160 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:38.332500 26118 raft_consensus.cc:3037] T 763e58d6ba644eea830a6c1664d39c9b P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:38.298631 26158 raft_consensus.cc:504] T 953eea268e17491ca76f3a555860a512 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:38.312568 25650 raft_consensus.cc:504] T df20671a7b874dbb98c0ef28371c1308 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:38.366822 23282 raft_consensus.cc:1226] T 763e58d6ba644eea830a6c1664d39c9b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 2. Current term is 3. Ops: []
I20260809 15:46:38.375984 23280 raft_consensus.cc:2454] T c5c2e425a74b4e64ae70b5def750434c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 3.
I20260809 15:46:38.316676 25980 consensus_queue.cc:1046] T 907bacfa79d04a55a68c8dbe60056462 P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.026s
I20260809 15:46:38.376964 26326 consensus_queue.cc:1046] T 563daf42d7784db195ea99c3cb21b370 P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.005s
I20260809 15:46:38.382737 26326 raft_consensus.cc:3032] T 907bacfa79d04a55a68c8dbe60056462 P 61e69695258f48baa984c9d395e8ce3f [term 2 LEADER]: Stepping down as leader of term 2
I20260809 15:46:38.391676 23526 raft_consensus.cc:2454] T 7baa2492df0340c4a5ac241959ce1ac3 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 2.
I20260809 15:46:38.364183 26053 raft_consensus.cc:2726] T 297c82ed906d4b48b184985eed3de601 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260809 15:46:38.367213 25956 raft_consensus.cc:686] T ae6d2c08a7a94f58ac8dd698e3fd456c P c0f38c1908414566aa9115bb2cbdd85d [term 4 LEADER]: Becoming Leader. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:46:38.408704 25956 consensus_queue.cc:237] T ae6d2c08a7a94f58ac8dd698e3fd456c P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:38.412222 26231 leader_election.cc:288] T 28e17dcb5efe45b8b6a8c42da03b4819 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 4 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:38.413604 26177 leader_election.cc:288] T c5c2e425a74b4e64ae70b5def750434c P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:38.415429 25899 raft_consensus.cc:482] T 00a1881267a14ba8b78c81040f17a014 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:38.415532 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7660dfa2ef694927b8eda4100a2bcd9b" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:38.416998 23526 raft_consensus.cc:3037] T 7660dfa2ef694927b8eda4100a2bcd9b P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Advancing to term 5
I20260809 15:46:38.419416 26303 raft_consensus.cc:3032] T 563daf42d7784db195ea99c3cb21b370 P 61e69695258f48baa984c9d395e8ce3f [term 2 LEADER]: Stepping down as leader of term 2
I20260809 15:46:38.419804 26303 raft_consensus.cc:729] T 563daf42d7784db195ea99c3cb21b370 P 61e69695258f48baa984c9d395e8ce3f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 61e69695258f48baa984c9d395e8ce3f, State: Running, Role: LEADER
I20260809 15:46:38.420657 26303 consensus_queue.cc:260] T 563daf42d7784db195ea99c3cb21b370 P 61e69695258f48baa984c9d395e8ce3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:38.422009 26303 raft_consensus.cc:3037] T 563daf42d7784db195ea99c3cb21b370 P 61e69695258f48baa984c9d395e8ce3f [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:38.423456 25639 raft_consensus.cc:482] T 00a1881267a14ba8b78c81040f17a014 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:38.424069 25639 raft_consensus.cc:504] T 00a1881267a14ba8b78c81040f17a014 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:38.426030 25639 leader_election.cc:288] T 00a1881267a14ba8b78c81040f17a014 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:38.427374 26158 leader_election.cc:288] T 953eea268e17491ca76f3a555860a512 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 4 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:38.428858 24406 leader_election.cc:302] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f; no voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:38.414913 23353 leader_election.cc:302] T 7baa2492df0340c4a5ac241959ce1ac3 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:46:38.338246 25793 raft_consensus.cc:504] T f7715d71dbac447b9351fc5eb36c5e80 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:38.390609 26326 raft_consensus.cc:729] T 907bacfa79d04a55a68c8dbe60056462 P 61e69695258f48baa984c9d395e8ce3f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 61e69695258f48baa984c9d395e8ce3f, State: Running, Role: LEADER
I20260809 15:46:38.447685 26210 raft_consensus.cc:3037] T cee02fff2c9c48a19f43eb358a89e7b0 P 61e69695258f48baa984c9d395e8ce3f [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:38.446252 23526 raft_consensus.cc:2454] T 7660dfa2ef694927b8eda4100a2bcd9b P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 5.
I20260809 15:46:38.403643 26222 raft_consensus.cc:482] T d2d4f749fcfd4fb3bfb8115af3471598 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:38.406455 26284 raft_consensus.cc:482] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:38.439429 26040 leader_election.cc:288] T ba56725e42f74a1baa4c04d9660727f9 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:38.485719 25899 raft_consensus.cc:504] T 00a1881267a14ba8b78c81040f17a014 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:38.491137 25899 leader_election.cc:288] T 00a1881267a14ba8b78c81040f17a014 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:38.494593 23353 leader_election.cc:302] T 7660dfa2ef694927b8eda4100a2bcd9b P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:46:38.442669 26285 raft_consensus.cc:2781] T 7baa2492df0340c4a5ac241959ce1ac3 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Leader election won for term 2
I20260809 15:46:38.477974 26158 raft_consensus.cc:482] T 07fcabb569414bddbd39df6cedd24e64 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:38.501098 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00a1881267a14ba8b78c81040f17a014" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:38.501777 23526 raft_consensus.cc:2454] T 00a1881267a14ba8b78c81040f17a014 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 4.
I20260809 15:46:38.504814 23353 leader_election.cc:302] T 00a1881267a14ba8b78c81040f17a014 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:46:38.494223 26278 raft_consensus.cc:482] T d0721a4403a440f383318a3cea9d1442 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:38.478295 26333 consensus_queue.cc:1046] T 907bacfa79d04a55a68c8dbe60056462 P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.038s
I20260809 15:46:38.512118 23517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a88421dba80d4d3388aa5eb62e23853b" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:38.491978 26161 consensus_queue.cc:1046] T 763e58d6ba644eea830a6c1664d39c9b P 65c36d164b0540f099d592f38307ee6c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.019s
I20260809 15:46:38.522825 23283 raft_consensus.cc:1226] T 99bdb93503db4de4a1cfb0161ea0c07c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 LEADER]: Rejecting Update request from peer c0f38c1908414566aa9115bb2cbdd85d for earlier term 1. Current term is 2. Ops: []
W20260809 15:46:38.505082 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 163807us
I20260809 15:46:38.487717 26222 raft_consensus.cc:504] T d2d4f749fcfd4fb3bfb8115af3471598 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:38.542707 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472816994d814cb198b69e5248697440" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:38.554894 23277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00a1881267a14ba8b78c81040f17a014" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:38.505997 26284 raft_consensus.cc:504] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:38.557436 26284 leader_election.cc:288] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:38.506047 26053 raft_consensus.cc:2781] T 7660dfa2ef694927b8eda4100a2bcd9b P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Leader election won for term 5
I20260809 15:46:38.512429 23520 raft_consensus.cc:2379] T f7715d71dbac447b9351fc5eb36c5e80 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d in current term 4: Already voted for candidate 65c36d164b0540f099d592f38307ee6c in this term.
I20260809 15:46:38.555672 23277 raft_consensus.cc:2454] T 00a1881267a14ba8b78c81040f17a014 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 3.
I20260809 15:46:38.559341 26286 raft_consensus.cc:3037] T e14d118b2eaa42aa8a7cd4025aba7128 P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Advancing to term 5
I20260809 15:46:38.559594 23397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7715d71dbac447b9351fc5eb36c5e80" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:38.559908 23277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "953eea268e17491ca76f3a555860a512" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:38.560766 23397 raft_consensus.cc:2454] T f7715d71dbac447b9351fc5eb36c5e80 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 3.
I20260809 15:46:38.560834 25650 leader_election.cc:288] T df20671a7b874dbb98c0ef28371c1308 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:38.560491 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2424fd9bd7f14e508fd106cbfda786a4" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:38.560535 23277 raft_consensus.cc:3037] T 953eea268e17491ca76f3a555860a512 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:46:38.563321 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a823d35f4f348d1b52ba9e6d80b00d2" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:38.564172 23527 raft_consensus.cc:2454] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 4.
I20260809 15:46:38.500730 26144 raft_consensus.cc:504] T 6ced718c1a5c4da79cb8c1ebd1435b87 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:38.500564 26194 raft_consensus.cc:504] T 28e17dcb5efe45b8b6a8c42da03b4819 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:38.529143 25899 raft_consensus.cc:2781] T 00a1881267a14ba8b78c81040f17a014 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260809 15:46:38.505429 25959 leader_election.cc:288] T a88421dba80d4d3388aa5eb62e23853b P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:38.512254 26096 raft_consensus.cc:482] T 2db057fd10e149a9895dff6ef8c9f038 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:38.558085 26258 raft_consensus.cc:3032] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P c0f38c1908414566aa9115bb2cbdd85d [term 1 LEADER]: Stepping down as leader of term 1
I20260809 15:46:38.578497 23470 leader_election.cc:302] T 00a1881267a14ba8b78c81040f17a014 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:46:38.559934 25746 raft_consensus.cc:3032] T 294d4bcc1e7a477cb56455581277a60f P 61e69695258f48baa984c9d395e8ce3f [term 4 LEADER]: Stepping down as leader of term 4
I20260809 15:46:38.585832 23517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "213543cdbe814c89a17a963469922c1b" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:38.586174 23353 leader_election.cc:302] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
W20260809 15:46:38.647792 23470 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 73385.5us
I20260809 15:46:38.506093 26285 raft_consensus.cc:686] T 7baa2492df0340c4a5ac241959ce1ac3 P c0f38c1908414566aa9115bb2cbdd85d [term 2 LEADER]: Becoming Leader. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:46:38.571739 26053 raft_consensus.cc:686] T 7660dfa2ef694927b8eda4100a2bcd9b P c0f38c1908414566aa9115bb2cbdd85d [term 5 LEADER]: Becoming Leader. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:46:38.585990 26333 raft_consensus.cc:482] T d3be1e356b254d22b995be32f1c12f6f P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:38.586090 26279 consensus_queue.cc:1035] T 3296ef50be254fc29cfc77d22a8f7a23 P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.009s
I20260809 15:46:38.595134 23276 raft_consensus.cc:1261] T 213543cdbe814c89a17a963469922c1b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260809 15:46:38.597419 25941 raft_consensus.cc:504] T 2424fd9bd7f14e508fd106cbfda786a4 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:38.601532 26222 leader_election.cc:288] T d2d4f749fcfd4fb3bfb8115af3471598 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:38.578122 26339 raft_consensus.cc:2726] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 15:46:38.605904 23270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df20671a7b874dbb98c0ef28371c1308" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:38.611677 26214 raft_consensus.cc:3032] T 763e58d6ba644eea830a6c1664d39c9b P 65c36d164b0540f099d592f38307ee6c [term 2 LEADER]: Stepping down as leader of term 2
I20260809 15:46:38.612594 26194 leader_election.cc:288] T 28e17dcb5efe45b8b6a8c42da03b4819 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:38.597357 23285 raft_consensus.cc:1261] T d1a134ba33ea4076bbaac0ae460c17ba P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 15:46:38.617795 23283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e17dcb5efe45b8b6a8c42da03b4819" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:38.669661 23283 raft_consensus.cc:2379] T 28e17dcb5efe45b8b6a8c42da03b4819 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c in current term 4: Already voted for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in this term.
I20260809 15:46:38.619592 26326 consensus_queue.cc:260] T 907bacfa79d04a55a68c8dbe60056462 P 61e69695258f48baa984c9d395e8ce3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:38.647248 25746 raft_consensus.cc:729] T 294d4bcc1e7a477cb56455581277a60f P 61e69695258f48baa984c9d395e8ce3f [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 61e69695258f48baa984c9d395e8ce3f, State: Running, Role: LEADER
I20260809 15:46:38.544114 26158 raft_consensus.cc:504] T 07fcabb569414bddbd39df6cedd24e64 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:46:38.668897 26139 consensus_meta.cc:287] T c0741ab44d99439abc3aa3bdd27ee160 P 65c36d164b0540f099d592f38307ee6c: Time spent flushing consensus metadata: real 0.567s user 0.009s sys 0.000s
I20260809 15:46:38.675292 23228 leader_election.cc:302] T a3622f6e3e3641cea1960e3a31498bb7 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:38.669616 25959 raft_consensus.cc:2781] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260809 15:46:38.668141 26214 raft_consensus.cc:729] T 763e58d6ba644eea830a6c1664d39c9b P 65c36d164b0540f099d592f38307ee6c [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:46:38.651913 23351 leader_election.cc:302] T 07fcabb569414bddbd39df6cedd24e64 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:38.652909 26333 raft_consensus.cc:504] T d3be1e356b254d22b995be32f1c12f6f P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:38.647042 26258 raft_consensus.cc:729] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P c0f38c1908414566aa9115bb2cbdd85d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:46:38.582666 25899 raft_consensus.cc:482] T 00a1881267a14ba8b78c81040f17a014 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260809 15:46:38.662655 23353 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 76556.6us
I20260809 15:46:38.666592 23270 raft_consensus.cc:2454] T df20671a7b874dbb98c0ef28371c1308 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 3.
I20260809 15:46:38.624717 26136 raft_consensus.cc:3032] T 99bdb93503db4de4a1cfb0161ea0c07c P c0f38c1908414566aa9115bb2cbdd85d [term 1 LEADER]: Stepping down as leader of term 1
I20260809 15:46:38.653891 25639 raft_consensus.cc:2781] T 00a1881267a14ba8b78c81040f17a014 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260809 15:46:38.676014 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2424fd9bd7f14e508fd106cbfda786a4" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 5 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:38.708949 23397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ced718c1a5c4da79cb8c1ebd1435b87" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:38.676617 23526 raft_consensus.cc:2379] T 2424fd9bd7f14e508fd106cbfda786a4 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 in current term 5: Already voted for candidate 65c36d164b0540f099d592f38307ee6c in this term.
I20260809 15:46:38.690336 23517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "538a228ddfb54bc6a36528d5ff07597b" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:38.514652 26278 raft_consensus.cc:504] T d0721a4403a440f383318a3cea9d1442 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:46:38.724793 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 49680.6us
I20260809 15:46:38.643350 23277 raft_consensus.cc:2454] T 953eea268e17491ca76f3a555860a512 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 4.
I20260809 15:46:38.677757 26275 raft_consensus.cc:482] T cee02fff2c9c48a19f43eb358a89e7b0 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:38.698083 23280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7715d71dbac447b9351fc5eb36c5e80" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:38.676959 23520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00a1881267a14ba8b78c81040f17a014" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:38.709038 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05ad7d8fdca64e3da527280ca5be9280" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:38.691109 25959 raft_consensus.cc:482] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:46:38.725435 23397 raft_consensus.cc:2379] T 6ced718c1a5c4da79cb8c1ebd1435b87 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c in current term 3: Already voted for candidate c0f38c1908414566aa9115bb2cbdd85d in this term.
I20260809 15:46:38.690811 25793 leader_election.cc:288] T f7715d71dbac447b9351fc5eb36c5e80 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 4 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:38.693485 26177 raft_consensus.cc:2726] T a3622f6e3e3641cea1960e3a31498bb7 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260809 15:46:38.727208 23280 raft_consensus.cc:3037] T f7715d71dbac447b9351fc5eb36c5e80 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:46:38.737296 23519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df20671a7b874dbb98c0ef28371c1308" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:38.739660 23519 raft_consensus.cc:2454] T df20671a7b874dbb98c0ef28371c1308 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 3.
I20260809 15:46:38.740541 23470 leader_election.cc:302] T df20671a7b874dbb98c0ef28371c1308 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:46:38.741745 23402 raft_consensus.cc:2454] T 2424fd9bd7f14e508fd106cbfda786a4 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 4.
I20260809 15:46:38.722842 25639 raft_consensus.cc:482] T 00a1881267a14ba8b78c81040f17a014 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:46:38.722543 26136 raft_consensus.cc:729] T 99bdb93503db4de4a1cfb0161ea0c07c P c0f38c1908414566aa9115bb2cbdd85d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
W20260809 15:46:38.748641 24406 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 319866us
I20260809 15:46:38.690892 26139 raft_consensus.cc:504] T c0741ab44d99439abc3aa3bdd27ee160 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:38.729501 25959 raft_consensus.cc:3037] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Advancing to term 5
I20260809 15:46:38.726473 25650 raft_consensus.cc:482] T c5c2e425a74b4e64ae70b5def750434c P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
I20260809 15:46:38.751183 23470 leader_election.cc:302] T 953eea268e17491ca76f3a555860a512 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:46:38.704592 26228 consensus_queue.cc:1035] T d1a134ba33ea4076bbaac0ae460c17ba P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.040s
I20260809 15:46:38.705947 26096 raft_consensus.cc:504] T 2db057fd10e149a9895dff6ef8c9f038 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:46:38.743113 24410 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.540s user 0.021s sys 0.072s
W20260809 15:46:38.755189 24410 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.552s user 0.021s sys 0.072s
W20260809 15:46:38.759845 24410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 556614us
I20260809 15:46:38.786487 23519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1a134ba33ea4076bbaac0ae460c17ba" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:38.741333 26284 raft_consensus.cc:2726] T 07fcabb569414bddbd39df6cedd24e64 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260809 15:46:38.746333 26148 raft_consensus.cc:482] T d3be1e356b254d22b995be32f1c12f6f P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:38.735642 25863 consensus_queue.cc:1035] T 213543cdbe814c89a17a963469922c1b P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.020s
I20260809 15:46:38.749166 25639 raft_consensus.cc:3037] T 00a1881267a14ba8b78c81040f17a014 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Advancing to term 5
I20260809 15:46:38.712393 25899 raft_consensus.cc:3037] T 00a1881267a14ba8b78c81040f17a014 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Advancing to term 5
I20260809 15:46:38.797071 23270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2424fd9bd7f14e508fd106cbfda786a4" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 5 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:38.753860 26231 raft_consensus.cc:2781] T df20671a7b874dbb98c0ef28371c1308 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260809 15:46:38.797688 23270 raft_consensus.cc:3037] T 2424fd9bd7f14e508fd106cbfda786a4 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Advancing to term 5
W20260809 15:46:38.797660 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 213461us
I20260809 15:46:38.790722 25967 raft_consensus.cc:2781] T 953eea268e17491ca76f3a555860a512 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Leader election won for term 4
I20260809 15:46:38.797216 23396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "953eea268e17491ca76f3a555860a512" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:38.790598 26185 raft_consensus.cc:482] T e71a05ba1d4d42bd95c3f85f9313399e P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:38.806066 23519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4564dfc6d4094c1b9240b367ccecf3b1" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:38.809648 23351 leader_election.cc:302] T 3296ef50be254fc29cfc77d22a8f7a23 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters: 61e69695258f48baa984c9d395e8ce3f
I20260809 15:46:38.792992 26144 leader_election.cc:288] T 6ced718c1a5c4da79cb8c1ebd1435b87 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:38.792091 23395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad8a3b25d7b49e5a2142d5709edd0c7" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
W20260809 15:46:38.817039 26342 log.cc:924] Time spent T 6ae14a74a26f434db08c652e097385c8 P 67b3aeadfa7a432abe8af4d292e04ab9: Append to log took a long time: real 0.065s user 0.000s sys 0.000s
I20260809 15:46:38.735283 26053 consensus_queue.cc:237] T 7660dfa2ef694927b8eda4100a2bcd9b P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
W20260809 15:46:38.827904 24406 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 44352.2us
I20260809 15:46:38.800432 26231 raft_consensus.cc:482] T df20671a7b874dbb98c0ef28371c1308 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:46:38.799054 26118 raft_consensus.cc:2726] T 763e58d6ba644eea830a6c1664d39c9b P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 15:46:38.828667 23277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07fcabb569414bddbd39df6cedd24e64" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:38.783604 25650 raft_consensus.cc:504] T c5c2e425a74b4e64ae70b5def750434c P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:38.830476 23277 raft_consensus.cc:2454] T 07fcabb569414bddbd39df6cedd24e64 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 4.
I20260809 15:46:38.798667 26346 raft_consensus.cc:482] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:46:38.748253 26285 consensus_queue.cc:237] T 7baa2492df0340c4a5ac241959ce1ac3 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:38.808296 25941 leader_election.cc:288] T 2424fd9bd7f14e508fd106cbfda786a4 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 5 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:38.838198 23228 leader_election.cc:302] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:38.812230 23276 raft_consensus.cc:1261] T 294d4bcc1e7a477cb56455581277a60f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260809 15:46:38.845151 23396 raft_consensus.cc:2454] T 953eea268e17491ca76f3a555860a512 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 3.
W20260809 15:46:38.828984 23223 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.68:34689, user_credentials={real_user=slave}} blocked reactor thread for 266204us
I20260809 15:46:38.798492 26177 consensus_queue.cc:1046] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.029s
I20260809 15:46:38.831959 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2d4f749fcfd4fb3bfb8115af3471598" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:38.792322 26275 raft_consensus.cc:504] T cee02fff2c9c48a19f43eb358a89e7b0 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:38.828032 25967 raft_consensus.cc:686] T 953eea268e17491ca76f3a555860a512 P 65c36d164b0540f099d592f38307ee6c [term 4 LEADER]: Becoming Leader. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:46:38.830535 26231 raft_consensus.cc:3037] T df20671a7b874dbb98c0ef28371c1308 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:46:38.824780 23392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3e94b7e50754a81b3d72a07476cb7b9" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:38.851998 23470 leader_election.cc:302] T 07fcabb569414bddbd39df6cedd24e64 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
W20260809 15:46:38.862298 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 58520us
I20260809 15:46:38.813021 26148 raft_consensus.cc:504] T d3be1e356b254d22b995be32f1c12f6f P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:38.864948 23519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "953eea268e17491ca76f3a555860a512" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:38.792500 25746 consensus_queue.cc:260] T 294d4bcc1e7a477cb56455581277a60f P 61e69695258f48baa984c9d395e8ce3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:38.874289 23396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3be1e356b254d22b995be32f1c12f6f" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:38.875592 23285 raft_consensus.cc:1261] T cee02fff2c9c48a19f43eb358a89e7b0 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260809 15:46:38.876907 23351 leader_election.cc:302] T 953eea268e17491ca76f3a555860a512 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c
I20260809 15:46:38.880517 23527 raft_consensus.cc:1226] T 0da4d5283c8a46878bd39b85e5c71f66 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Rejecting Update request from peer 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260809 15:46:38.825721 26185 raft_consensus.cc:504] T e71a05ba1d4d42bd95c3f85f9313399e P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:38.884527 23396 raft_consensus.cc:2454] T d3be1e356b254d22b995be32f1c12f6f P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 3.
I20260809 15:46:38.885571 23399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a823d35f4f348d1b52ba9e6d80b00d2" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
W20260809 15:46:38.894578 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 102846us
I20260809 15:46:38.849421 26346 raft_consensus.cc:504] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:38.891121 23270 raft_consensus.cc:2454] T 2424fd9bd7f14e508fd106cbfda786a4 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 5.
I20260809 15:46:38.794625 26258 consensus_queue.cc:260] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P c0f38c1908414566aa9115bb2cbdd85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:38.804946 26214 consensus_queue.cc:260] T 763e58d6ba644eea830a6c1664d39c9b P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:38.853180 26158 leader_election.cc:288] T 07fcabb569414bddbd39df6cedd24e64 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
W20260809 15:46:38.907603 23470 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 55694.7us
I20260809 15:46:38.862612 26333 leader_election.cc:288] T d3be1e356b254d22b995be32f1c12f6f P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:38.928746 26096 leader_election.cc:288] T 2db057fd10e149a9895dff6ef8c9f038 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:38.930581 23519 raft_consensus.cc:1226] T 7baa2492df0340c4a5ac241959ce1ac3 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Rejecting Update request from peer 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 1. Current term is 2. Ops: []
W20260809 15:46:38.933305 24406 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 48422.9us
I20260809 15:46:38.935442 23280 raft_consensus.cc:2454] T f7715d71dbac447b9351fc5eb36c5e80 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 4.
W20260809 15:46:38.939657 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 75409.6us
I20260809 15:46:38.920017 25956 raft_consensus.cc:2781] T 953eea268e17491ca76f3a555860a512 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader election won for term 3
I20260809 15:46:38.902067 26194 raft_consensus.cc:2726] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260809 15:46:38.938282 23274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2db057fd10e149a9895dff6ef8c9f038" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
W20260809 15:46:38.943812 23468 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.68:34689, user_credentials={real_user=slave}} blocked reactor thread for 243080us
I20260809 15:46:38.920527 26278 leader_election.cc:288] T d0721a4403a440f383318a3cea9d1442 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:38.926884 26228 raft_consensus.cc:2781] T 07fcabb569414bddbd39df6cedd24e64 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260809 15:46:38.942803 23274 raft_consensus.cc:2454] T 2db057fd10e149a9895dff6ef8c9f038 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 3.
I20260809 15:46:38.902510 23392 raft_consensus.cc:2454] T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 5.
I20260809 15:46:38.949575 23228 leader_election.cc:302] T 563daf42d7784db195ea99c3cb21b370 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:38.926358 25931 raft_consensus.cc:482] T df20671a7b874dbb98c0ef28371c1308 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:38.847548 26136 consensus_queue.cc:260] T 99bdb93503db4de4a1cfb0161ea0c07c P c0f38c1908414566aa9115bb2cbdd85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:38.955956 23470 leader_election.cc:302] T 2424fd9bd7f14e508fd106cbfda786a4 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:46:38.961903 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ced718c1a5c4da79cb8c1ebd1435b87" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:38.964206 23526 raft_consensus.cc:3037] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:38.962502 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2d4f749fcfd4fb3bfb8115af3471598" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:38.941584 25746 raft_consensus.cc:3037] T 294d4bcc1e7a477cb56455581277a60f P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Advancing to term 5
I20260809 15:46:38.901320 25959 raft_consensus.cc:504] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:38.896098 26335 consensus_queue.cc:1035] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.074s
I20260809 15:46:38.968521 23470 leader_election.cc:302] T f7715d71dbac447b9351fc5eb36c5e80 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:46:38.967360 23517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0721a4403a440f383318a3cea9d1442" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:38.950009 26258 raft_consensus.cc:3037] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:38.952702 26214 raft_consensus.cc:3037] T 763e58d6ba644eea830a6c1664d39c9b P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:38.950979 26228 raft_consensus.cc:482] T 07fcabb569414bddbd39df6cedd24e64 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:46:38.980840 23470 leader_election.cc:302] T 2db057fd10e149a9895dff6ef8c9f038 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:46:38.976415 24410 leader_election.cc:302] T d3e94b7e50754a81b3d72a07476cb7b9 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:46:38.970613 23517 raft_consensus.cc:2454] T d0721a4403a440f383318a3cea9d1442 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 4.
W20260809 15:46:38.977346 25816 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.519s user 0.025s sys 0.012s
I20260809 15:46:38.977524 23521 raft_consensus.cc:1261] T 0da4d5283c8a46878bd39b85e5c71f66 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20260809 15:46:38.936267 26158 consensus_queue.cc:1035] T 294d4bcc1e7a477cb56455581277a60f P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.031s
I20260809 15:46:38.952876 25956 raft_consensus.cc:686] T 953eea268e17491ca76f3a555860a512 P c0f38c1908414566aa9115bb2cbdd85d [term 3 LEADER]: Becoming Leader. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:46:38.956967 26139 leader_election.cc:288] T c0741ab44d99439abc3aa3bdd27ee160 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 4 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:38.907424 25967 consensus_queue.cc:237] T 953eea268e17491ca76f3a555860a512 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:38.954023 25863 raft_consensus.cc:482] T 4bcac4b24e7a4448abdc934bba5a624e P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
I20260809 15:46:38.983295 25999 raft_consensus.cc:2781] T f7715d71dbac447b9351fc5eb36c5e80 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Leader election won for term 4
I20260809 15:46:38.974009 25931 raft_consensus.cc:504] T df20671a7b874dbb98c0ef28371c1308 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:39.055374 23353 leader_election.cc:302] T d0721a4403a440f383318a3cea9d1442 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:46:39.056416 26346 leader_election.cc:288] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
W20260809 15:46:39.057844 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 126375us
I20260809 15:46:38.972190 23276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0741ab44d99439abc3aa3bdd27ee160" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:38.970927 25941 raft_consensus.cc:482] T d3e94b7e50754a81b3d72a07476cb7b9 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
I20260809 15:46:38.977747 26194 raft_consensus.cc:2726] T 563daf42d7784db195ea99c3cb21b370 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260809 15:46:39.092693 23529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0741ab44d99439abc3aa3bdd27ee160" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:39.093668 23529 raft_consensus.cc:2379] T c0741ab44d99439abc3aa3bdd27ee160 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 61e69695258f48baa984c9d395e8ce3f in current term 4: Already voted for candidate 65c36d164b0540f099d592f38307ee6c in this term.
I20260809 15:46:39.095777 25959 leader_election.cc:288] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 election: Requested vote from peers 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
W20260809 15:46:38.983789 25816 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.601s user 0.025s sys 0.012s
I20260809 15:46:39.063406 23274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5c2e425a74b4e64ae70b5def750434c" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:38.983456 26228 raft_consensus.cc:3037] T 07fcabb569414bddbd39df6cedd24e64 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Advancing to term 5
I20260809 15:46:39.072201 23517 raft_consensus.cc:1226] T 7660dfa2ef694927b8eda4100a2bcd9b P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Rejecting Update request from peer 61e69695258f48baa984c9d395e8ce3f for earlier term 4. Current term is 5. Ops: []
I20260809 15:46:39.072335 23521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3e94b7e50754a81b3d72a07476cb7b9" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:39.112286 23522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3aa6a9a52804df0ba9b0ca3c82f38c3" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
W20260809 15:46:39.112998 26358 log.cc:924] Time spent T ae6d2c08a7a94f58ac8dd698e3fd456c P c0f38c1908414566aa9115bb2cbdd85d: Append to log took a long time: real 0.135s user 0.000s sys 0.001s
I20260809 15:46:39.113695 25956 consensus_queue.cc:237] T 953eea268e17491ca76f3a555860a512 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:39.011471 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3aa6a9a52804df0ba9b0ca3c82f38c3" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f"
I20260809 15:46:39.057060 26333 raft_consensus.cc:2781] T d3e94b7e50754a81b3d72a07476cb7b9 P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260809 15:46:39.078188 23399 raft_consensus.cc:2379] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c in current term 5: Already voted for candidate c0f38c1908414566aa9115bb2cbdd85d in this term.
I20260809 15:46:39.078328 23519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5c2e425a74b4e64ae70b5def750434c" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:39.117328 23519 raft_consensus.cc:2454] T c5c2e425a74b4e64ae70b5def750434c P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 2.
W20260809 15:46:39.079174 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 138927us
W20260809 15:46:39.080309 23353 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 59474us
I20260809 15:46:39.015573 23392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e17dcb5efe45b8b6a8c42da03b4819" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:39.007344 24453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0741ab44d99439abc3aa3bdd27ee160" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:39.061264 25793 raft_consensus.cc:2781] T 2db057fd10e149a9895dff6ef8c9f038 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260809 15:46:39.084426 26143 raft_consensus.cc:2781] T 2424fd9bd7f14e508fd106cbfda786a4 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Leader election won for term 5
I20260809 15:46:39.085036 23528 raft_consensus.cc:1261] T 0c48a3049897452383257aa757834fb5 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Refusing update from remote peer 61e69695258f48baa984c9d395e8ce3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260809 15:46:39.125983 24410 leader_election.cc:302] T d3be1e356b254d22b995be32f1c12f6f P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d; no voters:
W20260809 15:46:39.087248 26345 log.cc:924] Time spent T 0da4d5283c8a46878bd39b85e5c71f66 P c0f38c1908414566aa9115bb2cbdd85d: Append to log took a long time: real 0.118s user 0.000s sys 0.000s
I20260809 15:46:39.090557 26040 raft_consensus.cc:482] T d3be1e356b254d22b995be32f1c12f6f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:39.091586 26148 leader_election.cc:288] T d3be1e356b254d22b995be32f1c12f6f P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:39.091367 25941 raft_consensus.cc:504] T d3e94b7e50754a81b3d72a07476cb7b9 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:39.092098 26284 raft_consensus.cc:2781] T d0721a4403a440f383318a3cea9d1442 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260809 15:46:39.097183 23274 raft_consensus.cc:2454] T c5c2e425a74b4e64ae70b5def750434c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 3.
I20260809 15:46:39.100132 23228 leader_election.cc:302] T cef3e7277d7d4760824a3436ed4a847d P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:39.099812 25999 raft_consensus.cc:686] T f7715d71dbac447b9351fc5eb36c5e80 P 65c36d164b0540f099d592f38307ee6c [term 4 LEADER]: Becoming Leader. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:46:39.102722 24448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "563daf42d7784db195ea99c3cb21b370" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:39.103430 26326 raft_consensus.cc:3037] T 907bacfa79d04a55a68c8dbe60056462 P 61e69695258f48baa984c9d395e8ce3f [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:39.104226 26185 leader_election.cc:288] T e71a05ba1d4d42bd95c3f85f9313399e P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:39.105221 23393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7715d71dbac447b9351fc5eb36c5e80" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:39.107792 24452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7660dfa2ef694927b8eda4100a2bcd9b" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f"
I20260809 15:46:39.113096 23522 raft_consensus.cc:2379] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 61e69695258f48baa984c9d395e8ce3f in current term 2: Already voted for candidate 65c36d164b0540f099d592f38307ee6c in this term.
I20260809 15:46:39.098177 23276 raft_consensus.cc:3037] T c0741ab44d99439abc3aa3bdd27ee160 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:46:39.115626 25885 raft_consensus.cc:3032] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 LEADER]: Stepping down as leader of term 2
I20260809 15:46:39.067484 25650 leader_election.cc:288] T c5c2e425a74b4e64ae70b5def750434c P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:39.073321 23526 raft_consensus.cc:2454] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 3.
I20260809 15:46:39.145833 26284 raft_consensus.cc:482] T d0721a4403a440f383318a3cea9d1442 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:46:39.086354 26275 leader_election.cc:288] T cee02fff2c9c48a19f43eb358a89e7b0 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:39.128772 23400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00a1881267a14ba8b78c81040f17a014" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:39.132099 26335 raft_consensus.cc:2781] T d3be1e356b254d22b995be32f1c12f6f P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260809 15:46:39.134253 23519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a823d35f4f348d1b52ba9e6d80b00d2" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:39.119061 26333 raft_consensus.cc:482] T d3e94b7e50754a81b3d72a07476cb7b9 P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:46:39.142628 23396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3aa6a9a52804df0ba9b0ca3c82f38c3" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:39.208639 23396 raft_consensus.cc:2454] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 2.
I20260809 15:46:39.164271 25959 raft_consensus.cc:482] T cef3e7277d7d4760824a3436ed4a847d P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:39.130268 24453 raft_consensus.cc:2454] T c0741ab44d99439abc3aa3bdd27ee160 P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 3.
I20260809 15:46:39.099954 25863 raft_consensus.cc:504] T 4bcac4b24e7a4448abdc934bba5a624e P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:39.166330 25929 raft_consensus.cc:2726] T cef3e7277d7d4760824a3436ed4a847d P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260809 15:46:39.166870 23521 raft_consensus.cc:2454] T d3e94b7e50754a81b3d72a07476cb7b9 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 4.
I20260809 15:46:39.170027 24471 raft_consensus.cc:1261] T e14d118b2eaa42aa8a7cd4025aba7128 P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260809 15:46:39.172772 23399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df20671a7b874dbb98c0ef28371c1308" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:39.082613 26136 raft_consensus.cc:3037] T 99bdb93503db4de4a1cfb0161ea0c07c P c0f38c1908414566aa9115bb2cbdd85d [term 1 FOLLOWER]: Advancing to term 2
W20260809 15:46:39.173235 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 76257.4us
I20260809 15:46:39.170061 24472 raft_consensus.cc:1261] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260809 15:46:39.179353 23397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "953eea268e17491ca76f3a555860a512" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:39.180677 25885 raft_consensus.cc:729] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Running, Role: LEADER
I20260809 15:46:39.144291 26143 raft_consensus.cc:686] T 2424fd9bd7f14e508fd106cbfda786a4 P 65c36d164b0540f099d592f38307ee6c [term 5 LEADER]: Becoming Leader. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:46:39.157933 25980 raft_consensus.cc:482] T 3d880de657e544468c82157769622a26 P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
W20260809 15:46:39.206590 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 85485.6us
I20260809 15:46:39.163822 24448 raft_consensus.cc:2454] T 563daf42d7784db195ea99c3cb21b370 P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 3.
I20260809 15:46:39.206732 23519 raft_consensus.cc:3037] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Advancing to term 5
I20260809 15:46:39.121853 25793 raft_consensus.cc:482] T 2db057fd10e149a9895dff6ef8c9f038 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260809 15:46:39.174180 26360 log.cc:924] Time spent T 4b30f28311964a0b9149a50495e83d8f P 67b3aeadfa7a432abe8af4d292e04ab9: Append to log took a long time: real 0.109s user 0.000s sys 0.000s
I20260809 15:46:39.210896 24410 leader_election.cc:302] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:46:39.168466 24451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472816994d814cb198b69e5248697440" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:39.168830 24463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3be1e356b254d22b995be32f1c12f6f" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:39.206486 26335 raft_consensus.cc:482] T d3be1e356b254d22b995be32f1c12f6f P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:46:39.212337 26333 raft_consensus.cc:3037] T d3e94b7e50754a81b3d72a07476cb7b9 P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Advancing to term 6
I20260809 15:46:39.212110 23396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2424fd9bd7f14e508fd106cbfda786a4" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 5 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:39.213164 24452 raft_consensus.cc:1261] T cee02fff2c9c48a19f43eb358a89e7b0 P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
W20260809 15:46:39.214046 26340 log.cc:924] Time spent T 3296ef50be254fc29cfc77d22a8f7a23 P 61e69695258f48baa984c9d395e8ce3f: Append to log took a long time: real 0.525s user 0.000s sys 0.004s
I20260809 15:46:39.166366 23393 raft_consensus.cc:3037] T f7715d71dbac447b9351fc5eb36c5e80 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:46:39.245309 25863 leader_election.cc:288] T 4bcac4b24e7a4448abdc934bba5a624e P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:39.182904 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a823d35f4f348d1b52ba9e6d80b00d2" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:39.246609 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad8a3b25d7b49e5a2142d5709edd0c7" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:39.204979 26284 raft_consensus.cc:3037] T d0721a4403a440f383318a3cea9d1442 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Advancing to term 5
I20260809 15:46:39.247629 23392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2db057fd10e149a9895dff6ef8c9f038" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:39.248312 23392 raft_consensus.cc:2454] T 2db057fd10e149a9895dff6ef8c9f038 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 2.
I20260809 15:46:39.220986 23399 raft_consensus.cc:2454] T df20671a7b874dbb98c0ef28371c1308 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 3.
W20260809 15:46:39.222962 23470 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 156415us
I20260809 15:46:39.227118 23228 leader_election.cc:302] T 73f6f44c259547f1b05a3a434616fa54 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:39.163245 26040 raft_consensus.cc:504] T d3be1e356b254d22b995be32f1c12f6f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:39.209568 26118 raft_consensus.cc:482] T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:39.238924 23396 raft_consensus.cc:3037] T 2424fd9bd7f14e508fd106cbfda786a4 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Advancing to term 5
I20260809 15:46:39.240669 24451 raft_consensus.cc:2379] T 472816994d814cb198b69e5248697440 P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c in current term 4: Already voted for candidate 61e69695258f48baa984c9d395e8ce3f in this term.
W20260809 15:46:39.275440 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 48400.5us
I20260809 15:46:39.205909 26231 raft_consensus.cc:504] T df20671a7b874dbb98c0ef28371c1308 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:39.216233 26258 consensus_queue.cc:1035] T 0da4d5283c8a46878bd39b85e5c71f66 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.021s
I20260809 15:46:39.231559 25793 raft_consensus.cc:3037] T 2db057fd10e149a9895dff6ef8c9f038 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:46:39.276881 23404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07fcabb569414bddbd39df6cedd24e64" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:39.220955 25959 raft_consensus.cc:504] T cef3e7277d7d4760824a3436ed4a847d P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:39.238392 26335 raft_consensus.cc:3037] T d3be1e356b254d22b995be32f1c12f6f P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:46:39.291137 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a823d35f4f348d1b52ba9e6d80b00d2" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:39.278415 23470 leader_election.cc:302] T c5c2e425a74b4e64ae70b5def750434c P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:46:39.238536 26179 raft_consensus.cc:2781] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260809 15:46:39.308036 23404 raft_consensus.cc:2454] T 07fcabb569414bddbd39df6cedd24e64 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 3.
I20260809 15:46:39.231757 25980 raft_consensus.cc:504] T 3d880de657e544468c82157769622a26 P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:39.281275 25931 leader_election.cc:288] T df20671a7b874dbb98c0ef28371c1308 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:39.329104 23399 raft_consensus.cc:1261] T d1a134ba33ea4076bbaac0ae460c17ba P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260809 15:46:39.327483 23470 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 49217.2us
I20260809 15:46:39.329493 23228 leader_election.cc:302] T 998c6cee72cf40228c6dba064c0db2da P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:39.351763 23351 leader_election.cc:302] T 998c6cee72cf40228c6dba064c0db2da P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:39.235997 25999 consensus_queue.cc:237] T f7715d71dbac447b9351fc5eb36c5e80 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:39.290774 26346 consensus_queue.cc:1046] T 7660dfa2ef694927b8eda4100a2bcd9b P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 4, Time since last communication: 0.061s
I20260809 15:46:39.292517 25929 raft_consensus.cc:2726] T 73f6f44c259547f1b05a3a434616fa54 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260809 15:46:39.360623 23228 leader_election.cc:302] T a88421dba80d4d3388aa5eb62e23853b P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:39.368024 23399 raft_consensus.cc:1226] T 763e58d6ba644eea830a6c1664d39c9b P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 2. Current term is 3. Ops: []
I20260809 15:46:39.294965 23397 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "00a1881267a14ba8b78c81040f17a014"
dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
mode: GRACEFUL
new_leader_uuid: "65c36d164b0540f099d592f38307ee6c"
from {username='slave'} at 127.0.0.1:51280
I20260809 15:46:39.375886 26381 consensus_queue.cc:1035] T 4564dfc6d4094c1b9240b367ccecf3b1 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.137s
I20260809 15:46:39.378026 26286 consensus_queue.cc:1035] T 0c48a3049897452383257aa757834fb5 P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260809 15:46:39.379887 26231 leader_election.cc:288] T df20671a7b874dbb98c0ef28371c1308 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 4 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:39.381212 26040 leader_election.cc:288] T d3be1e356b254d22b995be32f1c12f6f P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:39.308787 26179 raft_consensus.cc:482] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Starting leader election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
W20260809 15:46:39.386404 25899 consensus_meta.cc:287] T 00a1881267a14ba8b78c81040f17a014 P c0f38c1908414566aa9115bb2cbdd85d: Time spent flushing consensus metadata: real 0.552s user 0.008s sys 0.000s
W20260809 15:46:39.408267 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 47709.4us
I20260809 15:46:39.411633 23273 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53554 (request call id 1226) took 1418 ms. Trace:
I20260809 15:46:39.414628 23228 leader_election.cc:302] T 0c29937a80724ddb9d07a27534a146f3 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
W20260809 15:46:39.416632 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 123730us
I20260809 15:46:39.293274 26118 raft_consensus.cc:504] T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:39.411998 23273 rpcz_store.cc:276] 0809 15:46:37.980468 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:38.052033 (+ 71565us) service_pool.cc:227] Handling call
0809 15:46:38.073736 (+ 21703us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 15:46:38.074021 (+ 285us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:46:38.074043 (+ 22us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 15:46:38.618603 (+544560us) spinlock_profiling.cc:243] Waited 25.4 ms on lock 0x7b2c0000f228. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e7c03e11 00007f34e7c13442 00007f34e925bc11 00007f34e922058a 00007f34e921f99e 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772
0809 15:46:38.618957 (+ 354us) log.cc:842] Serialized 26 byte log entry
0809 15:46:38.843995 (+225038us) raft_consensus.cc:1631] Marking committed up to 1
0809 15:46:38.844028 (+ 33us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:38.844094 (+ 66us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 15:46:39.173473 (+329379us) spinlock_profiling.cc:243] Waited 1.93 ms on lock 0x7b44000232cc. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e613cf11 00007f34e620aa95 00007f34e620b4b2 00007f34e8eb3511 00007f34e8eaee1a 00007f34e8ea2bee 00007f34e8ea1f36 00007f34e8ea1b58 00007f34e8eab551 00007f34e8eab509 00007f34e8eab499
0809 15:46:39.398610 (+225137us) raft_consensus.cc:1671] finished
0809 15:46:39.398633 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:39.398762 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":62795520,"thread_start_us":224690,"threads_started":1,"wal-append.queue_time_us":361140}
I20260809 15:46:39.395998 26275 raft_consensus.cc:2726] T 998c6cee72cf40228c6dba064c0db2da P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260809 15:46:39.393419 25929 raft_consensus.cc:2726] T a88421dba80d4d3388aa5eb62e23853b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260809 15:46:39.412042 26363 log.cc:924] Time spent T a88421dba80d4d3388aa5eb62e23853b P c0f38c1908414566aa9115bb2cbdd85d: Append to log took a long time: real 0.201s user 0.001s sys 0.000s
I20260809 15:46:39.429216 23351 leader_election.cc:302] T e71a05ba1d4d42bd95c3f85f9313399e P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 61e69695258f48baa984c9d395e8ce3f, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:39.426295 24451 raft_consensus.cc:1261] T 3924b648461e463b9d7925e5671d68ae P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260809 15:46:39.398248 26384 raft_consensus.cc:482] T 297c82ed906d4b48b184985eed3de601 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:39.399004 25941 leader_election.cc:288] T d3e94b7e50754a81b3d72a07476cb7b9 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:39.330441 25885 consensus_queue.cc:260] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 67b3aeadfa7a432abe8af4d292e04ab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:39.418902 26381 raft_consensus.cc:2781] T c5c2e425a74b4e64ae70b5def750434c P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260809 15:46:39.426440 25931 raft_consensus.cc:2726] T 998c6cee72cf40228c6dba064c0db2da P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260809 15:46:39.352515 26143 consensus_queue.cc:237] T 2424fd9bd7f14e508fd106cbfda786a4 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:39.397454 26179 raft_consensus.cc:3037] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:39.392952 26214 consensus_queue.cc:1035] T cee02fff2c9c48a19f43eb358a89e7b0 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.038s
I20260809 15:46:39.442955 23404 raft_consensus.cc:1261] T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
W20260809 15:46:39.449572 23472 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 283832us
W20260809 15:46:39.451361 23394 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.505s user 0.011s sys 0.034s
W20260809 15:46:39.451726 23394 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.627s user 0.011s sys 0.034s
I20260809 15:46:39.439823 26040 raft_consensus.cc:2726] T 0c29937a80724ddb9d07a27534a146f3 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260809 15:46:39.455552 23472 leader_election.cc:302] T 0da4d5283c8a46878bd39b85e5c71f66 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:39.440320 26381 raft_consensus.cc:482] T c5c2e425a74b4e64ae70b5def750434c P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Starting leader election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
W20260809 15:46:39.459124 23353 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 296747us
I20260809 15:46:39.441316 26148 raft_consensus.cc:3037] T e71a05ba1d4d42bd95c3f85f9313399e P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:39.413684 25899 raft_consensus.cc:504] T 00a1881267a14ba8b78c81040f17a014 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:39.444566 26393 raft_consensus.cc:482] T 73f6f44c259547f1b05a3a434616fa54 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:46:39.454248 25885 raft_consensus.cc:3037] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:39.482328 23353 leader_election.cc:302] T 6ced718c1a5c4da79cb8c1ebd1435b87 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:46:39.482553 23521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df20671a7b874dbb98c0ef28371c1308" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:39.487082 23521 raft_consensus.cc:2379] T df20671a7b874dbb98c0ef28371c1308 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d in current term 4: Already voted for candidate 65c36d164b0540f099d592f38307ee6c in this term.
W20260809 15:46:39.480679 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 51551.8us
I20260809 15:46:39.502166 23522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cef3e7277d7d4760824a3436ed4a847d" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:39.505070 23228 leader_election.cc:302] T d3be1e356b254d22b995be32f1c12f6f P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:46:39.510464 23228 leader_election.cc:302] T df20671a7b874dbb98c0ef28371c1308 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:46:39.517802 23400 raft_consensus.cc:2379] T 00a1881267a14ba8b78c81040f17a014 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c in current term 5: Already voted for candidate c0f38c1908414566aa9115bb2cbdd85d in this term.
I20260809 15:46:39.502780 23522 raft_consensus.cc:2454] T cef3e7277d7d4760824a3436ed4a847d P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 4.
I20260809 15:46:39.519483 23353 leader_election.cc:302] T cef3e7277d7d4760824a3436ed4a847d P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:46:39.531559 23397 raft_consensus.cc:595] T 00a1881267a14ba8b78c81040f17a014 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Received request to transfer leadership to TS 65c36d164b0540f099d592f38307ee6c
I20260809 15:46:39.531953 23397 raft_consensus.cc:603] T 00a1881267a14ba8b78c81040f17a014 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260809 15:46:39.448330 26275 consensus_queue.cc:1035] T f9e7cadcd24d45fcbfa84209225e8f2d P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.036s
I20260809 15:46:39.480446 26231 raft_consensus.cc:482] T 998c6cee72cf40228c6dba064c0db2da P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:39.463176 23393 raft_consensus.cc:2454] T f7715d71dbac447b9351fc5eb36c5e80 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 4.
I20260809 15:46:39.460592 24448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0741ab44d99439abc3aa3bdd27ee160" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f"
I20260809 15:46:39.533259 23521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba56725e42f74a1baa4c04d9660727f9" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:39.463750 25959 leader_election.cc:288] T cef3e7277d7d4760824a3436ed4a847d P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:39.537599 23528 raft_consensus.cc:1226] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Rejecting Update request from peer 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 2. Current term is 3. Ops: []
I20260809 15:46:39.440654 26384 raft_consensus.cc:504] T 297c82ed906d4b48b184985eed3de601 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:46:39.550720 23472 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 100558us
I20260809 15:46:39.551338 23522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e17dcb5efe45b8b6a8c42da03b4819" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:39.555335 23522 raft_consensus.cc:2379] T 28e17dcb5efe45b8b6a8c42da03b4819 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 in current term 4: Already voted for candidate 65c36d164b0540f099d592f38307ee6c in this term.
I20260809 15:46:39.555258 23472 leader_election.cc:302] T f9e7cadcd24d45fcbfa84209225e8f2d P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters: 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:39.469292 26381 raft_consensus.cc:3037] T c5c2e425a74b4e64ae70b5def750434c P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:39.503137 24452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3aa6a9a52804df0ba9b0ca3c82f38c3" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f"
I20260809 15:46:39.557933 23517 raft_consensus.cc:1226] T 0acc47ce21c3422698d1ef7277c3ed63 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Rejecting Update request from peer 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 3. Current term is 4. Ops: []
I20260809 15:46:39.530103 26386 raft_consensus.cc:2726] T 0da4d5283c8a46878bd39b85e5c71f66 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260809 15:46:39.558954 24410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 249377us
I20260809 15:46:39.561487 23351 leader_election.cc:302] T 2db057fd10e149a9895dff6ef8c9f038 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:39.530254 26222 raft_consensus.cc:2781] T 6ced718c1a5c4da79cb8c1ebd1435b87 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader election won for term 3
I20260809 15:46:39.562482 23521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee02fff2c9c48a19f43eb358a89e7b0" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:39.530854 26194 raft_consensus.cc:2781] T d3be1e356b254d22b995be32f1c12f6f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader election won for term 3
I20260809 15:46:39.563413 23529 raft_consensus.cc:1261] T 99bdb93503db4de4a1cfb0161ea0c07c P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Refusing update from remote peer 67b3aeadfa7a432abe8af4d292e04ab9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 15:46:39.538121 25931 raft_consensus.cc:2781] T cef3e7277d7d4760824a3436ed4a847d P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260809 15:46:39.535751 24451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bcac4b24e7a4448abdc934bba5a624e" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:39.572139 23522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3924b648461e463b9d7925e5671d68ae" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:39.516947 26040 consensus_queue.cc:1046] T 0acc47ce21c3422698d1ef7277c3ed63 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } attrs { replace: true }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.050s
I20260809 15:46:39.550078 25980 leader_election.cc:288] T 3d880de657e544468c82157769622a26 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:39.564257 26278 raft_consensus.cc:482] T 907bacfa79d04a55a68c8dbe60056462 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 15:46:39.567845 25746 consensus_meta.cc:287] T 294d4bcc1e7a477cb56455581277a60f P 61e69695258f48baa984c9d395e8ce3f: Time spent flushing consensus metadata: real 0.583s user 0.003s sys 0.006s
I20260809 15:46:39.557181 24448 raft_consensus.cc:3037] T c0741ab44d99439abc3aa3bdd27ee160 P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:46:39.594180 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "213543cdbe814c89a17a963469922c1b" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:39.573227 25931 raft_consensus.cc:482] T cef3e7277d7d4760824a3436ed4a847d P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:46:39.602790 24472 raft_consensus.cc:1261] T 294d4bcc1e7a477cb56455581277a60f P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 6. (term mismatch)
I20260809 15:46:39.576066 26185 raft_consensus.cc:2781] T df20671a7b874dbb98c0ef28371c1308 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260809 15:46:39.551054 26231 raft_consensus.cc:504] T 998c6cee72cf40228c6dba064c0db2da P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:46:39.609717 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 56448.8us
I20260809 15:46:39.535060 26393 raft_consensus.cc:504] T 73f6f44c259547f1b05a3a434616fa54 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:39.607616 23529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d880de657e544468c82157769622a26" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:39.604475 24472 pending_rounds.cc:76] T 294d4bcc1e7a477cb56455581277a60f P 61e69695258f48baa984c9d395e8ce3f: Aborting all ops after (but not including) 5
I20260809 15:46:39.618949 24472 pending_rounds.cc:97] T 294d4bcc1e7a477cb56455581277a60f P 61e69695258f48baa984c9d395e8ce3f: Aborting uncommitted NO_OP operation due to leader change: 4.6
I20260809 15:46:39.575834 25959 raft_consensus.cc:482] T 73f6f44c259547f1b05a3a434616fa54 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:39.580967 25816 raft_consensus.cc:2781] T f9e7cadcd24d45fcbfa84209225e8f2d P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260809 15:46:39.562021 26333 raft_consensus.cc:504] T d3e94b7e50754a81b3d72a07476cb7b9 P 61e69695258f48baa984c9d395e8ce3f [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:39.605682 26275 raft_consensus.cc:482] T 3296ef50be254fc29cfc77d22a8f7a23 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:39.582626 26194 raft_consensus.cc:686] T d3be1e356b254d22b995be32f1c12f6f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 LEADER]: Becoming Leader. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Running, Role: LEADER
I20260809 15:46:39.621279 23351 leader_election.cc:398] T 8e4ffcf83c234ae7910ddef3ec0b4bda P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 election: Vote denied by peer 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459) with higher term. Message: Invalid argument: T 8e4ffcf83c234ae7910ddef3ec0b4bda P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d for earlier term 3. Current term is 4.
I20260809 15:46:39.631500 23351 leader_election.cc:401] T 8e4ffcf83c234ae7910ddef3ec0b4bda P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20260809 15:46:39.580739 24453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472816994d814cb198b69e5248697440" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:39.619234 24472 raft_consensus.cc:2866] T 294d4bcc1e7a477cb56455581277a60f P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 15:46:39.587445 26118 leader_election.cc:288] T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
W20260809 15:46:39.616878 26228 consensus_meta.cc:287] T 07fcabb569414bddbd39df6cedd24e64 P 65c36d164b0540f099d592f38307ee6c: Time spent flushing consensus metadata: real 0.508s user 0.009s sys 0.000s
I20260809 15:46:39.610574 25931 raft_consensus.cc:3037] T cef3e7277d7d4760824a3436ed4a847d P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Advancing to term 5
I20260809 15:46:39.613202 26285 raft_consensus.cc:3037] T 2db057fd10e149a9895dff6ef8c9f038 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:39.634967 23472 leader_election.cc:302] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:39.614217 26185 raft_consensus.cc:482] T df20671a7b874dbb98c0ef28371c1308 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:46:39.610210 25980 raft_consensus.cc:482] T e14d118b2eaa42aa8a7cd4025aba7128 P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:39.637609 23273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df20671a7b874dbb98c0ef28371c1308" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:39.640944 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "538a228ddfb54bc6a36528d5ff07597b" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:39.616331 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3e94b7e50754a81b3d72a07476cb7b9" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:39.634379 23393 raft_consensus.cc:1226] T f7715d71dbac447b9351fc5eb36c5e80 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Rejecting Update request from peer 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 2. Current term is 4. Ops: []
I20260809 15:46:39.628813 23395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2424fd9bd7f14e508fd106cbfda786a4" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
W20260809 15:46:39.647398 26391 log.cc:924] Time spent T 00000000000000000000000000000000 P a86a2513b8024a869d80303dcbf44163: Append to log took a long time: real 0.213s user 0.000s sys 0.001s
I20260809 15:46:39.614722 26278 raft_consensus.cc:504] T 907bacfa79d04a55a68c8dbe60056462 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
W20260809 15:46:39.715443 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 205025us
I20260809 15:46:39.699138 24453 raft_consensus.cc:2379] T 472816994d814cb198b69e5248697440 P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c in current term 4: Already voted for candidate 61e69695258f48baa984c9d395e8ce3f in this term.
I20260809 15:46:39.643965 24451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee02fff2c9c48a19f43eb358a89e7b0" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:39.658911 26185 raft_consensus.cc:3037] T df20671a7b874dbb98c0ef28371c1308 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:46:39.628049 25959 raft_consensus.cc:504] T 73f6f44c259547f1b05a3a434616fa54 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:46:39.697831 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 82665.8us
I20260809 15:46:39.842113 23351 leader_election.cc:302] T f7715d71dbac447b9351fc5eb36c5e80 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c
I20260809 15:46:39.811657 26409 consensus_queue.cc:1046] T 472816994d814cb198b69e5248697440 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260809 15:46:39.813233 26096 raft_consensus.cc:482] T 0c48a3049897452383257aa757834fb5 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:39.847021 26096 raft_consensus.cc:504] T 0c48a3049897452383257aa757834fb5 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:39.667411 25967 raft_consensus.cc:2726] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 15:46:39.660516 24452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e71a05ba1d4d42bd95c3f85f9313399e" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:39.848521 24452 raft_consensus.cc:2454] T e71a05ba1d4d42bd95c3f85f9313399e P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 3.
I20260809 15:46:39.849022 26096 leader_election.cc:288] T 0c48a3049897452383257aa757834fb5 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:39.826870 23345 leader_election.cc:302] T 563daf42d7784db195ea99c3cb21b370 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:46:39.761372 25899 leader_election.cc:288] T 00a1881267a14ba8b78c81040f17a014 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:39.694149 26118 raft_consensus.cc:2726] T 8e4ffcf83c234ae7910ddef3ec0b4bda P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459) with higher term. Message: Invalid argument: T 8e4ffcf83c234ae7910ddef3ec0b4bda P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d for earlier term 3. Current term is 4.
I20260809 15:46:39.635177 26275 raft_consensus.cc:504] T 3296ef50be254fc29cfc77d22a8f7a23 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:39.835029 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1a134ba33ea4076bbaac0ae460c17ba" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:39.864120 23402 raft_consensus.cc:2454] T d1a134ba33ea4076bbaac0ae460c17ba P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 2.
W20260809 15:46:39.836155 23396 consensus_meta.cc:287] T 2424fd9bd7f14e508fd106cbfda786a4 P c0f38c1908414566aa9115bb2cbdd85d: Time spent flushing consensus metadata: real 0.553s user 0.008s sys 0.000s
I20260809 15:46:39.868960 23396 raft_consensus.cc:2454] T 2424fd9bd7f14e508fd106cbfda786a4 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 5.
I20260809 15:46:39.648857 26228 raft_consensus.cc:504] T 07fcabb569414bddbd39df6cedd24e64 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:39.759447 26279 raft_consensus.cc:482] T cee02fff2c9c48a19f43eb358a89e7b0 P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 15:46:39.808579 26387 log.cc:924] Time spent T 8e4ffcf83c234ae7910ddef3ec0b4bda P 61e69695258f48baa984c9d395e8ce3f: Append to log took a long time: real 0.271s user 0.000s sys 0.001s
I20260809 15:46:39.844194 25956 consensus_queue.cc:1035] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260809 15:46:39.844583 25875 consensus_queue.cc:1035] T cee02fff2c9c48a19f43eb358a89e7b0 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260809 15:46:39.846503 25959 leader_election.cc:288] T 73f6f44c259547f1b05a3a434616fa54 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:39.682726 25980 raft_consensus.cc:504] T e14d118b2eaa42aa8a7cd4025aba7128 P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
W20260809 15:46:39.829434 23519 consensus_meta.cc:287] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 65c36d164b0540f099d592f38307ee6c: Time spent flushing consensus metadata: real 0.597s user 0.007s sys 0.000s
I20260809 15:46:39.917538 23519 raft_consensus.cc:2454] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 5.
I20260809 15:46:39.810972 23399 raft_consensus.cc:1226] T 7660dfa2ef694927b8eda4100a2bcd9b P c0f38c1908414566aa9115bb2cbdd85d [term 5 LEADER]: Rejecting Update request from peer 61e69695258f48baa984c9d395e8ce3f for earlier term 4. Current term is 5. Ops: []
I20260809 15:46:39.681938 26214 consensus_queue.cc:1035] T e14d118b2eaa42aa8a7cd4025aba7128 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.176s
I20260809 15:46:39.830858 26346 raft_consensus.cc:482] T 907bacfa79d04a55a68c8dbe60056462 P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:39.808040 26053 raft_consensus.cc:482] T 213543cdbe814c89a17a963469922c1b P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
W20260809 15:46:39.861658 23392 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.611s user 0.021s sys 0.049s
I20260809 15:46:39.921442 23522 raft_consensus.cc:1226] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Rejecting Update request from peer 61e69695258f48baa984c9d395e8ce3f for earlier term 4. Current term is 5. Ops: []
I20260809 15:46:39.873265 25899 raft_consensus.cc:2741] T f7715d71dbac447b9351fc5eb36c5e80 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
W20260809 15:46:39.920941 26380 log.cc:924] Time spent T 213543cdbe814c89a17a963469922c1b P 67b3aeadfa7a432abe8af4d292e04ab9: Append to log took a long time: real 0.338s user 0.001s sys 0.000s
I20260809 15:46:39.915313 26384 leader_election.cc:288] T 297c82ed906d4b48b184985eed3de601 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:39.902417 26222 raft_consensus.cc:686] T 6ced718c1a5c4da79cb8c1ebd1435b87 P c0f38c1908414566aa9115bb2cbdd85d [term 3 LEADER]: Becoming Leader. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:46:39.878294 26118 raft_consensus.cc:2781] T 563daf42d7784db195ea99c3cb21b370 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260809 15:46:39.922063 23395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4564dfc6d4094c1b9240b367ccecf3b1" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:39.988338 23519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3e94b7e50754a81b3d72a07476cb7b9" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:40.000842 23400 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55320 (request call id 1196) took 1022 ms. Trace:
I20260809 15:46:39.878041 24468 raft_consensus.cc:2354] T d3e94b7e50754a81b3d72a07476cb7b9 P 61e69695258f48baa984c9d395e8ce3f [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c for earlier term 5. Current term is 6.
I20260809 15:46:40.001466 23400 rpcz_store.cc:276] 0809 15:46:38.978694 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:39.099479 (+120785us) service_pool.cc:227] Handling call
0809 15:46:40.000660 (+901181us) spinlock_profiling.cc:243] Waited 314 ms on lock 0x7b540025e754. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e92242ae 00007f34ee161c27 00007f34e7849c04 00007f34e7849b92 00007f34e7849ac1 00007f34e7849a3c 00007f34e7848cf2 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:46:40.000788 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":721152768}
I20260809 15:46:40.000823 23519 raft_consensus.cc:3037] T d3e94b7e50754a81b3d72a07476cb7b9 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Advancing to term 6
I20260809 15:46:39.879333 26279 raft_consensus.cc:504] T cee02fff2c9c48a19f43eb358a89e7b0 P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:39.858448 26194 consensus_queue.cc:237] T d3be1e356b254d22b995be32f1c12f6f P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
W20260809 15:46:39.953096 23392 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.703s user 0.021s sys 0.050s
W20260809 15:46:39.920343 26284 consensus_meta.cc:287] T d0721a4403a440f383318a3cea9d1442 P c0f38c1908414566aa9115bb2cbdd85d: Time spent flushing consensus metadata: real 0.668s user 0.010s sys 0.000s
I20260809 15:46:39.961098 26399 raft_consensus.cc:482] T ae6d2c08a7a94f58ac8dd698e3fd456c P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:39.920991 24455 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3be1e356b254d22b995be32f1c12f6f" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:39.989404 26333 leader_election.cc:288] T d3e94b7e50754a81b3d72a07476cb7b9 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 6 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
W20260809 15:46:40.015767 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 203514us
I20260809 15:46:39.987524 26393 leader_election.cc:288] T 73f6f44c259547f1b05a3a434616fa54 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:39.987973 23393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e17dcb5efe45b8b6a8c42da03b4819" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:40.002352 26118 raft_consensus.cc:482] T 563daf42d7784db195ea99c3cb21b370 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting leader election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
I20260809 15:46:40.022290 23400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba56725e42f74a1baa4c04d9660727f9" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:40.008026 24475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16ca412a0dae4feebe85dd01bf1a9173" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:39.986774 26231 leader_election.cc:288] T 998c6cee72cf40228c6dba064c0db2da P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
W20260809 15:46:40.027944 26401 log.cc:924] Time spent T 538a228ddfb54bc6a36528d5ff07597b P c0f38c1908414566aa9115bb2cbdd85d: Append to log took a long time: real 0.367s user 0.001s sys 0.000s
I20260809 15:46:39.961747 26053 raft_consensus.cc:504] T 213543cdbe814c89a17a963469922c1b P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:39.955314 26346 raft_consensus.cc:504] T 907bacfa79d04a55a68c8dbe60056462 P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:40.034080 23274 raft_consensus.cc:1261] T 953eea268e17491ca76f3a555860a512 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260809 15:46:40.012101 23395 raft_consensus.cc:2454] T 4564dfc6d4094c1b9240b367ccecf3b1 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 3.
I20260809 15:46:40.040527 23270 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53554 (request call id 1265) took 1115 ms. Trace:
W20260809 15:46:40.044991 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 202946us
I20260809 15:46:40.045765 23351 leader_election.cc:302] T ba56725e42f74a1baa4c04d9660727f9 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:40.041193 23270 rpcz_store.cc:276] 0809 15:46:38.924782 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:38.925556 (+ 774us) service_pool.cc:227] Handling call
0809 15:46:38.925823 (+ 267us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 15:46:38.932667 (+ 6844us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:46:38.932693 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 15:46:38.939429 (+ 6736us) log.cc:842] Serialized 26 byte log entry
0809 15:46:39.333036 (+393607us) raft_consensus.cc:1631] Marking committed up to 1
0809 15:46:39.333070 (+ 34us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:39.333129 (+ 59us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 15:46:40.040325 (+707196us) raft_consensus.cc:1671] finished
0809 15:46:40.040344 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:40.040475 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":6614,"raft.run_cpu_time_us":122,"raft.run_wall_time_us":5055,"thread_start_us":288601,"threads_started":1,"wal-append.queue_time_us":491063}
I20260809 15:46:40.030109 26118 raft_consensus.cc:3037] T 563daf42d7784db195ea99c3cb21b370 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:40.046039 23519 raft_consensus.cc:2454] T d3e94b7e50754a81b3d72a07476cb7b9 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 6.
I20260809 15:46:40.017433 26284 raft_consensus.cc:504] T d0721a4403a440f383318a3cea9d1442 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:40.053131 23400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5c2e425a74b4e64ae70b5def750434c" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:40.053570 23351 leader_election.cc:302] T 05ad7d8fdca64e3da527280ca5be9280 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:40.054783 23400 raft_consensus.cc:2454] T c5c2e425a74b4e64ae70b5def750434c P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 3.
I20260809 15:46:40.036252 24451 raft_consensus.cc:2454] T cee02fff2c9c48a19f43eb358a89e7b0 P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 3.
I20260809 15:46:40.056981 26457 consensus_queue.cc:1046] T 0da4d5283c8a46878bd39b85e5c71f66 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.015s
I20260809 15:46:40.069211 26279 leader_election.cc:288] T cee02fff2c9c48a19f43eb358a89e7b0 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:40.072142 26346 leader_election.cc:288] T 907bacfa79d04a55a68c8dbe60056462 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:40.072784 26450 consensus_queue.cc:1046] T 7baa2492df0340c4a5ac241959ce1ac3 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 15:46:40.073370 25929 raft_consensus.cc:482] T 0c48a3049897452383257aa757834fb5 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:40.073865 25929 raft_consensus.cc:504] T 0c48a3049897452383257aa757834fb5 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
W20260809 15:46:40.074476 26179 consensus_meta.cc:287] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 61e69695258f48baa984c9d395e8ce3f: Time spent flushing consensus metadata: real 0.625s user 0.005s sys 0.000s
I20260809 15:46:40.074589 23393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00a1881267a14ba8b78c81040f17a014" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:40.040777 23286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "297c82ed906d4b48b184985eed3de601" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:40.067555 26275 leader_election.cc:288] T 3296ef50be254fc29cfc77d22a8f7a23 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:40.061777 23400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df20671a7b874dbb98c0ef28371c1308" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:40.062842 23270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73f6f44c259547f1b05a3a434616fa54" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:40.086830 23270 raft_consensus.cc:2379] T 73f6f44c259547f1b05a3a434616fa54 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c in current term 4: Already voted for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in this term.
W20260809 15:46:40.058877 24410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 43826.4us
I20260809 15:46:40.058044 23274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c48a3049897452383257aa757834fb5" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:40.078362 23228 leader_election.cc:302] T c5c2e425a74b4e64ae70b5def750434c P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:46:40.086230 23400 raft_consensus.cc:2454] T df20671a7b874dbb98c0ef28371c1308 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 3.
I20260809 15:46:40.097699 24410 leader_election.cc:302] T d3e94b7e50754a81b3d72a07476cb7b9 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c; no voters:
I20260809 15:46:40.092374 23527 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56770 (request call id 1232) took 1123 ms. Trace:
I20260809 15:46:40.048709 26278 leader_election.cc:288] T 907bacfa79d04a55a68c8dbe60056462 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:40.098941 23527 rpcz_store.cc:276] 0809 15:46:38.969271 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:38.974737 (+ 5466us) service_pool.cc:227] Handling call
0809 15:46:39.022485 (+ 47748us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 15:46:39.236339 (+213854us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:46:39.236367 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 15:46:39.362038 (+125671us) spinlock_profiling.cc:243] Waited 1.62 ms on lock 0x7b2c00092468. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e7c03e11 00007f34e7c13442 00007f34e925bc11 00007f34e922058a 00007f34e921f99e 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772
0809 15:46:39.362408 (+ 370us) log.cc:842] Serialized 26 byte log entry
0809 15:46:39.576142 (+213734us) raft_consensus.cc:1631] Marking committed up to 2
0809 15:46:39.576178 (+ 36us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:39.576235 (+ 57us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 15:46:40.075530 (+499295us) raft_consensus.cc:1671] finished
0809 15:46:40.075549 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:40.092311 (+ 16762us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":61266,"raft.run_cpu_time_us":171,"raft.run_wall_time_us":5115,"spinlock_wait_cycles":3736320,"thread_start_us":372187,"threads_started":2,"wal-append.queue_time_us":660645}
I20260809 15:46:40.088326 23274 raft_consensus.cc:2454] T 0c48a3049897452383257aa757834fb5 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 2.
W20260809 15:46:40.064847 26413 log.cc:924] Time spent T 953eea268e17491ca76f3a555860a512 P 65c36d164b0540f099d592f38307ee6c: Append to log took a long time: real 0.053s user 0.000s sys 0.000s
W20260809 15:46:40.061419 25793 consensus_meta.cc:287] T 2db057fd10e149a9895dff6ef8c9f038 P 65c36d164b0540f099d592f38307ee6c: Time spent flushing consensus metadata: real 0.770s user 0.007s sys 0.000s
I20260809 15:46:40.054422 25956 raft_consensus.cc:2726] T ba56725e42f74a1baa4c04d9660727f9 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260809 15:46:40.063230 26335 consensus_meta.cc:287] T d3be1e356b254d22b995be32f1c12f6f P 61e69695258f48baa984c9d395e8ce3f: Time spent flushing consensus metadata: real 0.733s user 0.008s sys 0.000s
I20260809 15:46:40.075361 23393 raft_consensus.cc:2354] T 00a1881267a14ba8b78c81040f17a014 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 4. Current term is 5.
I20260809 15:46:40.022043 26399 raft_consensus.cc:504] T ae6d2c08a7a94f58ac8dd698e3fd456c P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:40.074890 26179 raft_consensus.cc:504] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 61e69695258f48baa984c9d395e8ce3f [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
W20260809 15:46:40.132117 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 53824.3us
I20260809 15:46:40.106849 23394 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52856 (request call id 134) took 1284 ms. Trace:
I20260809 15:46:40.107194 26442 raft_consensus.cc:2726] T 05ad7d8fdca64e3da527280ca5be9280 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260809 15:46:40.107753 23270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998c6cee72cf40228c6dba064c0db2da" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:40.108048 23274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07fcabb569414bddbd39df6cedd24e64" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
W20260809 15:46:40.108737 26435 log.cc:924] Time spent T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 65c36d164b0540f099d592f38307ee6c: Append to log took a long time: real 0.060s user 0.000s sys 0.000s
W20260809 15:46:40.112183 23276 consensus_meta.cc:287] T c0741ab44d99439abc3aa3bdd27ee160 P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent flushing consensus metadata: real 0.944s user 0.005s sys 0.005s
I20260809 15:46:40.115617 25931 raft_consensus.cc:504] T cef3e7277d7d4760824a3436ed4a847d P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:40.103075 23286 raft_consensus.cc:2454] T 297c82ed906d4b48b184985eed3de601 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 5.
W20260809 15:46:40.119469 23223 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.68:34689, user_credentials={real_user=slave}} blocked reactor thread for 165915us
I20260809 15:46:40.064465 26053 leader_election.cc:288] T 213543cdbe814c89a17a963469922c1b P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:40.064978 26405 raft_consensus.cc:3032] T 472816994d814cb198b69e5248697440 P c0f38c1908414566aa9115bb2cbdd85d [term 3 LEADER]: Stepping down as leader of term 3
W20260809 15:46:40.123013 23345 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.68:34689, user_credentials={real_user=slave}} blocked reactor thread for 296199us
I20260809 15:46:40.128779 23400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3e94b7e50754a81b3d72a07476cb7b9" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:40.048417 26222 consensus_queue.cc:237] T 6ced718c1a5c4da79cb8c1ebd1435b87 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:40.092409 26228 leader_election.cc:288] T 07fcabb569414bddbd39df6cedd24e64 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 5 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
W20260809 15:46:40.107090 26437 log.cc:924] Time spent T 6ae14a74a26f434db08c652e097385c8 P 65c36d164b0540f099d592f38307ee6c: Append to log took a long time: real 0.231s user 0.000s sys 0.001s
I20260809 15:46:40.107084 25941 raft_consensus.cc:482] T 99bdb93503db4de4a1cfb0161ea0c07c P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:40.192286 23274 raft_consensus.cc:3037] T 07fcabb569414bddbd39df6cedd24e64 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Advancing to term 5
I20260809 15:46:40.134711 23394 rpcz_store.cc:276] 0809 15:46:38.822696 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:38.823539 (+ 843us) service_pool.cc:227] Handling call
0809 15:46:38.823851 (+ 312us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:38.824057 (+ 206us) raft_consensus.cc:1520] Early marking committed up to index 5
0809 15:46:39.454891 (+630834us) log.cc:842] Serialized 14 byte log entry
0809 15:46:40.104406 (+649515us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 15:46:40.104449 (+ 43us) raft_consensus.cc:1631] Marking committed up to 5
0809 15:46:40.104488 (+ 39us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:40.104545 (+ 57us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:40.106796 (+ 2251us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5375,"prepare.queue_time_us":263464,"prepare.run_cpu_time_us":23,"prepare.run_wall_time_us":24,"raft.queue_time_us":29665,"raft.run_cpu_time_us":121,"raft.run_wall_time_us":35989,"thread_start_us":1271109,"threads_started":3,"wal-append.queue_time_us":554856}
I20260809 15:46:40.104722 26144 raft_consensus.cc:2741] T 3296ef50be254fc29cfc77d22a8f7a23 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Leader pre-election decision vote started in defunct term 1: won
I20260809 15:46:40.192643 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e14d118b2eaa42aa8a7cd4025aba7128" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:40.192705 23270 raft_consensus.cc:2454] T 998c6cee72cf40228c6dba064c0db2da P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 6.
I20260809 15:46:40.148144 26457 raft_consensus.cc:3032] T 7baa2492df0340c4a5ac241959ce1ac3 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 LEADER]: Stepping down as leader of term 1
I20260809 15:46:40.121086 26279 raft_consensus.cc:2781] T d3e94b7e50754a81b3d72a07476cb7b9 P 61e69695258f48baa984c9d395e8ce3f [term 6 FOLLOWER]: Leader election won for term 6
I20260809 15:46:40.134734 25980 leader_election.cc:288] T e14d118b2eaa42aa8a7cd4025aba7128 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
W20260809 15:46:40.124922 25885 consensus_meta.cc:287] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent flushing consensus metadata: real 0.621s user 0.009s sys 0.000s
I20260809 15:46:40.098356 23351 leader_election.cc:302] T c5c2e425a74b4e64ae70b5def750434c P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c
I20260809 15:46:40.204116 26405 raft_consensus.cc:729] T 472816994d814cb198b69e5248697440 P c0f38c1908414566aa9115bb2cbdd85d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:46:40.198482 23400 raft_consensus.cc:3037] T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Advancing to term 6
I20260809 15:46:40.147552 26214 consensus_queue.cc:1035] T 953eea268e17491ca76f3a555860a512 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.095s
I20260809 15:46:40.107057 23518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee02fff2c9c48a19f43eb358a89e7b0" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:40.206617 24475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3924b648461e463b9d7925e5671d68ae" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:40.135069 23395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05ad7d8fdca64e3da527280ca5be9280" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:40.196550 25929 leader_election.cc:288] T 0c48a3049897452383257aa757834fb5 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:40.194288 23276 raft_consensus.cc:2454] T c0741ab44d99439abc3aa3bdd27ee160 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 4.
I20260809 15:46:40.198239 26420 raft_consensus.cc:482] T 472816994d814cb198b69e5248697440 P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 15:46:40.213889 26285 consensus_meta.cc:287] T 2db057fd10e149a9895dff6ef8c9f038 P c0f38c1908414566aa9115bb2cbdd85d: Time spent flushing consensus metadata: real 0.557s user 0.009s sys 0.000s
W20260809 15:46:40.215973 25639 consensus_meta.cc:287] T 00a1881267a14ba8b78c81040f17a014 P 65c36d164b0540f099d592f38307ee6c: Time spent flushing consensus metadata: real 1.395s user 0.008s sys 0.004s
W20260809 15:46:40.194480 23470 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 44692.9us
W20260809 15:46:40.246910 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 148635us
W20260809 15:46:40.128748 23468 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.68:34689, user_credentials={real_user=slave}} blocked reactor thread for 322525us
I20260809 15:46:40.284003 23468 leader_election.cc:302] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:40.279721 23223 leader_election.cc:302] T e71a05ba1d4d42bd95c3f85f9313399e P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
W20260809 15:46:40.258008 26148 consensus_meta.cc:287] T e71a05ba1d4d42bd95c3f85f9313399e P c0f38c1908414566aa9115bb2cbdd85d: Time spent flushing consensus metadata: real 0.776s user 0.009s sys 0.000s
I20260809 15:46:40.252249 26279 raft_consensus.cc:686] T d3e94b7e50754a81b3d72a07476cb7b9 P 61e69695258f48baa984c9d395e8ce3f [term 6 LEADER]: Becoming Leader. State: Replica: 61e69695258f48baa984c9d395e8ce3f, State: Running, Role: LEADER
I20260809 15:46:40.212669 25746 raft_consensus.cc:482] T 294d4bcc1e7a477cb56455581277a60f P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:40.253641 23394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5c2e425a74b4e64ae70b5def750434c" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:40.242749 26457 raft_consensus.cc:729] T 7baa2492df0340c4a5ac241959ce1ac3 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Running, Role: LEADER
I20260809 15:46:40.248101 26194 raft_consensus.cc:482] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 15:46:40.251425 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 58916.8us
I20260809 15:46:40.266360 23404 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55320 (request call id 1198) took 1173 ms. Trace:
I20260809 15:46:40.276459 23400 raft_consensus.cc:2454] T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 6.
I20260809 15:46:40.261533 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3be1e356b254d22b995be32f1c12f6f" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "61e69695258f48baa984c9d395e8ce3f"
I20260809 15:46:40.265336 23397 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:51280 (request call id 10) took 1120 ms. Trace:
I20260809 15:46:40.132236 26335 raft_consensus.cc:504] T d3be1e356b254d22b995be32f1c12f6f P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:40.311461 24463 raft_consensus.cc:2379] T d3be1e356b254d22b995be32f1c12f6f P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d in current term 4: Already voted for candidate 61e69695258f48baa984c9d395e8ce3f in this term.
I20260809 15:46:40.312288 26335 leader_election.cc:288] T d3be1e356b254d22b995be32f1c12f6f P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 4 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:40.277796 24475 raft_consensus.cc:2396] T 3924b648461e463b9d7925e5671d68ae P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d for term 6 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260809 15:46:40.314119 23345 leader_election.cc:302] T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c
I20260809 15:46:40.253872 26275 raft_consensus.cc:2781] T c5c2e425a74b4e64ae70b5def750434c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260809 15:46:40.258404 26441 log.cc:924] Time spent T d2d4f749fcfd4fb3bfb8115af3471598 P c0f38c1908414566aa9115bb2cbdd85d: Append to log took a long time: real 0.165s user 0.001s sys 0.000s
W20260809 15:46:40.263870 24448 consensus_meta.cc:287] T c0741ab44d99439abc3aa3bdd27ee160 P 61e69695258f48baa984c9d395e8ce3f: Time spent flushing consensus metadata: real 0.635s user 0.006s sys 0.002s
I20260809 15:46:40.254560 26468 consensus_queue.cc:1046] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.156s
I20260809 15:46:40.256675 25816 raft_consensus.cc:482] T f9e7cadcd24d45fcbfa84209225e8f2d P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Starting leader election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
I20260809 15:46:40.284814 26420 raft_consensus.cc:504] T 472816994d814cb198b69e5248697440 P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:40.254966 26485 consensus_queue.cc:1046] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.035s
I20260809 15:46:40.256618 26405 consensus_queue.cc:260] T 472816994d814cb198b69e5248697440 P c0f38c1908414566aa9115bb2cbdd85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:40.257899 23393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e71a05ba1d4d42bd95c3f85f9313399e" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:40.254817 26436 raft_consensus.cc:482] T d0721a4403a440f383318a3cea9d1442 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:40.287112 23470 leader_election.cc:302] T 0c48a3049897452383257aa757834fb5 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:46:40.120837 25793 raft_consensus.cc:504] T 2db057fd10e149a9895dff6ef8c9f038 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:40.295855 23468 leader_election.cc:302] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:40.340503 23280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae6d2c08a7a94f58ac8dd698e3fd456c" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
W20260809 15:46:40.296088 24410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 44421.4us
I20260809 15:46:40.253218 24455 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a823d35f4f348d1b52ba9e6d80b00d2" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f"
I20260809 15:46:40.259529 26293 raft_consensus.cc:3032] T 7660dfa2ef694927b8eda4100a2bcd9b P 61e69695258f48baa984c9d395e8ce3f [term 4 LEADER]: Stepping down as leader of term 4
I20260809 15:46:40.241899 25941 raft_consensus.cc:504] T 99bdb93503db4de4a1cfb0161ea0c07c P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:40.306224 23400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df20671a7b874dbb98c0ef28371c1308" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:40.354205 23400 raft_consensus.cc:3037] T df20671a7b874dbb98c0ef28371c1308 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:46:40.290779 26148 raft_consensus.cc:2726] T e71a05ba1d4d42bd95c3f85f9313399e P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 15:46:40.278175 23395 raft_consensus.cc:2454] T 05ad7d8fdca64e3da527280ca5be9280 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 3.
I20260809 15:46:40.264179 24453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "907bacfa79d04a55a68c8dbe60056462" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:40.250353 26474 raft_consensus.cc:2781] T c5c2e425a74b4e64ae70b5def750434c P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260809 15:46:40.309726 26284 leader_election.cc:288] T d0721a4403a440f383318a3cea9d1442 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:40.309513 26457 consensus_queue.cc:260] T 7baa2492df0340c4a5ac241959ce1ac3 P 67b3aeadfa7a432abe8af4d292e04ab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:40.305375 23404 rpcz_store.cc:276] 0809 15:46:39.092789 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:39.326832 (+234043us) service_pool.cc:227] Handling call
0809 15:46:39.352304 (+ 25472us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:39.444093 (+ 91789us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:40.266143 (+822050us) spinlock_profiling.cc:243] Waited 85 ms on lock 0x7b54002206d4. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e9220f51 00007f34e921f99e 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772 00007f34e764a94c 00007f34e7649d86
0809 15:46:40.266307 (+ 164us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":195456896}
I20260809 15:46:40.307655 23397 rpcz_store.cc:276] 0809 15:46:39.160358 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:39.253817 (+ 93459us) service_pool.cc:227] Handling call
0809 15:46:40.265159 (+1011342us) spinlock_profiling.cc:243] Waited 156 ms on lock 0x7b540025e754. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e921abc3 00007f34ee16481a 00007f34e7857d04 00007f34e7857c92 00007f34e7857bc1 00007f34e7857b3c 00007f34e7856df2 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:46:40.265261 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":358288640}
I20260809 15:46:40.327416 26194 raft_consensus.cc:504] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:40.374943 23282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0721a4403a440f383318a3cea9d1442" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:40.375882 26194 leader_election.cc:288] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:40.376367 26194 raft_consensus.cc:482] T 763e58d6ba644eea830a6c1664d39c9b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:40.376748 26194 raft_consensus.cc:504] T 763e58d6ba644eea830a6c1664d39c9b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:40.377416 23397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998c6cee72cf40228c6dba064c0db2da" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:40.377980 23397 raft_consensus.cc:2454] T 998c6cee72cf40228c6dba064c0db2da P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 6.
I20260809 15:46:40.292874 25931 leader_election.cc:288] T cef3e7277d7d4760824a3436ed4a847d P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:40.316900 26275 raft_consensus.cc:482] T c5c2e425a74b4e64ae70b5def750434c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260809 15:46:40.289206 23276 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53554 (request call id 1267) took 1358 ms (client timeout 1756 ms). Trace:
I20260809 15:46:40.289143 26285 raft_consensus.cc:2726] T 2db057fd10e149a9895dff6ef8c9f038 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 15:46:40.357410 25941 leader_election.cc:288] T 99bdb93503db4de4a1cfb0161ea0c07c P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:40.347404 26436 raft_consensus.cc:504] T d0721a4403a440f383318a3cea9d1442 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:40.291594 26279 consensus_queue.cc:237] T d3e94b7e50754a81b3d72a07476cb7b9 P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
W20260809 15:46:40.330366 23472 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 174769us
I20260809 15:46:40.296936 23223 leader_election.cc:302] T 16ca412a0dae4feebe85dd01bf1a9173 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c
I20260809 15:46:40.287380 25929 raft_consensus.cc:2781] T e71a05ba1d4d42bd95c3f85f9313399e P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260809 15:46:40.284729 25639 raft_consensus.cc:504] T 00a1881267a14ba8b78c81040f17a014 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:40.312232 26179 leader_election.cc:288] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 2 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:40.288079 26228 raft_consensus.cc:2726] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260809 15:46:40.349255 23280 raft_consensus.cc:2379] T ae6d2c08a7a94f58ac8dd698e3fd456c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d in current term 4: Already voted for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in this term.
W20260809 15:46:40.341964 23470 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 57741.2us
W20260809 15:46:40.331621 26040 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.723s user 0.006s sys 0.016s
W20260809 15:46:40.350919 23468 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.68:34689, user_credentials={real_user=slave}} blocked reactor thread for 55128.2us
I20260809 15:46:40.351116 23277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "538a228ddfb54bc6a36528d5ff07597b" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:40.350473 26293 raft_consensus.cc:729] T 7660dfa2ef694927b8eda4100a2bcd9b P 61e69695258f48baa984c9d395e8ce3f [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 61e69695258f48baa984c9d395e8ce3f, State: Running, Role: LEADER
I20260809 15:46:40.356158 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3aa6a9a52804df0ba9b0ca3c82f38c3" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:40.315925 25746 raft_consensus.cc:504] T 294d4bcc1e7a477cb56455581277a60f P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:40.356611 23518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c48a3049897452383257aa757834fb5" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:40.331882 24448 raft_consensus.cc:2454] T c0741ab44d99439abc3aa3bdd27ee160 P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 4.
I20260809 15:46:40.357187 23270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "563daf42d7784db195ea99c3cb21b370" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:40.357594 26161 raft_consensus.cc:2781] T 0c48a3049897452383257aa757834fb5 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260809 15:46:40.357909 25956 raft_consensus.cc:482] T 30c1740ef6874df0a332fbd6fd4b03ce P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:40.353928 26278 raft_consensus.cc:482] T 0c29937a80724ddb9d07a27534a146f3 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 15:46:40.359172 26185 consensus_meta.cc:287] T df20671a7b874dbb98c0ef28371c1308 P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent flushing consensus metadata: real 0.502s user 0.007s sys 0.000s
I20260809 15:46:40.359313 23393 raft_consensus.cc:2454] T e71a05ba1d4d42bd95c3f85f9313399e P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 3.
I20260809 15:46:40.335428 25816 raft_consensus.cc:3037] T f9e7cadcd24d45fcbfa84209225e8f2d P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:46:40.356031 26469 raft_consensus.cc:3032] T 0da4d5283c8a46878bd39b85e5c71f66 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 LEADER]: Stepping down as leader of term 3
I20260809 15:46:40.341255 26326 raft_consensus.cc:3032] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 61e69695258f48baa984c9d395e8ce3f [term 4 LEADER]: Stepping down as leader of term 4
I20260809 15:46:40.456633 23400 raft_consensus.cc:2454] T df20671a7b874dbb98c0ef28371c1308 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 4.
I20260809 15:46:40.637049 26326 raft_consensus.cc:729] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 61e69695258f48baa984c9d395e8ce3f [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 61e69695258f48baa984c9d395e8ce3f, State: Running, Role: LEADER
I20260809 15:46:40.338367 26442 raft_consensus.cc:2726] T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260809 15:46:40.366014 26381 consensus_meta.cc:287] T c5c2e425a74b4e64ae70b5def750434c P 65c36d164b0540f099d592f38307ee6c: Time spent flushing consensus metadata: real 0.787s user 0.006s sys 0.001s
I20260809 15:46:40.366555 23286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2d4f749fcfd4fb3bfb8115af3471598" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:40.305027 26231 consensus_queue.cc:1035] T 3924b648461e463b9d7925e5671d68ae P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.043s
I20260809 15:46:40.368031 26148 raft_consensus.cc:482] T 0a51c0baaa6e432997c9d3552af4f254 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
I20260809 15:46:40.638171 23397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bcac4b24e7a4448abdc934bba5a624e" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:40.637733 23400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "297c82ed906d4b48b184985eed3de601" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
W20260809 15:46:40.639113 23223 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.68:34689, user_credentials={real_user=slave}} blocked reactor thread for 342230us
I20260809 15:46:40.554644 23529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472816994d814cb198b69e5248697440" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:40.640156 23400 raft_consensus.cc:2454] T 297c82ed906d4b48b184985eed3de601 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 5.
I20260809 15:46:40.642274 23274 raft_consensus.cc:2454] T 07fcabb569414bddbd39df6cedd24e64 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 5.
I20260809 15:46:40.410619 23404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73f6f44c259547f1b05a3a434616fa54" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:40.661932 23270 raft_consensus.cc:2379] T 563daf42d7784db195ea99c3cb21b370 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d in current term 3: Already voted for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in this term.
I20260809 15:46:40.661804 23274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73f6f44c259547f1b05a3a434616fa54" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:40.662744 23274 raft_consensus.cc:2454] T 73f6f44c259547f1b05a3a434616fa54 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 4.
I20260809 15:46:40.648219 23404 raft_consensus.cc:2454] T 73f6f44c259547f1b05a3a434616fa54 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 4.
I20260809 15:46:40.647604 23468 leader_election.cc:302] T 472816994d814cb198b69e5248697440 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d
W20260809 15:46:40.367795 26136 consensus_meta.cc:287] T 99bdb93503db4de4a1cfb0161ea0c07c P c0f38c1908414566aa9115bb2cbdd85d: Time spent flushing consensus metadata: real 1.140s user 0.007s sys 0.003s
I20260809 15:46:40.362381 26485 raft_consensus.cc:3032] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 LEADER]: Stepping down as leader of term 2
W20260809 15:46:40.407286 26404 log.cc:924] Time spent T 7baa2492df0340c4a5ac241959ce1ac3 P c0f38c1908414566aa9115bb2cbdd85d: Append to log took a long time: real 0.575s user 0.001s sys 0.002s
I20260809 15:46:40.667145 24448 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38734 (request call id 234) took 1229 ms. Trace:
I20260809 15:46:40.643457 26148 raft_consensus.cc:504] T 0a51c0baaa6e432997c9d3552af4f254 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:40.670070 24452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e4ffcf83c234ae7910ddef3ec0b4bda" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:40.670516 23519 raft_consensus.cc:1226] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Rejecting Update request from peer 61e69695258f48baa984c9d395e8ce3f for earlier term 4. Current term is 5. Ops: [4.5-4.5]
I20260809 15:46:40.408401 26420 leader_election.cc:288] T 472816994d814cb198b69e5248697440 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:40.670720 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c48a3049897452383257aa757834fb5" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:40.670459 25956 raft_consensus.cc:504] T 30c1740ef6874df0a332fbd6fd4b03ce P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:40.669072 25931 raft_consensus.cc:482] T 99bdb93503db4de4a1cfb0161ea0c07c P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:40.675678 26485 raft_consensus.cc:729] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Running, Role: LEADER
I20260809 15:46:40.685710 25931 raft_consensus.cc:504] T 99bdb93503db4de4a1cfb0161ea0c07c P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:40.683768 24448 rpcz_store.cc:276] 0809 15:46:39.434911 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:39.435664 (+ 753us) service_pool.cc:227] Handling call
0809 15:46:40.644503 (+1208839us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:46:40.378466 26194 leader_election.cc:288] T 763e58d6ba644eea830a6c1664d39c9b P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:40.671483 24455 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3296ef50be254fc29cfc77d22a8f7a23" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:40.378571 26275 raft_consensus.cc:3037] T c5c2e425a74b4e64ae70b5def750434c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:46:40.643730 25863 raft_consensus.cc:482] T 3d880de657e544468c82157769622a26 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
I20260809 15:46:40.672104 23286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a88421dba80d4d3388aa5eb62e23853b" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:40.673079 23468 leader_election.cc:398] T d3e94b7e50754a81b3d72a07476cb7b9 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 5 pre-election: Vote denied by peer 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689) with higher term. Message: Invalid argument: T d3e94b7e50754a81b3d72a07476cb7b9 P 61e69695258f48baa984c9d395e8ce3f [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c for earlier term 5. Current term is 6.
I20260809 15:46:40.697853 23468 leader_election.cc:401] T d3e94b7e50754a81b3d72a07476cb7b9 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 5 pre-election: Cancelling election due to peer responding with higher term
I20260809 15:46:40.697764 25863 raft_consensus.cc:504] T 3d880de657e544468c82157769622a26 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:40.698446 23286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cef3e7277d7d4760824a3436ed4a847d" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:40.698823 23468 leader_election.cc:302] T c0741ab44d99439abc3aa3bdd27ee160 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c; no voters:
I20260809 15:46:40.699237 23286 raft_consensus.cc:2454] T cef3e7277d7d4760824a3436ed4a847d P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 4.
I20260809 15:46:40.699736 25863 leader_election.cc:288] T 3d880de657e544468c82157769622a26 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:40.700088 23286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00a1881267a14ba8b78c81040f17a014" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:40.700253 25863 raft_consensus.cc:2726] T d3e94b7e50754a81b3d72a07476cb7b9 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: Vote denied by peer 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689) with higher term. Message: Invalid argument: T d3e94b7e50754a81b3d72a07476cb7b9 P 61e69695258f48baa984c9d395e8ce3f [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c for earlier term 5. Current term is 6.
I20260809 15:46:40.700716 23286 raft_consensus.cc:2454] T 00a1881267a14ba8b78c81040f17a014 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 3.
I20260809 15:46:40.701521 23286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ced718c1a5c4da79cb8c1ebd1435b87" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:40.702162 25746 leader_election.cc:288] T 294d4bcc1e7a477cb56455581277a60f P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:40.702638 25746 raft_consensus.cc:482] T 3924b648461e463b9d7925e5671d68ae P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:40.372049 24453 raft_consensus.cc:2454] T 907bacfa79d04a55a68c8dbe60056462 P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 3.
I20260809 15:46:40.703027 25746 raft_consensus.cc:504] T 3924b648461e463b9d7925e5671d68ae P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:40.704818 25746 leader_election.cc:288] T 3924b648461e463b9d7925e5671d68ae P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:40.619123 26335 raft_consensus.cc:482] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:40.360415 26487 consensus_queue.cc:1035] T 99bdb93503db4de4a1cfb0161ea0c07c P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.322s
I20260809 15:46:40.363041 26474 raft_consensus.cc:482] T c5c2e425a74b4e64ae70b5def750434c P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:46:40.362196 26214 raft_consensus.cc:2726] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260809 15:46:40.675194 23282 raft_consensus.cc:2454] T d0721a4403a440f383318a3cea9d1442 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 4.
I20260809 15:46:40.677426 23270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00a1881267a14ba8b78c81040f17a014" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:40.713630 23270 raft_consensus.cc:3037] T 00a1881267a14ba8b78c81040f17a014 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Advancing to term 5
I20260809 15:46:40.678272 23274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "213543cdbe814c89a17a963469922c1b" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:40.675626 23397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3be1e356b254d22b995be32f1c12f6f" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:40.674595 26450 raft_consensus.cc:2726] T 16ca412a0dae4feebe85dd01bf1a9173 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260809 15:46:40.661522 26278 raft_consensus.cc:504] T 0c29937a80724ddb9d07a27534a146f3 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:40.643632 26381 raft_consensus.cc:504] T c5c2e425a74b4e64ae70b5def750434c P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:46:40.639331 23276 rpcz_store.cc:269] 0809 15:46:38.925399 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:38.929517 (+ 4118us) service_pool.cc:227] Handling call
0809 15:46:40.283837 (+1354320us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:46:40.682436 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3be1e356b254d22b995be32f1c12f6f" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:40.483850 25899 raft_consensus.cc:482] T 6ae14a74a26f434db08c652e097385c8 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:40.683683 23400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "907bacfa79d04a55a68c8dbe60056462" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:40.735008 23400 raft_consensus.cc:2454] T 907bacfa79d04a55a68c8dbe60056462 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 3.
I20260809 15:46:40.639611 26161 raft_consensus.cc:482] T 0c48a3049897452383257aa757834fb5 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Starting leader election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
I20260809 15:46:40.636773 26469 raft_consensus.cc:729] T 0da4d5283c8a46878bd39b85e5c71f66 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Running, Role: LEADER
I20260809 15:46:40.442634 25793 leader_election.cc:288] T 2db057fd10e149a9895dff6ef8c9f038 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 4 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:40.684779 23404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3e94b7e50754a81b3d72a07476cb7b9" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:40.744421 23404 raft_consensus.cc:2354] T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c for earlier term 5. Current term is 6.
I20260809 15:46:40.742115 23528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00a1881267a14ba8b78c81040f17a014" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:40.744670 23274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3be1e356b254d22b995be32f1c12f6f" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:40.663497 26399 leader_election.cc:288] T ae6d2c08a7a94f58ac8dd698e3fd456c P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:40.671404 26326 consensus_queue.cc:260] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 61e69695258f48baa984c9d395e8ce3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:40.686893 25956 leader_election.cc:288] T 30c1740ef6874df0a332fbd6fd4b03ce P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:40.696388 26436 leader_election.cc:288] T d0721a4403a440f383318a3cea9d1442 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:40.697291 26148 leader_election.cc:288] T 0a51c0baaa6e432997c9d3552af4f254 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:40.673422 25941 raft_consensus.cc:3037] T 472816994d814cb198b69e5248697440 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:46:40.406687 26463 raft_consensus.cc:482] T f9e7cadcd24d45fcbfa84209225e8f2d P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:40.671314 26490 raft_consensus.cc:482] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:40.710920 23521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3924b648461e463b9d7925e5671d68ae" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:40.743340 26161 raft_consensus.cc:3037] T 0c48a3049897452383257aa757834fb5 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Advancing to term 3
W20260809 15:46:40.662029 26040 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.043s user 0.012s sys 0.018s
I20260809 15:46:40.749650 26473 raft_consensus.cc:482] T 297c82ed906d4b48b184985eed3de601 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:40.769551 23528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73f6f44c259547f1b05a3a434616fa54" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:40.769759 23404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3924b648461e463b9d7925e5671d68ae" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:40.491051 26118 raft_consensus.cc:504] T 563daf42d7784db195ea99c3cb21b370 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:40.723905 23527 raft_consensus.cc:2379] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 61e69695258f48baa984c9d395e8ce3f in current term 2: Already voted for candidate 65c36d164b0540f099d592f38307ee6c in this term.
I20260809 15:46:40.716085 23519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "294d4bcc1e7a477cb56455581277a60f" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:40.742120 23270 raft_consensus.cc:2454] T 00a1881267a14ba8b78c81040f17a014 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 5.
I20260809 15:46:40.742206 24410 leader_election.cc:302] T 907bacfa79d04a55a68c8dbe60056462 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:46:40.737937 23277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "907bacfa79d04a55a68c8dbe60056462" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:40.643527 26185 raft_consensus.cc:504] T df20671a7b874dbb98c0ef28371c1308 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:40.568866 26457 raft_consensus.cc:3037] T 7baa2492df0340c4a5ac241959ce1ac3 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:40.779731 23228 leader_election.cc:302] T 213543cdbe814c89a17a963469922c1b P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:40.711964 26449 raft_consensus.cc:2781] T c0741ab44d99439abc3aa3bdd27ee160 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Leader election won for term 4
I20260809 15:46:40.565182 23392 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55320 (request call id 1193) took 1656 ms. Trace:
I20260809 15:46:40.571617 23395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3296ef50be254fc29cfc77d22a8f7a23" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:40.747823 23399 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52856 (request call id 139) took 1513 ms. Trace:
I20260809 15:46:40.781030 23399 rpcz_store.cc:276] 0809 15:46:39.234713 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:39.369565 (+134852us) service_pool.cc:227] Handling call
0809 15:46:39.369796 (+ 231us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:39.922324 (+552528us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:40.747623 (+825299us) spinlock_profiling.cc:243] Waited 394 ms on lock 0x7b540001d6d4. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e9220f51 00007f34e921f99e 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772 00007f34e764a94c 00007f34e7649d86
0809 15:46:40.747774 (+ 151us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":905977088}
I20260809 15:46:40.750331 23400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3aa6a9a52804df0ba9b0ca3c82f38c3" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:40.725894 26465 raft_consensus.cc:482] T ae6d2c08a7a94f58ac8dd698e3fd456c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:40.700867 25863 consensus_queue.cc:1035] T 294d4bcc1e7a477cb56455581277a60f P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.034s
I20260809 15:46:40.715618 26474 raft_consensus.cc:3037] T c5c2e425a74b4e64ae70b5def750434c P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:46:40.752714 23345 leader_election.cc:302] T 907bacfa79d04a55a68c8dbe60056462 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:46:40.726148 26477 raft_consensus.cc:482] T 763e58d6ba644eea830a6c1664d39c9b P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:40.726012 25929 raft_consensus.cc:482] T e71a05ba1d4d42bd95c3f85f9313399e P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:46:40.792495 23273 raft_consensus.cc:2379] T df20671a7b874dbb98c0ef28371c1308 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c in current term 4: Already voted for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in this term.
W20260809 15:46:40.744215 26396 log.cc:924] Time spent T a3622f6e3e3641cea1960e3a31498bb7 P c0f38c1908414566aa9115bb2cbdd85d: Append to log took a long time: real 1.116s user 0.000s sys 0.005s
I20260809 15:46:40.770159 23528 raft_consensus.cc:2454] T 73f6f44c259547f1b05a3a434616fa54 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 3.
I20260809 15:46:40.714746 23397 raft_consensus.cc:3037] T d3be1e356b254d22b995be32f1c12f6f P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:46:40.775302 26420 raft_consensus.cc:2781] T 907bacfa79d04a55a68c8dbe60056462 P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260809 15:46:40.458132 26222 raft_consensus.cc:482] T 4b30f28311964a0b9149a50495e83d8f P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:40.773720 23270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df20671a7b874dbb98c0ef28371c1308" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:40.774120 23277 raft_consensus.cc:2454] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 3.
I20260809 15:46:40.775321 23396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "953eea268e17491ca76f3a555860a512" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:40.779976 23274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7baa2492df0340c4a5ac241959ce1ac3" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:40.746536 23280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0721a4403a440f383318a3cea9d1442" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:40.754789 24452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c48a3049897452383257aa757834fb5" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:40.768532 26326 raft_consensus.cc:3037] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Advancing to term 5
I20260809 15:46:40.787864 23399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00a1881267a14ba8b78c81040f17a014" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:40.816570 23399 raft_consensus.cc:2379] T 00a1881267a14ba8b78c81040f17a014 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c in current term 5: Already voted for candidate c0f38c1908414566aa9115bb2cbdd85d in this term.
I20260809 15:46:40.816866 24475 raft_consensus.cc:1226] T d3be1e356b254d22b995be32f1c12f6f P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Rejecting Update request from peer 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 3. Current term is 4. Ops: []
I20260809 15:46:40.756501 26053 raft_consensus.cc:482] T a3622f6e3e3641cea1960e3a31498bb7 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:40.770833 26399 raft_consensus.cc:482] T 7baa2492df0340c4a5ac241959ce1ac3 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:40.793161 23228 leader_election.cc:302] T 538a228ddfb54bc6a36528d5ff07597b P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:40.777637 23520 raft_consensus.cc:2354] T 00a1881267a14ba8b78c81040f17a014 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 4. Current term is 5.
I20260809 15:46:40.711952 26335 raft_consensus.cc:504] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:40.748672 25899 raft_consensus.cc:504] T 6ae14a74a26f434db08c652e097385c8 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:40.827976 23228 leader_election.cc:302] T d1a134ba33ea4076bbaac0ae460c17ba P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c
I20260809 15:46:40.828715 26335 leader_election.cc:288] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:40.771848 23404 raft_consensus.cc:2454] T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 5.
I20260809 15:46:40.775919 26143 raft_consensus.cc:482] T 0c29937a80724ddb9d07a27534a146f3 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:40.829005 23228 leader_election.cc:302] T 2424fd9bd7f14e508fd106cbfda786a4 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:40.829802 26436 raft_consensus.cc:2781] T d1a134ba33ea4076bbaac0ae460c17ba P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260809 15:46:40.830083 23228 leader_election.cc:302] T ba56725e42f74a1baa4c04d9660727f9 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:40.778744 23276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cef3e7277d7d4760824a3436ed4a847d" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:40.771006 26463 raft_consensus.cc:504] T f9e7cadcd24d45fcbfa84209225e8f2d P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:40.830758 23276 raft_consensus.cc:3037] T cef3e7277d7d4760824a3436ed4a847d P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Advancing to term 5
I20260809 15:46:40.831854 25639 leader_election.cc:288] T 00a1881267a14ba8b78c81040f17a014 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 5 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:40.818055 26222 raft_consensus.cc:504] T 4b30f28311964a0b9149a50495e83d8f P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:40.835372 26464 raft_consensus.cc:2726] T 538a228ddfb54bc6a36528d5ff07597b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260809 15:46:40.835795 26275 raft_consensus.cc:504] T c5c2e425a74b4e64ae70b5def750434c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:40.798085 25929 raft_consensus.cc:3037] T e71a05ba1d4d42bd95c3f85f9313399e P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:46:40.837643 26275 leader_election.cc:288] T c5c2e425a74b4e64ae70b5def750434c P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:40.838637 26053 raft_consensus.cc:504] T a3622f6e3e3641cea1960e3a31498bb7 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:40.836432 26222 leader_election.cc:288] T 4b30f28311964a0b9149a50495e83d8f P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:40.839612 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5712f6b6fffc4cc5a8ddfbe63e4097df" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:40.821550 23395 raft_consensus.cc:2396] T 3296ef50be254fc29cfc77d22a8f7a23 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 for term 3 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260809 15:46:40.841113 26381 leader_election.cc:288] T c5c2e425a74b4e64ae70b5def750434c P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:40.841324 23395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae6d2c08a7a94f58ac8dd698e3fd456c" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:40.806054 26136 raft_consensus.cc:482] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
I20260809 15:46:40.842181 26185 leader_election.cc:288] T df20671a7b874dbb98c0ef28371c1308 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:40.842485 23395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99bdb93503db4de4a1cfb0161ea0c07c" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
W20260809 15:46:40.809711 24463 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38724 (request call id 444) took 1668 ms (client timeout 2178 ms). Trace:
I20260809 15:46:40.776106 26293 consensus_queue.cc:260] T 7660dfa2ef694927b8eda4100a2bcd9b P 61e69695258f48baa984c9d395e8ce3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:40.843977 26293 raft_consensus.cc:3037] T 7660dfa2ef694927b8eda4100a2bcd9b P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Advancing to term 5
I20260809 15:46:40.846944 23404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07fcabb569414bddbd39df6cedd24e64" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:40.797966 26477 raft_consensus.cc:504] T 763e58d6ba644eea830a6c1664d39c9b P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:46:40.847354 26286 consensus_queue.cc:1159] T 7660dfa2ef694927b8eda4100a2bcd9b P 61e69695258f48baa984c9d395e8ce3f [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "c0f38c1908414566aa9115bb2cbdd85d" responder_term: 5 status { last_received { term: 5 index: 7 } last_committed_idx: 6 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 61e69695258f48baa984c9d395e8ce3f for earlier term 4. Current term is 5. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260809 15:46:40.847973 24410 leader_election.cc:302] T 3924b648461e463b9d7925e5671d68ae P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c
I20260809 15:46:40.848835 26477 leader_election.cc:288] T 763e58d6ba644eea830a6c1664d39c9b P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
W20260809 15:46:40.809293 26383 log.cc:924] Time spent T 5ad8a3b25d7b49e5a2142d5709edd0c7 P c0f38c1908414566aa9115bb2cbdd85d: Append to log took a long time: real 1.351s user 0.011s sys 0.003s
I20260809 15:46:40.787196 23392 rpcz_store.cc:276] 0809 15:46:38.908662 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:39.248925 (+340263us) service_pool.cc:227] Handling call
0809 15:46:39.249250 (+ 325us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 15:46:39.249598 (+ 348us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:46:39.249630 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 15:46:39.250056 (+ 426us) log.cc:842] Serialized 26 byte log entry
0809 15:46:40.020661 (+770605us) raft_consensus.cc:1631] Marking committed up to 1
0809 15:46:40.026740 (+ 6079us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:40.030250 (+ 3510us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 15:46:40.412981 (+382731us) raft_consensus.cc:1671] finished
0809 15:46:40.441966 (+ 28985us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:40.532820 (+ 90854us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":4058,"raft.run_cpu_time_us":105,"raft.run_wall_time_us":5265,"thread_start_us":761411,"threads_started":1,"wal-append.queue_time_us":786695}
I20260809 15:46:40.814335 23396 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55332 (request call id 1250) took 1230 ms. Trace:
I20260809 15:46:40.817256 23277 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0a51c0baaa6e432997c9d3552af4f254"
dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
mode: GRACEFUL
new_leader_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
from {username='slave'} at 127.0.0.1:33468
I20260809 15:46:40.789752 23400 raft_consensus.cc:2454] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 2.
I20260809 15:46:40.806237 23472 leader_election.cc:302] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:40.795042 26442 raft_consensus.cc:2781] T 907bacfa79d04a55a68c8dbe60056462 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260809 15:46:40.792948 26487 raft_consensus.cc:2726] T 213543cdbe814c89a17a963469922c1b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260809 15:46:40.772709 26490 raft_consensus.cc:504] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:40.782877 26469 consensus_queue.cc:260] T 0da4d5283c8a46878bd39b85e5c71f66 P 67b3aeadfa7a432abe8af4d292e04ab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:40.803907 26420 raft_consensus.cc:482] T 907bacfa79d04a55a68c8dbe60056462 P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:46:40.804579 26449 raft_consensus.cc:686] T c0741ab44d99439abc3aa3bdd27ee160 P 65c36d164b0540f099d592f38307ee6c [term 4 LEADER]: Becoming Leader. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:46:40.795244 26465 raft_consensus.cc:504] T ae6d2c08a7a94f58ac8dd698e3fd456c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:40.821574 26502 raft_consensus.cc:482] T f7715d71dbac447b9351fc5eb36c5e80 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:40.742333 26485 consensus_queue.cc:260] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 67b3aeadfa7a432abe8af4d292e04ab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:40.822341 23399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2db057fd10e149a9895dff6ef8c9f038" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:40.793123 24455 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a51c0baaa6e432997c9d3552af4f254" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
W20260809 15:46:40.792948 23470 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 53163.5us
I20260809 15:46:40.823124 23397 raft_consensus.cc:2454] T d3be1e356b254d22b995be32f1c12f6f P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 4.
I20260809 15:46:40.830142 26436 raft_consensus.cc:482] T d1a134ba33ea4076bbaac0ae460c17ba P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:40.808543 23519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "213543cdbe814c89a17a963469922c1b" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
W20260809 15:46:40.768117 25746 consensus_queue.cc:1159] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 61e69695258f48baa984c9d395e8ce3f [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "65c36d164b0540f099d592f38307ee6c" responder_term: 5 status { last_received { term: 1 index: 4 } last_committed_idx: 4 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 61e69695258f48baa984c9d395e8ce3f for earlier term 4. Current term is 5. Ops: [4.5-4.5]" } } last_received_current_leader { term: 0 index: 0 } }
I20260809 15:46:40.815598 26488 raft_consensus.cc:482] T 2db057fd10e149a9895dff6ef8c9f038 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:40.817483 23353 leader_election.cc:302] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters: 61e69695258f48baa984c9d395e8ce3f
I20260809 15:46:40.779338 26473 raft_consensus.cc:504] T 297c82ed906d4b48b184985eed3de601 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:40.850664 23404 raft_consensus.cc:3037] T 07fcabb569414bddbd39df6cedd24e64 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Advancing to term 5
W20260809 15:46:41.100492 23472 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 294311us
I20260809 15:46:41.101367 23472 leader_election.cc:302] T 28e17dcb5efe45b8b6a8c42da03b4819 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:40.923897 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cef3e7277d7d4760824a3436ed4a847d" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:41.126207 23526 raft_consensus.cc:3037] T cef3e7277d7d4760824a3436ed4a847d P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Advancing to term 5
I20260809 15:46:41.127242 23277 raft_consensus.cc:595] T 0a51c0baaa6e432997c9d3552af4f254 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Received request to transfer leadership to TS c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:41.127785 23277 raft_consensus.cc:603] T 0a51c0baaa6e432997c9d3552af4f254 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260809 15:46:41.134264 23228 leader_election.cc:302] T 4564dfc6d4094c1b9240b367ccecf3b1 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c
I20260809 15:46:41.135303 24410 leader_election.cc:302] T d3be1e356b254d22b995be32f1c12f6f P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d; no voters:
W20260809 15:46:41.137324 23353 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 322423us
I20260809 15:46:40.883056 23528 raft_consensus.cc:1226] T 28e17dcb5efe45b8b6a8c42da03b4819 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Rejecting Update request from peer c0f38c1908414566aa9115bb2cbdd85d for earlier term 3. Current term is 4. Ops: []
I20260809 15:46:41.124225 23396 rpcz_store.cc:276] 0809 15:46:39.583956 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:39.889675 (+305719us) service_pool.cc:227] Handling call
0809 15:46:40.775154 (+885479us) spinlock_profiling.cc:243] Waited 25.9 ms on lock 0x7b58001d7c98. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e613cf11 00007f34e62c0851 00007f34e62c0702 00007f34ef1b0b09 00007f34ee0ec730 00007f34ee1b279b 00007f34ee1b2821 00007f34ee1621b0 00007f34ee161a94 00007f34e7849c04 00007f34e7849b92
0809 15:46:40.814283 (+ 39129us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":59506432}
I20260809 15:46:41.099181 23399 raft_consensus.cc:3037] T 2db057fd10e149a9895dff6ef8c9f038 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:46:41.141078 23470 leader_election.cc:302] T 297c82ed906d4b48b184985eed3de601 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:46:41.144461 23228 leader_election.cc:302] T 28e17dcb5efe45b8b6a8c42da03b4819 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:41.145434 26335 raft_consensus.cc:2781] T d3be1e356b254d22b995be32f1c12f6f P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Leader election won for term 4
I20260809 15:46:41.146083 26335 raft_consensus.cc:686] T d3be1e356b254d22b995be32f1c12f6f P 61e69695258f48baa984c9d395e8ce3f [term 4 LEADER]: Becoming Leader. State: Replica: 61e69695258f48baa984c9d395e8ce3f, State: Running, Role: LEADER
I20260809 15:46:41.146987 26335 consensus_queue.cc:237] T d3be1e356b254d22b995be32f1c12f6f P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:41.150846 26469 raft_consensus.cc:3037] T 0da4d5283c8a46878bd39b85e5c71f66 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:46:41.151126 26512 raft_consensus.cc:482] T 763e58d6ba644eea830a6c1664d39c9b P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:41.151605 26512 raft_consensus.cc:504] T 763e58d6ba644eea830a6c1664d39c9b P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:41.153088 25929 raft_consensus.cc:504] T e71a05ba1d4d42bd95c3f85f9313399e P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:41.153579 26512 leader_election.cc:288] T 763e58d6ba644eea830a6c1664d39c9b P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:41.153993 26488 raft_consensus.cc:504] T 2db057fd10e149a9895dff6ef8c9f038 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:41.155381 26381 raft_consensus.cc:2726] T 28e17dcb5efe45b8b6a8c42da03b4819 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260809 15:46:41.156440 26488 leader_election.cc:288] T 2db057fd10e149a9895dff6ef8c9f038 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:41.157027 24451 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38722 (request call id 385) took 1537 ms. Trace:
I20260809 15:46:41.157536 24451 rpcz_store.cc:276] 0809 15:46:39.620001 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:39.622293 (+ 2292us) service_pool.cc:227] Handling call
0809 15:46:41.156794 (+1534501us) spinlock_profiling.cc:243] Waited 193 ms on lock 0x7b540047ead4. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e92242ae 00007f34ee161c27 00007f34e7849c04 00007f34e7849b92 00007f34e7849ac1 00007f34e7849a3c 00007f34e7848cf2 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:46:41.156961 (+ 167us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":443458304}
I20260809 15:46:40.835698 26118 leader_election.cc:288] T 563daf42d7784db195ea99c3cb21b370 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:41.158758 24451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d880de657e544468c82157769622a26" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:41.158922 26179 raft_consensus.cc:2781] T 3924b648461e463b9d7925e5671d68ae P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260809 15:46:41.162416 23270 raft_consensus.cc:1261] T ae6d2c08a7a94f58ac8dd698e3fd456c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260809 15:46:41.165786 23400 raft_consensus.cc:1226] T 0da4d5283c8a46878bd39b85e5c71f66 P c0f38c1908414566aa9115bb2cbdd85d [term 4 LEADER]: Rejecting Update request from peer 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260809 15:46:41.166054 23470 leader_election.cc:302] T 998c6cee72cf40228c6dba064c0db2da P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:46:41.166321 23353 leader_election.cc:302] T 73f6f44c259547f1b05a3a434616fa54 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:46:41.165284 23397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2d4f749fcfd4fb3bfb8115af3471598" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:41.168449 24472 raft_consensus.cc:1261] T 7660dfa2ef694927b8eda4100a2bcd9b P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260809 15:46:41.155026 25929 leader_election.cc:288] T e71a05ba1d4d42bd95c3f85f9313399e P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:41.171582 23396 raft_consensus.cc:1226] T 7baa2492df0340c4a5ac241959ce1ac3 P c0f38c1908414566aa9115bb2cbdd85d [term 2 LEADER]: Rejecting Update request from peer 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 1. Current term is 2. Ops: []
I20260809 15:46:40.843375 26278 leader_election.cc:288] T 0c29937a80724ddb9d07a27534a146f3 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:40.835978 26464 raft_consensus.cc:2726] T 2424fd9bd7f14e508fd106cbfda786a4 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260809 15:46:41.091763 25941 raft_consensus.cc:2726] T 472816994d814cb198b69e5248697440 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260809 15:46:41.105953 26477 raft_consensus.cc:2726] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260809 15:46:41.179198 26490 leader_election.cc:288] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:41.170349 24452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "563daf42d7784db195ea99c3cb21b370" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f"
I20260809 15:46:41.152186 25816 raft_consensus.cc:504] T f9e7cadcd24d45fcbfa84209225e8f2d P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:41.169121 23397 raft_consensus.cc:2396] T d2d4f749fcfd4fb3bfb8115af3471598 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 for term 5 because replica has last-logged OpId of term: 4 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260809 15:46:41.172170 23396 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55332 (request call id 1242) took 1597 ms. Trace:
W20260809 15:46:40.915699 26429 log.cc:924] Time spent T 294d4bcc1e7a477cb56455581277a60f P 67b3aeadfa7a432abe8af4d292e04ab9: Append to log took a long time: real 0.893s user 0.003s sys 0.000s
I20260809 15:46:41.169569 23400 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55332 (request call id 1241) took 1595 ms. Trace:
I20260809 15:46:41.189323 23400 rpcz_store.cc:276] 0809 15:46:39.574259 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:41.165119 (+1590860us) service_pool.cc:227] Handling call
0809 15:46:41.165563 (+ 444us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 15:46:41.169336 (+ 3773us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:41.169516 (+ 180us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:46:41.168967 23223 leader_election.cc:302] T cee02fff2c9c48a19f43eb358a89e7b0 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c
I20260809 15:46:40.837581 26463 leader_election.cc:288] T f9e7cadcd24d45fcbfa84209225e8f2d P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:41.109254 26436 raft_consensus.cc:3037] T d1a134ba33ea4076bbaac0ae460c17ba P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:40.829036 25899 leader_election.cc:288] T 6ae14a74a26f434db08c652e097385c8 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:41.109753 26442 raft_consensus.cc:482] T 907bacfa79d04a55a68c8dbe60056462 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:46:41.176491 23519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30c1740ef6874df0a332fbd6fd4b03ce" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
W20260809 15:46:41.170873 26503 log.cc:924] Time spent T 7660dfa2ef694927b8eda4100a2bcd9b P 65c36d164b0540f099d592f38307ee6c: Append to log took a long time: real 0.109s user 0.000s sys 0.000s
I20260809 15:46:40.855003 26040 raft_consensus.cc:2726] T ba56725e42f74a1baa4c04d9660727f9 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260809 15:46:41.165613 26502 raft_consensus.cc:504] T f7715d71dbac447b9351fc5eb36c5e80 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:41.177690 23228 leader_election.cc:398] T 00a1881267a14ba8b78c81040f17a014 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 pre-election: Vote denied by peer c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263) with higher term. Message: Invalid argument: T 00a1881267a14ba8b78c81040f17a014 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 4. Current term is 5.
I20260809 15:46:41.208616 23228 leader_election.cc:401] T 00a1881267a14ba8b78c81040f17a014 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
I20260809 15:46:40.843168 23395 raft_consensus.cc:2454] T 99bdb93503db4de4a1cfb0161ea0c07c P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 2.
I20260809 15:46:41.153787 26473 leader_election.cc:288] T 297c82ed906d4b48b184985eed3de601 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:41.177886 24455 raft_consensus.cc:2379] T 0a51c0baaa6e432997c9d3552af4f254 P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d in current term 3: Already voted for candidate 61e69695258f48baa984c9d395e8ce3f in this term.
W20260809 15:46:41.178143 26377 log.cc:924] Time spent T d1a134ba33ea4076bbaac0ae460c17ba P 67b3aeadfa7a432abe8af4d292e04ab9: Append to log took a long time: real 1.751s user 0.012s sys 0.012s
I20260809 15:46:41.210742 26464 raft_consensus.cc:2726] T 00a1881267a14ba8b78c81040f17a014 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263) with higher term. Message: Invalid argument: T 00a1881267a14ba8b78c81040f17a014 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 4. Current term is 5.
I20260809 15:46:41.210843 23228 leader_election.cc:302] T 05ad7d8fdca64e3da527280ca5be9280 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c
I20260809 15:46:41.178648 26512 raft_consensus.cc:2781] T 73f6f44c259547f1b05a3a434616fa54 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260809 15:46:41.180929 23528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ae14a74a26f434db08c652e097385c8" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:41.145416 24410 leader_election.cc:302] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:46:40.883055 26053 leader_election.cc:288] T a3622f6e3e3641cea1960e3a31498bb7 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:41.142035 26420 raft_consensus.cc:3037] T 907bacfa79d04a55a68c8dbe60056462 P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:46:41.164011 26148 raft_consensus.cc:2781] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Leader election won for term 5
I20260809 15:46:41.184031 23396 rpcz_store.cc:276] 0809 15:46:39.574514 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:41.167107 (+1592593us) service_pool.cc:227] Handling call
0809 15:46:41.171432 (+ 4325us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:41.171963 (+ 531us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:41.172124 (+ 161us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:46:41.184257 23277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30c1740ef6874df0a332fbd6fd4b03ce" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:41.167888 26507 raft_consensus.cc:2781] T 4564dfc6d4094c1b9240b367ccecf3b1 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260809 15:46:41.213744 26512 raft_consensus.cc:482] T 73f6f44c259547f1b05a3a434616fa54 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:46:41.188812 23397 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55332 (request call id 1249) took 1608 ms. Trace:
I20260809 15:46:41.217343 26512 raft_consensus.cc:3037] T 73f6f44c259547f1b05a3a434616fa54 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Advancing to term 5
I20260809 15:46:41.217361 23397 rpcz_store.cc:276] 0809 15:46:39.580011 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:41.165017 (+1585006us) service_pool.cc:227] Handling call
0809 15:46:41.188768 (+ 23751us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:46:41.218461 23395 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52856 (request call id 142) took 1114 ms. Trace:
I20260809 15:46:41.218809 23395 rpcz_store.cc:276] 0809 15:46:40.104411 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:41.217698 (+1113287us) service_pool.cc:227] Handling call
0809 15:46:41.217993 (+ 295us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:41.218155 (+ 162us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:46:41.218190 (+ 35us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 15:46:41.218227 (+ 37us) raft_consensus.cc:1631] Marking committed up to 6
0809 15:46:41.218258 (+ 31us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:41.218305 (+ 47us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:41.218409 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:46:41.220382 23395 pending_rounds.cc:76] T 6ae14a74a26f434db08c652e097385c8 P c0f38c1908414566aa9115bb2cbdd85d: Aborting all ops after (but not including) 1
I20260809 15:46:41.220161 26161 raft_consensus.cc:504] T 0c48a3049897452383257aa757834fb5 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:41.219228 23397 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55320 (request call id 1206) took 1008 ms. Trace:
I20260809 15:46:41.221284 23397 rpcz_store.cc:276] 0809 15:46:40.211089 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:41.218200 (+1007111us) service_pool.cc:227] Handling call
0809 15:46:41.218728 (+ 528us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:41.218906 (+ 178us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:46:41.218928 (+ 22us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 15:46:41.218978 (+ 50us) raft_consensus.cc:1631] Marking committed up to 3
0809 15:46:41.219049 (+ 71us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:41.219098 (+ 49us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:41.219185 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:46:41.222410 23277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ae14a74a26f434db08c652e097385c8" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:40.846874 25885 consensus_queue.cc:1046] T 0acc47ce21c3422698d1ef7277c3ed63 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.329s
I20260809 15:46:41.192730 23400 raft_consensus.cc:1226] T 6ced718c1a5c4da79cb8c1ebd1435b87 P c0f38c1908414566aa9115bb2cbdd85d [term 3 LEADER]: Rejecting Update request from peer 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 2. Current term is 3. Ops: []
I20260809 15:46:41.193504 26490 raft_consensus.cc:2781] T cee02fff2c9c48a19f43eb358a89e7b0 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260809 15:46:41.172434 26381 raft_consensus.cc:2781] T 297c82ed906d4b48b184985eed3de601 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260809 15:46:41.198071 23472 leader_election.cc:302] T df20671a7b874dbb98c0ef28371c1308 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:46:41.176488 25863 raft_consensus.cc:2781] T 998c6cee72cf40228c6dba064c0db2da P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260809 15:46:41.179529 26438 log.cc:924] Time spent T 05ad7d8fdca64e3da527280ca5be9280 P 65c36d164b0540f099d592f38307ee6c: Append to log took a long time: real 0.891s user 0.003s sys 0.000s
I20260809 15:46:40.829339 26143 raft_consensus.cc:504] T 0c29937a80724ddb9d07a27534a146f3 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:41.198882 23526 raft_consensus.cc:2454] T cef3e7277d7d4760824a3436ed4a847d P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 5.
W20260809 15:46:40.843467 24463 rpcz_store.cc:269] 0809 15:46:39.141346 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:39.168464 (+ 27118us) service_pool.cc:227] Handling call
0809 15:46:40.809521 (+1641057us) spinlock_profiling.cc:243] Waited 1.08 s on lock 0x7b54004bd7d4. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e92242ae 00007f34ee161c27 00007f34e7849c04 00007f34e7849b92 00007f34e7849ac1 00007f34e7849a3c 00007f34e7848cf2 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:46:40.809656 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2472839424}
I20260809 15:46:41.181824 25929 raft_consensus.cc:482] T 563daf42d7784db195ea99c3cb21b370 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:40.842514 26136 raft_consensus.cc:504] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:41.203423 26477 consensus_queue.cc:1035] T 3924b648461e463b9d7925e5671d68ae P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.024s
I20260809 15:46:41.204770 23519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b30f28311964a0b9149a50495e83d8f" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:41.175993 26293 raft_consensus.cc:482] T 16ca412a0dae4feebe85dd01bf1a9173 P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:41.180287 26214 consensus_queue.cc:1035] T d1a134ba33ea4076bbaac0ae460c17ba P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.011s
I20260809 15:46:41.225684 26279 raft_consensus.cc:2781] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 61e69695258f48baa984c9d395e8ce3f [term 2 FOLLOWER]: Leader election won for term 2
I20260809 15:46:41.209053 23470 leader_election.cc:302] T 07fcabb569414bddbd39df6cedd24e64 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:46:41.184656 26487 raft_consensus.cc:2726] T 28e17dcb5efe45b8b6a8c42da03b4819 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260809 15:46:40.849203 26399 raft_consensus.cc:504] T 7baa2492df0340c4a5ac241959ce1ac3 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:46:41.213980 24410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 68634.5us
I20260809 15:46:41.215669 23228 leader_election.cc:302] T d3be1e356b254d22b995be32f1c12f6f P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d
W20260809 15:46:41.272425 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 56832.2us
I20260809 15:46:41.273062 23228 leader_election.cc:302] T 3296ef50be254fc29cfc77d22a8f7a23 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:41.216459 23393 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55332 (request call id 1255) took 1554 ms. Trace:
I20260809 15:46:41.216497 23528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3622f6e3e3641cea1960e3a31498bb7" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:41.188097 23351 leader_election.cc:302] T d2d4f749fcfd4fb3bfb8115af3471598 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
W20260809 15:46:41.281204 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 93188.7us
I20260809 15:46:41.281934 23351 leader_election.cc:302] T a88421dba80d4d3388aa5eb62e23853b P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:41.220620 26502 leader_election.cc:288] T f7715d71dbac447b9351fc5eb36c5e80 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:41.223096 23277 raft_consensus.cc:2396] T 6ae14a74a26f434db08c652e097385c8 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d for term 5 because replica has last-logged OpId of term: 4 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260809 15:46:41.224403 23400 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55332 (request call id 1253) took 1597 ms. Trace:
I20260809 15:46:41.284035 23400 rpcz_store.cc:276] 0809 15:46:39.626671 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:41.190069 (+1563398us) service_pool.cc:227] Handling call
0809 15:46:41.192557 (+ 2488us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:41.224237 (+ 31680us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:41.224364 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:46:41.284716 23351 leader_election.cc:302] T 213543cdbe814c89a17a963469922c1b P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:41.295480 23228 leader_election.cc:302] T 953eea268e17491ca76f3a555860a512 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:41.208016 26442 raft_consensus.cc:3037] T 907bacfa79d04a55a68c8dbe60056462 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:46:41.220678 23395 pending_rounds.cc:97] T 6ae14a74a26f434db08c652e097385c8 P c0f38c1908414566aa9115bb2cbdd85d: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20260809 15:46:41.308269 26457 raft_consensus.cc:2726] T d3be1e356b254d22b995be32f1c12f6f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 LEADER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260809 15:46:41.308566 23395 raft_consensus.cc:2866] T 6ae14a74a26f434db08c652e097385c8 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 15:46:41.308903 26502 raft_consensus.cc:2726] T 3296ef50be254fc29cfc77d22a8f7a23 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 15:46:41.309834 26457 raft_consensus.cc:2726] T 953eea268e17491ca76f3a555860a512 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260809 15:46:41.311232 23521 raft_consensus.cc:1261] T 7baa2492df0340c4a5ac241959ce1ac3 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 15:46:41.293708 26510 raft_consensus.cc:2726] T d2d4f749fcfd4fb3bfb8115af3471598 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260809 15:46:41.280422 23393 rpcz_store.cc:276] 0809 15:46:39.661799 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:40.213233 (+551434us) service_pool.cc:227] Handling call
0809 15:46:41.216258 (+1003025us) spinlock_profiling.cc:243] Waited 20.4 ms on lock 0x7b54003746d4. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e92242ae 00007f34ee161c27 00007f34e7849c04 00007f34e7849b92 00007f34e7849ac1 00007f34e7849a3c 00007f34e7848cf2 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:46:41.216377 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":47023360}
I20260809 15:46:41.313308 23228 leader_election.cc:302] T d2d4f749fcfd4fb3bfb8115af3471598 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:41.314354 26450 raft_consensus.cc:2726] T d2d4f749fcfd4fb3bfb8115af3471598 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260809 15:46:41.280071 23399 raft_consensus.cc:2454] T 2db057fd10e149a9895dff6ef8c9f038 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 4.
I20260809 15:46:41.312055 26399 leader_election.cc:288] T 7baa2492df0340c4a5ac241959ce1ac3 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:41.227607 23396 raft_consensus.cc:1226] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P c0f38c1908414566aa9115bb2cbdd85d [term 3 LEADER]: Rejecting Update request from peer 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 2. Current term is 3. Ops: [2.5-2.5]
I20260809 15:46:41.228340 23472 leader_election.cc:302] T 4bcac4b24e7a4448abdc934bba5a624e P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:41.217271 26507 raft_consensus.cc:482] T 4564dfc6d4094c1b9240b367ccecf3b1 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:41.230429 23283 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53554 (request call id 1264) took 2397 ms. Trace:
I20260809 15:46:41.212136 25899 raft_consensus.cc:482] T 4bcac4b24e7a4448abdc934bba5a624e P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
I20260809 15:46:41.209559 26474 raft_consensus.cc:504] T c5c2e425a74b4e64ae70b5def750434c P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:41.232831 23518 raft_consensus.cc:1226] T 0c48a3049897452383257aa757834fb5 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Rejecting Update request from peer 61e69695258f48baa984c9d395e8ce3f for earlier term 2. Current term is 3. Ops: []
I20260809 15:46:41.227341 26381 raft_consensus.cc:482] T 297c82ed906d4b48b184985eed3de601 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:46:41.161835 26449 consensus_queue.cc:237] T c0741ab44d99439abc3aa3bdd27ee160 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:41.229396 25863 raft_consensus.cc:482] T 998c6cee72cf40228c6dba064c0db2da P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:46:41.225721 26490 raft_consensus.cc:482] T cee02fff2c9c48a19f43eb358a89e7b0 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:41.229708 26148 raft_consensus.cc:686] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d [term 5 LEADER]: Becoming Leader. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:46:41.343550 26507 raft_consensus.cc:3037] T 4564dfc6d4094c1b9240b367ccecf3b1 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:46:41.221618 26464 raft_consensus.cc:2781] T 05ad7d8fdca64e3da527280ca5be9280 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260809 15:46:41.240976 23270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a51c0baaa6e432997c9d3552af4f254" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:41.241040 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0721a4403a440f383318a3cea9d1442" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:41.358378 23526 raft_consensus.cc:3037] T d0721a4403a440f383318a3cea9d1442 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Advancing to term 5
I20260809 15:46:41.237012 23282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "563daf42d7784db195ea99c3cb21b370" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:41.243191 23274 raft_consensus.cc:2454] T 7baa2492df0340c4a5ac241959ce1ac3 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 2.
I20260809 15:46:41.249297 23285 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53554 (request call id 1270) took 1996 ms. Trace:
I20260809 15:46:41.250824 26279 raft_consensus.cc:686] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 61e69695258f48baa984c9d395e8ce3f [term 2 LEADER]: Becoming Leader. State: Replica: 61e69695258f48baa984c9d395e8ce3f, State: Running, Role: LEADER
I20260809 15:46:41.251024 24463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad8a3b25d7b49e5a2142d5709edd0c7" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:41.235204 25816 leader_election.cc:288] T f9e7cadcd24d45fcbfa84209225e8f2d P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 4 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:41.253446 26463 consensus_queue.cc:1035] T 7660dfa2ef694927b8eda4100a2bcd9b P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260809 15:46:41.375973 26279 consensus_queue.cc:237] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:41.376688 24475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e71a05ba1d4d42bd95c3f85f9313399e" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f"
I20260809 15:46:41.377276 24475 raft_consensus.cc:3037] T e71a05ba1d4d42bd95c3f85f9313399e P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:46:41.253865 23519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c29937a80724ddb9d07a27534a146f3" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:41.259006 26228 raft_consensus.cc:2781] T 07fcabb569414bddbd39df6cedd24e64 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Leader election won for term 5
I20260809 15:46:41.380649 23519 raft_consensus.cc:2454] T 0c29937a80724ddb9d07a27534a146f3 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 4.
I20260809 15:46:41.380867 26228 raft_consensus.cc:686] T 07fcabb569414bddbd39df6cedd24e64 P 65c36d164b0540f099d592f38307ee6c [term 5 LEADER]: Becoming Leader. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:46:41.381611 26148 consensus_queue.cc:237] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:41.381850 26228 consensus_queue.cc:237] T 07fcabb569414bddbd39df6cedd24e64 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:41.249595 26293 raft_consensus.cc:504] T 16ca412a0dae4feebe85dd01bf1a9173 P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:41.259963 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472816994d814cb198b69e5248697440" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:41.243700 23276 raft_consensus.cc:2454] T cef3e7277d7d4760824a3436ed4a847d P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 5.
I20260809 15:46:41.230760 25941 raft_consensus.cc:2781] T df20671a7b874dbb98c0ef28371c1308 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Leader election won for term 4
I20260809 15:46:41.264981 23404 raft_consensus.cc:2454] T 07fcabb569414bddbd39df6cedd24e64 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 5.
I20260809 15:46:41.388557 23404 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55320 (request call id 1218) took 1147 ms. Trace:
I20260809 15:46:41.388882 23404 rpcz_store.cc:276] 0809 15:46:40.240852 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:40.846711 (+605859us) service_pool.cc:227] Handling call
0809 15:46:41.388505 (+541794us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:46:41.235333 26136 leader_election.cc:288] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:41.259939 24452 raft_consensus.cc:2454] T 563daf42d7784db195ea99c3cb21b370 P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 3.
I20260809 15:46:41.265818 26143 leader_election.cc:288] T 0c29937a80724ddb9d07a27534a146f3 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
W20260809 15:46:41.266861 26519 log.cc:924] Time spent T 3924b648461e463b9d7925e5671d68ae P 61e69695258f48baa984c9d395e8ce3f: Append to log took a long time: real 0.086s user 0.000s sys 0.001s
I20260809 15:46:41.235970 26179 raft_consensus.cc:482] T 3924b648461e463b9d7925e5671d68ae P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Starting leader election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:41.268000 26405 raft_consensus.cc:3037] T 472816994d814cb198b69e5248697440 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:46:41.268289 25931 leader_election.cc:288] T 99bdb93503db4de4a1cfb0161ea0c07c P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
W20260809 15:46:41.269652 23520 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56770 (request call id 1221) took 2630 ms (client timeout 2063 ms). Trace:
I20260809 15:46:41.307687 23528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9e7cadcd24d45fcbfa84209225e8f2d" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:41.347978 25899 raft_consensus.cc:504] T 4bcac4b24e7a4448abdc934bba5a624e P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:41.321112 23396 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55332 (request call id 1256) took 1637 ms. Trace:
W20260809 15:46:41.329273 23472 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 101004us
W20260809 15:46:41.342283 23273 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53554 (request call id 1274) took 1720 ms (client timeout 1842 ms). Trace:
W20260809 15:46:41.347663 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 63035us
I20260809 15:46:41.327693 23283 rpcz_store.cc:276] 0809 15:46:38.830031 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:38.837397 (+ 7366us) service_pool.cc:227] Handling call
0809 15:46:38.853654 (+ 16257us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 15:46:38.895475 (+ 41821us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:46:38.895506 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 15:46:38.908703 (+ 13197us) log.cc:842] Serialized 26 byte log entry
0809 15:46:39.184229 (+275526us) raft_consensus.cc:1631] Marking committed up to 1
0809 15:46:39.184266 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:39.184330 (+ 64us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 15:46:41.175961 (+1991631us) spinlock_profiling.cc:243] Waited 1.77 ms on lock 0x7b44000232cc. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e613cf11 00007f34e620aa95 00007f34e620b4b2 00007f34e8eb3511 00007f34e8eaee1a 00007f34e8ea2bee 00007f34e8ea1f36 00007f34e8ea1b58 00007f34e8eab551 00007f34e8eab509 00007f34e8eab499
0809 15:46:41.211381 (+ 35420us) raft_consensus.cc:1671] finished
0809 15:46:41.211400 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:41.226964 (+ 15564us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6947,"raft.queue_time_us":17371,"raft.run_cpu_time_us":81,"raft.run_wall_time_us":81,"spinlock_wait_cycles":4069120,"thread_start_us":227576,"threads_started":1,"wal-append.queue_time_us":429495}
I20260809 15:46:41.427587 23472 leader_election.cc:302] T 73f6f44c259547f1b05a3a434616fa54 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters: 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:41.328991 26144 raft_consensus.cc:2726] T a88421dba80d4d3388aa5eb62e23853b P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260809 15:46:41.346817 25639 raft_consensus.cc:2726] T 4bcac4b24e7a4448abdc934bba5a624e P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 15:46:41.360841 26464 raft_consensus.cc:482] T 05ad7d8fdca64e3da527280ca5be9280 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:41.359886 23285 rpcz_store.cc:276] 0809 15:46:39.252550 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:39.252691 (+ 141us) service_pool.cc:227] Handling call
0809 15:46:39.281724 (+ 29033us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 15:46:39.327085 (+ 45361us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:46:39.327119 (+ 34us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 15:46:39.365336 (+ 38217us) log.cc:842] Serialized 26 byte log entry
0809 15:46:39.715175 (+349839us) raft_consensus.cc:1631] Marking committed up to 5
0809 15:46:39.715213 (+ 38us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:39.715294 (+ 81us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 15:46:40.915043 (+1199749us) spinlock_profiling.cc:243] Waited 4.92 ms on lock 0x7b44000232cc. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e613cf11 00007f34e620aa95 00007f34e620b4b2 00007f34e8eb3511 00007f34e8eaee1a 00007f34e8ea2bee 00007f34e8ea1f36 00007f34e8ea1b58 00007f34e8eab551 00007f34e8eab509 00007f34e8eab499
0809 15:46:41.246627 (+331584us) raft_consensus.cc:1671] finished
0809 15:46:41.246648 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:41.249234 (+ 2586us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6658,"raft.queue_time_us":18419,"raft.run_cpu_time_us":146,"raft.run_wall_time_us":22503,"spinlock_wait_cycles":11305216,"thread_start_us":287101,"threads_started":1,"wal-append.queue_time_us":506183}
I20260809 15:46:41.365217 26436 raft_consensus.cc:504] T d1a134ba33ea4076bbaac0ae460c17ba P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:41.430243 23283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b30f28311964a0b9149a50495e83d8f" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:41.355538 26490 raft_consensus.cc:3037] T cee02fff2c9c48a19f43eb358a89e7b0 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:46:41.368239 23526 raft_consensus.cc:2454] T d0721a4403a440f383318a3cea9d1442 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 5.
I20260809 15:46:41.372396 26474 leader_election.cc:288] T c5c2e425a74b4e64ae70b5def750434c P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 4 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:41.385375 23394 raft_consensus.cc:2354] T c5c2e425a74b4e64ae70b5def750434c P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c for earlier term 3. Current term is 4.
I20260809 15:46:41.357906 23270 raft_consensus.cc:2454] T 0a51c0baaa6e432997c9d3552af4f254 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 3.
I20260809 15:46:41.384683 26293 leader_election.cc:288] T 16ca412a0dae4feebe85dd01bf1a9173 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:41.353363 26381 raft_consensus.cc:3037] T 297c82ed906d4b48b184985eed3de601 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Advancing to term 6
I20260809 15:46:41.354355 25863 raft_consensus.cc:3037] T 998c6cee72cf40228c6dba064c0db2da P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Advancing to term 7
I20260809 15:46:41.396401 23404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7715d71dbac447b9351fc5eb36c5e80" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:41.422811 26179 raft_consensus.cc:3037] T 3924b648461e463b9d7925e5671d68ae P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Advancing to term 6
I20260809 15:46:41.418023 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f9d6565aa4c45a3b3f01210d0eb71f7" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
W20260809 15:46:41.387684 23228 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.20.202.67:40553 timed out after 2.063s (SENT) [suppressed 7 similar messages]
I20260809 15:46:41.307847 23277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3622f6e3e3641cea1960e3a31498bb7" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:41.425761 23345 leader_election.cc:302] T 563daf42d7784db195ea99c3cb21b370 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:46:41.426628 23528 raft_consensus.cc:2379] T f9e7cadcd24d45fcbfa84209225e8f2d P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d in current term 4: Already voted for candidate 65c36d164b0540f099d592f38307ee6c in this term.
I20260809 15:46:41.416193 23353 leader_election.cc:302] T cef3e7277d7d4760824a3436ed4a847d P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:46:41.233404 25929 raft_consensus.cc:504] T 563daf42d7784db195ea99c3cb21b370 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
W20260809 15:46:41.427407 23273 rpcz_store.cc:269] 0809 15:46:39.621983 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:39.622681 (+ 698us) service_pool.cc:227] Handling call
0809 15:46:41.342090 (+1719409us) spinlock_profiling.cc:243] Waited 1.12 s on lock 0x7b54002f53d4. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e92242ae 00007f34ee161c27 00007f34e7849c04 00007f34e7849b92 00007f34e7849ac1 00007f34e7849a3c 00007f34e7848cf2 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:46:41.342226 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2578814848}
I20260809 15:46:41.428303 23351 leader_election.cc:302] T 538a228ddfb54bc6a36528d5ff07597b P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:41.445768 23351 leader_election.cc:302] T d3be1e356b254d22b995be32f1c12f6f P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 61e69695258f48baa984c9d395e8ce3f, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:41.446105 23353 leader_election.cc:302] T 0c29937a80724ddb9d07a27534a146f3 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:46:41.446870 23351 leader_election.cc:302] T 00a1881267a14ba8b78c81040f17a014 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c
I20260809 15:46:41.428937 23396 rpcz_store.cc:276] 0809 15:46:39.683518 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:41.214967 (+1531449us) service_pool.cc:227] Handling call
0809 15:46:41.227371 (+ 12404us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 15:46:41.320932 (+ 93561us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:41.321072 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:46:41.434178 23472 leader_election.cc:398] T d3e94b7e50754a81b3d72a07476cb7b9 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 5 pre-election: Vote denied by peer c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263) with higher term. Message: Invalid argument: T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c for earlier term 5. Current term is 6.
I20260809 15:46:41.447170 26136 raft_consensus.cc:2726] T 538a228ddfb54bc6a36528d5ff07597b P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260809 15:46:41.447877 26222 raft_consensus.cc:2726] T d3be1e356b254d22b995be32f1c12f6f P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260809 15:46:41.448616 26222 raft_consensus.cc:2781] T 00a1881267a14ba8b78c81040f17a014 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Leader election won for term 5
I20260809 15:46:41.448899 23472 leader_election.cc:302] T 99bdb93503db4de4a1cfb0161ea0c07c P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:46:41.449048 26222 raft_consensus.cc:686] T 00a1881267a14ba8b78c81040f17a014 P c0f38c1908414566aa9115bb2cbdd85d [term 5 LEADER]: Becoming Leader. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:46:41.447825 23396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e4ffcf83c234ae7910ddef3ec0b4bda" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:41.403730 23519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7715d71dbac447b9351fc5eb36c5e80" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:41.450291 23396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "763e58d6ba644eea830a6c1664d39c9b" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:41.450572 25639 raft_consensus.cc:2781] T 99bdb93503db4de4a1cfb0161ea0c07c P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260809 15:46:41.450896 23396 raft_consensus.cc:2454] T 763e58d6ba644eea830a6c1664d39c9b P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 3.
I20260809 15:46:41.442616 23277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c29937a80724ddb9d07a27534a146f3" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:41.440662 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99bdb93503db4de4a1cfb0161ea0c07c" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:41.451624 23277 raft_consensus.cc:2454] T 0c29937a80724ddb9d07a27534a146f3 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 4.
I20260809 15:46:41.451779 23396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e71a05ba1d4d42bd95c3f85f9313399e" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:41.452164 23472 leader_election.cc:302] T 2db057fd10e149a9895dff6ef8c9f038 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:46:41.452335 23396 raft_consensus.cc:3037] T e71a05ba1d4d42bd95c3f85f9313399e P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:46:41.452402 23277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3be1e356b254d22b995be32f1c12f6f" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:41.453203 25875 raft_consensus.cc:2781] T 2db057fd10e149a9895dff6ef8c9f038 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Leader election won for term 4
I20260809 15:46:41.453679 25875 raft_consensus.cc:686] T 2db057fd10e149a9895dff6ef8c9f038 P 65c36d164b0540f099d592f38307ee6c [term 4 LEADER]: Becoming Leader. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:46:41.453724 23277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0741ab44d99439abc3aa3bdd27ee160" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:41.454478 23277 raft_consensus.cc:2379] T c0741ab44d99439abc3aa3bdd27ee160 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 61e69695258f48baa984c9d395e8ce3f in current term 4: Already voted for candidate 65c36d164b0540f099d592f38307ee6c in this term.
I20260809 15:46:41.455324 23519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "763e58d6ba644eea830a6c1664d39c9b" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:41.455164 23277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c48a3049897452383257aa757834fb5" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:41.442775 24475 raft_consensus.cc:2454] T e71a05ba1d4d42bd95c3f85f9313399e P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 4.
I20260809 15:46:41.444298 23280 raft_consensus.cc:3032] T 0acc47ce21c3422698d1ef7277c3ed63 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 LEADER]: Stepping down as leader of term 3
I20260809 15:46:41.455947 23519 raft_consensus.cc:2454] T 763e58d6ba644eea830a6c1664d39c9b P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 3.
I20260809 15:46:41.456229 23280 raft_consensus.cc:729] T 0acc47ce21c3422698d1ef7277c3ed63 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Running, Role: LEADER
I20260809 15:46:41.456736 23519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad8a3b25d7b49e5a2142d5709edd0c7" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:41.456945 23280 consensus_queue.cc:260] T 0acc47ce21c3422698d1ef7277c3ed63 P 67b3aeadfa7a432abe8af4d292e04ab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } attrs { replace: true } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } }
I20260809 15:46:41.457741 23519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2db057fd10e149a9895dff6ef8c9f038" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:41.458060 23280 raft_consensus.cc:3037] T 0acc47ce21c3422698d1ef7277c3ed63 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:46:41.458680 23519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df20671a7b874dbb98c0ef28371c1308" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:41.459043 23396 raft_consensus.cc:2454] T e71a05ba1d4d42bd95c3f85f9313399e P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 4.
I20260809 15:46:41.447970 26463 raft_consensus.cc:2781] T 0c29937a80724ddb9d07a27534a146f3 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260809 15:46:41.459612 26463 raft_consensus.cc:482] T 0c29937a80724ddb9d07a27534a146f3 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:46:41.460031 26463 raft_consensus.cc:3037] T 0c29937a80724ddb9d07a27534a146f3 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Advancing to term 5
I20260809 15:46:41.460199 23396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5c2e425a74b4e64ae70b5def750434c" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:41.450002 26222 consensus_queue.cc:237] T 00a1881267a14ba8b78c81040f17a014 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:41.435186 23404 raft_consensus.cc:2454] T f7715d71dbac447b9351fc5eb36c5e80 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 4.
I20260809 15:46:41.450942 25639 raft_consensus.cc:482] T 99bdb93503db4de4a1cfb0161ea0c07c P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:46:41.422606 25941 raft_consensus.cc:686] T df20671a7b874dbb98c0ef28371c1308 P 65c36d164b0540f099d592f38307ee6c [term 4 LEADER]: Becoming Leader. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:46:41.440536 23285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3be1e356b254d22b995be32f1c12f6f" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:41.465272 23277 raft_consensus.cc:3037] T 0c48a3049897452383257aa757834fb5 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:41.463713 23270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee02fff2c9c48a19f43eb358a89e7b0" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:41.446938 23353 leader_election.cc:302] T d0721a4403a440f383318a3cea9d1442 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:46:41.430271 26464 raft_consensus.cc:3037] T 05ad7d8fdca64e3da527280ca5be9280 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:46:41.477124 23283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2db057fd10e149a9895dff6ef8c9f038" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:41.477569 23353 leader_election.cc:302] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c
I20260809 15:46:41.477805 23283 raft_consensus.cc:3037] T 2db057fd10e149a9895dff6ef8c9f038 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:46:41.421927 26487 consensus_queue.cc:1046] T d3be1e356b254d22b995be32f1c12f6f P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.022s
I20260809 15:46:41.435648 25793 raft_consensus.cc:2781] T 73f6f44c259547f1b05a3a434616fa54 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260809 15:46:41.489809 23282 raft_consensus.cc:2379] T 563daf42d7784db195ea99c3cb21b370 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d in current term 3: Already voted for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in this term.
I20260809 15:46:41.489595 23528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16ca412a0dae4feebe85dd01bf1a9173" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:41.488528 23396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f9d6565aa4c45a3b3f01210d0eb71f7" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:41.389447 26510 consensus_queue.cc:1046] T 28e17dcb5efe45b8b6a8c42da03b4819 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.075s
I20260809 15:46:41.434913 26473 raft_consensus.cc:2726] T 213543cdbe814c89a17a963469922c1b P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260809 15:46:41.496176 23280 raft_consensus.cc:1261] T 0acc47ce21c3422698d1ef7277c3ed63 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 2. (term mismatch)
I20260809 15:46:41.500581 23280 pending_rounds.cc:76] T 0acc47ce21c3422698d1ef7277c3ed63 P 67b3aeadfa7a432abe8af4d292e04ab9: Aborting all ops after (but not including) 1
W20260809 15:46:41.496680 24406 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 99627.4us
I20260809 15:46:41.489230 23273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae6d2c08a7a94f58ac8dd698e3fd456c" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:41.500888 23280 pending_rounds.cc:97] T 0acc47ce21c3422698d1ef7277c3ed63 P 67b3aeadfa7a432abe8af4d292e04ab9: Aborting uncommitted CHANGE_CONFIG_OP operation due to leader change: 1.2
I20260809 15:46:41.505411 23273 raft_consensus.cc:2454] T ae6d2c08a7a94f58ac8dd698e3fd456c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 4.
I20260809 15:46:41.440121 23276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5c2e425a74b4e64ae70b5def750434c" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:41.505613 23280 raft_consensus.cc:2894] T 0acc47ce21c3422698d1ef7277c3ed63 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Aborting config change with OpId 1.2: Aborted: Op aborted by new leader
I20260809 15:46:41.499366 23404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2db057fd10e149a9895dff6ef8c9f038" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:41.464116 26474 raft_consensus.cc:2781] T 563daf42d7784db195ea99c3cb21b370 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader election won for term 3
I20260809 15:46:41.447827 26136 raft_consensus.cc:2781] T cef3e7277d7d4760824a3436ed4a847d P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Leader election won for term 5
I20260809 15:46:41.505796 23270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9e7cadcd24d45fcbfa84209225e8f2d" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:41.506120 23285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e71a05ba1d4d42bd95c3f85f9313399e" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:41.506330 23404 raft_consensus.cc:2379] T 2db057fd10e149a9895dff6ef8c9f038 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 in current term 4: Already voted for candidate 65c36d164b0540f099d592f38307ee6c in this term.
I20260809 15:46:41.509688 23285 raft_consensus.cc:2354] T e71a05ba1d4d42bd95c3f85f9313399e P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 61e69695258f48baa984c9d395e8ce3f for earlier term 3. Current term is 4.
I20260809 15:46:41.510016 23404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df20671a7b874dbb98c0ef28371c1308" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:41.510390 23285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a51c0baaa6e432997c9d3552af4f254" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:41.510695 23404 raft_consensus.cc:2379] T df20671a7b874dbb98c0ef28371c1308 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 in current term 4: Already voted for candidate 65c36d164b0540f099d592f38307ee6c in this term.
I20260809 15:46:41.511101 26436 leader_election.cc:288] T d1a134ba33ea4076bbaac0ae460c17ba P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 3 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:41.511504 23404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9e7cadcd24d45fcbfa84209225e8f2d" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:41.512050 23404 raft_consensus.cc:3037] T f9e7cadcd24d45fcbfa84209225e8f2d P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:46:41.512490 26420 raft_consensus.cc:504] T 907bacfa79d04a55a68c8dbe60056462 P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:41.513677 26507 raft_consensus.cc:504] T 4564dfc6d4094c1b9240b367ccecf3b1 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:41.514153 23527 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44018 (request call id 156) took 1171 ms. Trace:
I20260809 15:46:41.485718 26161 leader_election.cc:288] T 0c48a3049897452383257aa757834fb5 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:41.514518 23527 rpcz_store.cc:276] 0809 15:46:40.343061 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:40.343209 (+ 148us) service_pool.cc:227] Handling call
0809 15:46:41.513961 (+1170752us) spinlock_profiling.cc:243] Waited 82.5 ms on lock 0x7b54001fc054. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e92242ae 00007f34ee161c27 00007f34e7849c04 00007f34e7849b92 00007f34e7849ac1 00007f34e7849a3c 00007f34e7848cf2 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:46:41.514076 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":189656832}
I20260809 15:46:41.516242 23276 raft_consensus.cc:2354] T c5c2e425a74b4e64ae70b5def750434c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c for earlier term 3. Current term is 4.
I20260809 15:46:41.516693 23528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5c2e425a74b4e64ae70b5def750434c" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:41.517076 23276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9e7cadcd24d45fcbfa84209225e8f2d" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:41.517616 26474 raft_consensus.cc:686] T 563daf42d7784db195ea99c3cb21b370 P c0f38c1908414566aa9115bb2cbdd85d [term 3 LEADER]: Becoming Leader. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:46:41.518168 23276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d880de657e544468c82157769622a26" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:41.477116 25639 raft_consensus.cc:3037] T 99bdb93503db4de4a1cfb0161ea0c07c P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:41.521333 25929 leader_election.cc:288] T 563daf42d7784db195ea99c3cb21b370 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:41.523502 26136 raft_consensus.cc:686] T cef3e7277d7d4760824a3436ed4a847d P c0f38c1908414566aa9115bb2cbdd85d [term 5 LEADER]: Becoming Leader. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:46:41.518469 26474 consensus_queue.cc:237] T 563daf42d7784db195ea99c3cb21b370 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:41.524456 26136 consensus_queue.cc:237] T cef3e7277d7d4760824a3436ed4a847d P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:46:41.424399 23520 rpcz_store.cc:269] 0809 15:46:38.639681 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:38.647187 (+ 7506us) service_pool.cc:227] Handling call
0809 15:46:41.269448 (+2622261us) spinlock_profiling.cc:243] Waited 2.02 s on lock 0x7b540016ec54. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e92242ae 00007f34ee161c27 00007f34e7849c04 00007f34e7849b92 00007f34e7849ac1 00007f34e7849a3c 00007f34e7848cf2 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:46:41.269569 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4640005376}
I20260809 15:46:41.506835 23280 raft_consensus.cc:2866] T 0acc47ce21c3422698d1ef7277c3ed63 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted by new leader
I20260809 15:46:41.517233 23528 raft_consensus.cc:3037] T c5c2e425a74b4e64ae70b5def750434c P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:46:41.485790 26144 raft_consensus.cc:2781] T d0721a4403a440f383318a3cea9d1442 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Leader election won for term 5
I20260809 15:46:41.494400 25793 raft_consensus.cc:482] T 73f6f44c259547f1b05a3a434616fa54 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Starting leader election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:46:41.499660 25959 raft_consensus.cc:2726] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260809 15:46:41.528414 23404 raft_consensus.cc:2454] T f9e7cadcd24d45fcbfa84209225e8f2d P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 4.
I20260809 15:46:41.508026 24475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c48a3049897452383257aa757834fb5" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f"
I20260809 15:46:41.507872 23470 leader_election.cc:302] T ae6d2c08a7a94f58ac8dd698e3fd456c P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters: c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:41.541563 23404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00a1881267a14ba8b78c81040f17a014" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:41.541896 23472 leader_election.cc:302] T f9e7cadcd24d45fcbfa84209225e8f2d P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:46:41.542589 23470 leader_election.cc:398] T c5c2e425a74b4e64ae70b5def750434c P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 election: Vote denied by peer 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459) with higher term. Message: Invalid argument: T c5c2e425a74b4e64ae70b5def750434c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c for earlier term 3. Current term is 4.
I20260809 15:46:41.542894 23470 leader_election.cc:401] T c5c2e425a74b4e64ae70b5def750434c P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20260809 15:46:41.514603 26420 leader_election.cc:288] T 907bacfa79d04a55a68c8dbe60056462 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 4 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:41.538172 23273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00a1881267a14ba8b78c81040f17a014" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:41.535773 23223 leader_election.cc:302] T e71a05ba1d4d42bd95c3f85f9313399e P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:46:41.539634 23270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "907bacfa79d04a55a68c8dbe60056462" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:41.544967 26143 raft_consensus.cc:2781] T ae6d2c08a7a94f58ac8dd698e3fd456c P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260809 15:46:41.623138 23270 raft_consensus.cc:2454] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 3.
I20260809 15:46:41.626065 23273 raft_consensus.cc:2379] T 00a1881267a14ba8b78c81040f17a014 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c in current term 5: Already voted for candidate c0f38c1908414566aa9115bb2cbdd85d in this term.
W20260809 15:46:41.626144 23223 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.68:34689, user_credentials={real_user=slave}} blocked reactor thread for 90441.3us
I20260809 15:46:41.525754 23392 raft_consensus.cc:1226] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d [term 5 LEADER]: Rejecting Update request from peer 61e69695258f48baa984c9d395e8ce3f for earlier term 4. Current term is 5. Ops: []
I20260809 15:46:41.454555 25875 consensus_queue.cc:237] T 2db057fd10e149a9895dff6ef8c9f038 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:41.546114 25941 consensus_queue.cc:237] T df20671a7b874dbb98c0ef28371c1308 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:41.502200 26512 raft_consensus.cc:504] T 73f6f44c259547f1b05a3a434616fa54 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:41.631824 26469 raft_consensus.cc:2781] T e71a05ba1d4d42bd95c3f85f9313399e P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader election won for term 4
I20260809 15:46:41.633613 26469 raft_consensus.cc:686] T e71a05ba1d4d42bd95c3f85f9313399e P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 LEADER]: Becoming Leader. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Running, Role: LEADER
I20260809 15:46:41.540697 25793 raft_consensus.cc:3037] T 73f6f44c259547f1b05a3a434616fa54 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:46:41.621124 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5c2e425a74b4e64ae70b5def750434c" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:41.638788 23270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "294d4bcc1e7a477cb56455581277a60f" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:41.508786 23277 raft_consensus.cc:2454] T 0c48a3049897452383257aa757834fb5 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 3.
I20260809 15:46:41.624164 26436 raft_consensus.cc:482] T a3622f6e3e3641cea1960e3a31498bb7 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:41.628803 24475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "563daf42d7784db195ea99c3cb21b370" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:41.631130 23280 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.BulkChangeConfig from 127.0.0.1:55420 (request call id 9) took 18715 ms. Trace:
I20260809 15:46:41.631311 23283 raft_consensus.cc:2454] T 2db057fd10e149a9895dff6ef8c9f038 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 4.
I20260809 15:46:41.630165 25885 raft_consensus.cc:482] T a88421dba80d4d3388aa5eb62e23853b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
W20260809 15:46:41.636073 23470 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 93621.8us
I20260809 15:46:41.619992 23396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7baa2492df0340c4a5ac241959ce1ac3" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:41.640690 25885 raft_consensus.cc:504] T a88421dba80d4d3388aa5eb62e23853b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:41.641434 23470 leader_election.cc:302] T 00a1881267a14ba8b78c81040f17a014 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:41.641494 23396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5c2e425a74b4e64ae70b5def750434c" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:41.642683 25885 leader_election.cc:288] T a88421dba80d4d3388aa5eb62e23853b P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:41.640091 24475 raft_consensus.cc:2454] T 563daf42d7784db195ea99c3cb21b370 P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 3.
I20260809 15:46:41.647361 23223 leader_election.cc:302] T 563daf42d7784db195ea99c3cb21b370 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:46:41.648367 23283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e14d118b2eaa42aa8a7cd4025aba7128" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:41.640089 23277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "763e58d6ba644eea830a6c1664d39c9b" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:41.543634 26144 raft_consensus.cc:686] T d0721a4403a440f383318a3cea9d1442 P c0f38c1908414566aa9115bb2cbdd85d [term 5 LEADER]: Becoming Leader. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:46:41.647035 23273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d880de657e544468c82157769622a26" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:41.671648 23270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5712f6b6fffc4cc5a8ddfbe63e4097df" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:41.630589 26143 raft_consensus.cc:482] T ae6d2c08a7a94f58ac8dd698e3fd456c P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:46:41.650437 26436 raft_consensus.cc:504] T a3622f6e3e3641cea1960e3a31498bb7 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:41.637665 26507 leader_election.cc:288] T 4564dfc6d4094c1b9240b367ccecf3b1 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:41.634742 26512 leader_election.cc:288] T 73f6f44c259547f1b05a3a434616fa54 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
W20260809 15:46:41.695379 26531 log.cc:924] Time spent T 05ad7d8fdca64e3da527280ca5be9280 P c0f38c1908414566aa9115bb2cbdd85d: Append to log took a long time: real 0.094s user 0.000s sys 0.005s
I20260809 15:46:41.716373 26381 raft_consensus.cc:504] T 297c82ed906d4b48b184985eed3de601 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:41.634419 26469 consensus_queue.cc:237] T e71a05ba1d4d42bd95c3f85f9313399e P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:41.625653 26398 raft_consensus.cc:2781] T f9e7cadcd24d45fcbfa84209225e8f2d P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Leader election won for term 4
I20260809 15:46:41.716972 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73f6f44c259547f1b05a3a434616fa54" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:41.717514 23285 raft_consensus.cc:2454] T 0a51c0baaa6e432997c9d3552af4f254 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 3.
I20260809 15:46:41.717250 26144 consensus_queue.cc:237] T d0721a4403a440f383318a3cea9d1442 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:41.647652 23280 rpcz_store.cc:276] 0809 15:46:22.932546 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:22.960746 (+ 28200us) service_pool.cc:227] Handling call
0809 15:46:24.140060 (+1179314us) spinlock_profiling.cc:243] Waited 1.1 s on lock 0x7b58001d63b0. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e919bf63 00007f34e91ffaba 00007f34e921ba7a 00007f34e921cada 00007f34e921bc8b 00007f34e9227084 00007f34e9226743 00007f34ee162f39 00007f34e784e704 00007f34e784e692
0809 15:46:24.212994 (+ 72934us) log.cc:842] Serialized 506 byte log entry
0809 15:46:41.537839 (+17324845us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8176,"raft.queue_time_us":15917,"raft.run_cpu_time_us":1425,"raft.run_wall_time_us":2389,"spinlock_wait_cycles":2541102976,"thread_start_us":132229,"threads_started":1,"wal-append.queue_time_us":158444,"wal-append.run_cpu_time_us":5970,"wal-append.run_wall_time_us":1047951}
I20260809 15:46:41.638238 26214 raft_consensus.cc:482] T 0acc47ce21c3422698d1ef7277c3ed63 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:41.718642 25885 raft_consensus.cc:2781] T 563daf42d7784db195ea99c3cb21b370 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260809 15:46:41.719370 26442 raft_consensus.cc:504] T 907bacfa79d04a55a68c8dbe60056462 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:41.631983 25929 raft_consensus.cc:482] T e14d118b2eaa42aa8a7cd4025aba7128 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:41.720440 23396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "763e58d6ba644eea830a6c1664d39c9b" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
W20260809 15:46:41.720961 24406 leader_election.cc:334] T e71a05ba1d4d42bd95c3f85f9313399e P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459): Timed out: RequestConsensusVote RPC to 127.20.202.65:35459 timed out after 2.055s (SENT)
I20260809 15:46:41.793138 24406 leader_election.cc:302] T e71a05ba1d4d42bd95c3f85f9313399e P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f; no voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:41.649367 23277 raft_consensus.cc:2454] T 763e58d6ba644eea830a6c1664d39c9b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 3.
I20260809 15:46:41.724179 26179 raft_consensus.cc:504] T 3924b648461e463b9d7925e5671d68ae P 61e69695258f48baa984c9d395e8ce3f [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:41.724069 25639 raft_consensus.cc:504] T 99bdb93503db4de4a1cfb0161ea0c07c P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:41.725613 26148 raft_consensus.cc:3032] T 28e17dcb5efe45b8b6a8c42da03b4819 P c0f38c1908414566aa9115bb2cbdd85d [term 3 LEADER]: Stepping down as leader of term 3
I20260809 15:46:41.726210 26490 raft_consensus.cc:504] T cee02fff2c9c48a19f43eb358a89e7b0 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:41.729240 23400 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55320 (request call id 1211) took 1516 ms. Trace:
I20260809 15:46:41.630919 23276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7baa2492df0340c4a5ac241959ce1ac3" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:41.780964 23404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5c2e425a74b4e64ae70b5def750434c" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:41.863896 23400 rpcz_store.cc:276] 0809 15:46:40.212496 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:41.284880 (+1072384us) service_pool.cc:227] Handling call
0809 15:46:41.308614 (+ 23734us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 15:46:41.344123 (+ 35509us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:46:41.344152 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 15:46:41.348496 (+ 4344us) log.cc:842] Serialized 26 byte log entry
0809 15:46:41.437816 (+ 89320us) raft_consensus.cc:1631] Marking committed up to 2
0809 15:46:41.728756 (+290940us) log.cc:842] Serialized 14 byte log entry
0809 15:46:41.728965 (+ 209us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:41.729026 (+ 61us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 15:46:41.729081 (+ 55us) raft_consensus.cc:1671] finished
0809 15:46:41.729092 (+ 11us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:41.729191 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":10667,"prepare.queue_time_us":77090,"prepare.run_cpu_time_us":27,"prepare.run_wall_time_us":28,"raft.queue_time_us":38442,"raft.run_cpu_time_us":283,"raft.run_wall_time_us":48322,"thread_start_us":300527,"threads_started":2,"wal-append.queue_time_us":151869}
W20260809 15:46:41.782584 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 394897us
I20260809 15:46:41.777562 23528 raft_consensus.cc:2454] T c5c2e425a74b4e64ae70b5def750434c P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 4.
I20260809 15:46:41.868129 23528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bcac4b24e7a4448abdc934bba5a624e" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:41.790524 23273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99bdb93503db4de4a1cfb0161ea0c07c" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:41.783167 23402 raft_consensus.cc:2454] T 472816994d814cb198b69e5248697440 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 4.
I20260809 15:46:41.849056 23280 pending_rounds.cc:97] T 0acc47ce21c3422698d1ef7277c3ed63 P 67b3aeadfa7a432abe8af4d292e04ab9: Aborting uncommitted NO_OP operation due to leader change: 3.3
I20260809 15:46:41.868163 23228 leader_election.cc:302] T 0c48a3049897452383257aa757834fb5 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c
I20260809 15:46:41.869196 23280 raft_consensus.cc:2866] T 0acc47ce21c3422698d1ef7277c3ed63 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 15:46:41.779095 26143 raft_consensus.cc:3037] T ae6d2c08a7a94f58ac8dd698e3fd456c P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Advancing to term 5
W20260809 15:46:41.853550 23470 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 212146us
I20260809 15:46:41.792689 23396 raft_consensus.cc:2454] T 763e58d6ba644eea830a6c1664d39c9b P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 3.
I20260809 15:46:41.872707 23470 leader_election.cc:302] T 0c48a3049897452383257aa757834fb5 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 61e69695258f48baa984c9d395e8ce3f
I20260809 15:46:41.858028 25816 raft_consensus.cc:2726] T 00a1881267a14ba8b78c81040f17a014 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260809 15:46:41.858491 25929 raft_consensus.cc:504] T e14d118b2eaa42aa8a7cd4025aba7128 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:41.865087 23285 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0da4d5283c8a46878bd39b85e5c71f66"
dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
mode: GRACEFUL
new_leader_uuid: "65c36d164b0540f099d592f38307ee6c"
from {username='slave'} at 127.0.0.1:33468
I20260809 15:46:41.720184 26464 raft_consensus.cc:504] T 05ad7d8fdca64e3da527280ca5be9280 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:41.789778 23270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c29937a80724ddb9d07a27534a146f3" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:41.795246 25885 raft_consensus.cc:482] T 563daf42d7784db195ea99c3cb21b370 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:46:41.869824 23277 raft_consensus.cc:1226] T 28e17dcb5efe45b8b6a8c42da03b4819 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Rejecting Update request from peer c0f38c1908414566aa9115bb2cbdd85d for earlier term 3. Current term is 4. Ops: []
I20260809 15:46:41.874287 23470 leader_election.cc:302] T 3d880de657e544468c82157769622a26 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 61e69695258f48baa984c9d395e8ce3f, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:41.876132 23228 leader_election.cc:302] T 763e58d6ba644eea830a6c1664d39c9b P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:46:41.874253 23285 raft_consensus.cc:595] T 0da4d5283c8a46878bd39b85e5c71f66 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Received request to transfer leadership to TS 65c36d164b0540f099d592f38307ee6c
I20260809 15:46:41.876929 23285 raft_consensus.cc:603] T 0da4d5283c8a46878bd39b85e5c71f66 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260809 15:46:41.878114 23277 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53562 (request call id 1622) took 1504 ms. Trace:
I20260809 15:46:41.878880 23280 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53562 (request call id 1616) took 1534 ms. Trace:
I20260809 15:46:41.879508 23280 rpcz_store.cc:276] 0809 15:46:40.344819 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:40.807635 (+462816us) service_pool.cc:227] Handling call
0809 15:46:40.814445 (+ 6810us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:41.444208 (+629763us) spinlock_profiling.cc:243] Waited 433 ms on lock 0x7b54003302a0. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e91b0730 00007f34e9222300 00007f34e9223371 00007f34e921fedf 00007f34e921f99e 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc
0809 15:46:41.878670 (+434462us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:41.878834 (+ 164us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":996233984}
I20260809 15:46:41.881747 23526 raft_consensus.cc:2379] T 99bdb93503db4de4a1cfb0161ea0c07c P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d in current term 3: Already voted for candidate 65c36d164b0540f099d592f38307ee6c in this term.
I20260809 15:46:41.875483 23270 raft_consensus.cc:2454] T 0c29937a80724ddb9d07a27534a146f3 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 4.
I20260809 15:46:41.888700 23400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "907bacfa79d04a55a68c8dbe60056462" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:41.889431 23400 raft_consensus.cc:2379] T 907bacfa79d04a55a68c8dbe60056462 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 61e69695258f48baa984c9d395e8ce3f in current term 4: Already voted for candidate c0f38c1908414566aa9115bb2cbdd85d in this term.
I20260809 15:46:41.860003 26523 raft_consensus.cc:2726] T e71a05ba1d4d42bd95c3f85f9313399e P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260809 15:46:41.892226 23404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0721a4403a440f383318a3cea9d1442" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:41.878614 23277 rpcz_store.cc:276] 0809 15:46:40.373782 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:41.869360 (+1495578us) service_pool.cc:227] Handling call
0809 15:46:41.869658 (+ 298us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:41.877590 (+ 7932us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:41.877938 (+ 348us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:46:41.860967 25899 leader_election.cc:288] T 4bcac4b24e7a4448abdc934bba5a624e P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
W20260809 15:46:41.915757 24406 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 194805us
I20260809 15:46:41.915484 23273 raft_consensus.cc:1261] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 5. (term mismatch)
I20260809 15:46:41.864204 26148 raft_consensus.cc:729] T 28e17dcb5efe45b8b6a8c42da03b4819 P c0f38c1908414566aa9115bb2cbdd85d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:46:41.921449 23528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "763e58d6ba644eea830a6c1664d39c9b" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
W20260809 15:46:41.922537 24406 leader_election.cc:334] T 0a51c0baaa6e432997c9d3552af4f254 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459): Timed out: RequestConsensusVote RPC to 127.20.202.65:35459 timed out after 2.089s (SENT)
I20260809 15:46:41.922947 24406 leader_election.cc:302] T 0a51c0baaa6e432997c9d3552af4f254 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f; no voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:41.923655 23273 pending_rounds.cc:76] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 67b3aeadfa7a432abe8af4d292e04ab9: Aborting all ops after (but not including) 4
I20260809 15:46:41.923899 23273 pending_rounds.cc:97] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 67b3aeadfa7a432abe8af4d292e04ab9: Aborting uncommitted NO_OP operation due to leader change: 2.5
I20260809 15:46:41.924113 23273 raft_consensus.cc:2866] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 15:46:41.925554 23283 raft_consensus.cc:2396] T e14d118b2eaa42aa8a7cd4025aba7128 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 61e69695258f48baa984c9d395e8ce3f for term 6 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260809 15:46:41.922060 23528 raft_consensus.cc:2454] T 763e58d6ba644eea830a6c1664d39c9b P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 3.
I20260809 15:46:41.871290 26507 raft_consensus.cc:482] T 6ae14a74a26f434db08c652e097385c8 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:41.871691 25650 raft_consensus.cc:2726] T c5c2e425a74b4e64ae70b5def750434c P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459) with higher term. Message: Invalid argument: T c5c2e425a74b4e64ae70b5def750434c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c for earlier term 3. Current term is 4.
I20260809 15:46:41.929116 23400 raft_consensus.cc:1261] T 99bdb93503db4de4a1cfb0161ea0c07c P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Refusing update from remote peer 67b3aeadfa7a432abe8af4d292e04ab9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 15:46:41.931820 23273 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53562 (request call id 1624) took 1557 ms. Trace:
I20260809 15:46:41.881567 25885 raft_consensus.cc:3037] T 563daf42d7784db195ea99c3cb21b370 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:46:41.934051 23400 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55332 (request call id 1258) took 1597 ms. Trace:
I20260809 15:46:41.933900 23273 rpcz_store.cc:276] 0809 15:46:40.374408 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:41.870461 (+1496053us) service_pool.cc:227] Handling call
0809 15:46:41.891745 (+ 21284us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:41.931585 (+ 39840us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:41.931769 (+ 184us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:46:41.934509 23400 rpcz_store.cc:276] 0809 15:46:40.336836 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:41.889977 (+1553141us) service_pool.cc:227] Handling call
0809 15:46:41.928821 (+ 38844us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:41.933823 (+ 5002us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:41.934001 (+ 178us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:46:41.938691 23404 raft_consensus.cc:1226] T f9e7cadcd24d45fcbfa84209225e8f2d P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Rejecting Update request from peer 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260809 15:46:41.937804 23270 raft_consensus.cc:1261] T 0da4d5283c8a46878bd39b85e5c71f66 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 2. (term mismatch)
I20260809 15:46:41.939503 23270 pending_rounds.cc:76] T 0da4d5283c8a46878bd39b85e5c71f66 P 67b3aeadfa7a432abe8af4d292e04ab9: Aborting all ops after (but not including) 1
I20260809 15:46:41.939776 23270 pending_rounds.cc:97] T 0da4d5283c8a46878bd39b85e5c71f66 P 67b3aeadfa7a432abe8af4d292e04ab9: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20260809 15:46:41.940064 23270 raft_consensus.cc:2866] T 0da4d5283c8a46878bd39b85e5c71f66 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 15:46:41.940119 24475 raft_consensus.cc:1226] T d3be1e356b254d22b995be32f1c12f6f P 61e69695258f48baa984c9d395e8ce3f [term 4 LEADER]: Rejecting Update request from peer 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 3. Current term is 4. Ops: [3.5-3.5]
I20260809 15:46:41.942165 23404 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55332 (request call id 1259) took 1605 ms. Trace:
I20260809 15:46:41.917196 23396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c29937a80724ddb9d07a27534a146f3" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:41.925705 24455 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bcac4b24e7a4448abdc934bba5a624e" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
W20260809 15:46:41.976387 26531 log.cc:924] Time spent T 05ad7d8fdca64e3da527280ca5be9280 P c0f38c1908414566aa9115bb2cbdd85d: Append to log took a long time: real 0.060s user 0.000s sys 0.000s
I20260809 15:46:41.871577 26214 raft_consensus.cc:504] T 0acc47ce21c3422698d1ef7277c3ed63 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:41.925856 23395 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55320 (request call id 1207) took 1710 ms. Trace:
W20260809 15:46:41.986359 26538 log.cc:924] Time spent T 4564dfc6d4094c1b9240b367ccecf3b1 P c0f38c1908414566aa9115bb2cbdd85d: Append to log took a long time: real 0.081s user 0.000s sys 0.000s
I20260809 15:46:41.988981 23518 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44018 (request call id 161) took 1281 ms. Trace:
W20260809 15:46:41.989109 26537 log.cc:924] Time spent T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d: Append to log took a long time: real 0.147s user 0.001s sys 0.000s
W20260809 15:46:41.932629 26524 log.cc:924] Time spent T 0c48a3049897452383257aa757834fb5 P 61e69695258f48baa984c9d395e8ce3f: Append to log took a long time: real 0.162s user 0.000s sys 0.006s
I20260809 15:46:41.990724 23528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0721a4403a440f383318a3cea9d1442" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:42.007300 23528 raft_consensus.cc:2379] T d0721a4403a440f383318a3cea9d1442 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 in current term 5: Already voted for candidate c0f38c1908414566aa9115bb2cbdd85d in this term.
I20260809 15:46:41.925554 24448 raft_consensus.cc:1226] T 3924b648461e463b9d7925e5671d68ae P 61e69695258f48baa984c9d395e8ce3f [term 6 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 5. Current term is 6. Ops: []
I20260809 15:46:41.992754 23273 raft_consensus.cc:1226] T 4564dfc6d4094c1b9240b367ccecf3b1 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 3. Current term is 4. Ops: []
I20260809 15:46:41.993494 23400 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52856 (request call id 146) took 1638 ms. Trace:
W20260809 15:46:41.987126 26431 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 4 current term: 4 is higher.
I20260809 15:46:42.023564 23270 raft_consensus.cc:1226] T 294d4bcc1e7a477cb56455581277a60f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Rejecting Update request from peer 61e69695258f48baa984c9d395e8ce3f for earlier term 4. Current term is 5. Ops: [4.6-4.6]
W20260809 15:46:42.009266 26534 log.cc:924] Time spent T 7660dfa2ef694927b8eda4100a2bcd9b P 61e69695258f48baa984c9d395e8ce3f: Append to log took a long time: real 0.367s user 0.001s sys 0.000s
I20260809 15:46:42.025059 26506 raft_consensus.cc:2726] T 0c48a3049897452383257aa757834fb5 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 15:46:42.007238 26148 consensus_queue.cc:260] T 28e17dcb5efe45b8b6a8c42da03b4819 P c0f38c1908414566aa9115bb2cbdd85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:42.025915 26506 consensus_queue.cc:1046] T d3be1e356b254d22b995be32f1c12f6f P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260809 15:46:42.026520 26465 leader_election.cc:288] T ae6d2c08a7a94f58ac8dd698e3fd456c P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:42.027593 26148 raft_consensus.cc:3037] T 28e17dcb5efe45b8b6a8c42da03b4819 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:46:42.028177 23282 raft_consensus.cc:1261] T f7715d71dbac447b9351fc5eb36c5e80 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20260809 15:46:42.030937 23394 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55320 (request call id 1205) took 1820 ms (client timeout 1500 ms). Trace:
W20260809 15:46:42.031306 23394 rpcz_store.cc:269] 0809 15:46:40.210754 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:40.243509 (+ 32755us) service_pool.cc:227] Handling call
0809 15:46:42.030724 (+1787215us) spinlock_profiling.cc:243] Waited 1.06 s on lock 0x7b540009f6d4. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e92242ae 00007f34ee161c27 00007f34e7849c04 00007f34e7849b92 00007f34e7849ac1 00007f34e7849a3c 00007f34e7848cf2 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:46:42.030842 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2440673920}
I20260809 15:46:42.032613 23394 raft_consensus.cc:1261] T 2424fd9bd7f14e508fd106cbfda786a4 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 5 index: 2. (index mismatch)
I20260809 15:46:42.033133 23394 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55320 (request call id 1219) took 1283 ms. Trace:
I20260809 15:46:42.033414 23394 rpcz_store.cc:276] 0809 15:46:40.749529 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:42.032042 (+1282513us) service_pool.cc:227] Handling call
0809 15:46:42.032326 (+ 284us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:42.032939 (+ 613us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:42.033087 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:46:42.033800 26214 leader_election.cc:288] T 0acc47ce21c3422698d1ef7277c3ed63 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:41.997157 26420 raft_consensus.cc:2726] T 0a51c0baaa6e432997c9d3552af4f254 P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260809 15:46:42.035636 23526 raft_consensus.cc:1261] T 0acc47ce21c3422698d1ef7277c3ed63 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20260809 15:46:42.036226 23526 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56766 (request call id 1775) took 1191 ms. Trace:
I20260809 15:46:42.036520 23526 rpcz_store.cc:276] 0809 15:46:40.844874 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:42.035061 (+1190187us) service_pool.cc:227] Handling call
0809 15:46:42.035349 (+ 288us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:42.035971 (+ 622us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:42.036127 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:46:42.036911 26179 leader_election.cc:288] T 3924b648461e463b9d7925e5671d68ae P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 6 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:41.995411 26447 raft_consensus.cc:2781] T 0c48a3049897452383257aa757834fb5 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Leader election won for term 3
I20260809 15:46:41.942929 23404 rpcz_store.cc:276] 0809 15:46:40.337133 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:41.893106 (+1555973us) service_pool.cc:227] Handling call
0809 15:46:41.938478 (+ 45372us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 15:46:41.941941 (+ 3463us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:41.942118 (+ 177us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:46:41.927354 26488 raft_consensus.cc:2781] T 763e58d6ba644eea830a6c1664d39c9b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260809 15:46:41.941723 26530 log.cc:924] Time spent T 213543cdbe814c89a17a963469922c1b P c0f38c1908414566aa9115bb2cbdd85d: Append to log took a long time: real 0.085s user 0.000s sys 0.006s
W20260809 15:46:41.990088 26542 log.cc:924] Time spent T cee02fff2c9c48a19f43eb358a89e7b0 P 65c36d164b0540f099d592f38307ee6c: Append to log took a long time: real 0.141s user 0.000s sys 0.001s
I20260809 15:46:42.004853 23399 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55320 (request call id 1217) took 1764 ms. Trace:
W20260809 15:46:42.005537 26525 log.cc:924] Time spent T 4b30f28311964a0b9149a50495e83d8f P c0f38c1908414566aa9115bb2cbdd85d: Append to log took a long time: real 0.375s user 0.001s sys 0.000s
I20260809 15:46:42.005903 26386 raft_consensus.cc:2726] T 3d880de657e544468c82157769622a26 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 15:46:42.064924 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3924b648461e463b9d7925e5671d68ae" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:42.063654 23399 rpcz_store.cc:276] 0809 15:46:40.240621 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:41.315843 (+1075222us) service_pool.cc:227] Handling call
0809 15:46:41.316128 (+ 285us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 15:46:41.316317 (+ 189us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:46:41.316340 (+ 23us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 15:46:41.316747 (+ 407us) log.cc:842] Serialized 26 byte log entry
0809 15:46:41.353961 (+ 37214us) raft_consensus.cc:1631] Marking committed up to 3
0809 15:46:41.423697 (+ 69736us) log.cc:842] Serialized 14 byte log entry
0809 15:46:41.423887 (+ 190us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:41.423952 (+ 65us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 15:46:42.004652 (+580700us) raft_consensus.cc:1671] finished
0809 15:46:42.004674 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:42.004797 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6921,"prepare.queue_time_us":140673,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":34,"raft.queue_time_us":74,"raft.run_cpu_time_us":78,"raft.run_wall_time_us":77,"thread_start_us":93081,"threads_started":2,"wal-append.queue_time_us":127011}
I20260809 15:46:42.065935 23400 rpcz_store.cc:276] 0809 15:46:40.355037 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:41.935580 (+1580543us) service_pool.cc:227] Handling call
0809 15:46:41.993048 (+ 57468us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:41.993205 (+ 157us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:46:41.993229 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 15:46:41.993266 (+ 37us) raft_consensus.cc:1631] Marking committed up to 5
0809 15:46:41.993292 (+ 26us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:41.993339 (+ 47us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:41.993443 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:46:41.937665 26507 raft_consensus.cc:504] T 6ae14a74a26f434db08c652e097385c8 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:42.078245 23526 raft_consensus.cc:1261] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260809 15:46:42.078305 23404 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52856 (request call id 149) took 1258 ms. Trace:
I20260809 15:46:42.084336 23404 rpcz_store.cc:276] 0809 15:46:40.819737 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:42.049021 (+1229284us) service_pool.cc:227] Handling call
0809 15:46:42.077880 (+ 28859us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:42.078039 (+ 159us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:46:42.078062 (+ 23us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 15:46:42.078097 (+ 35us) raft_consensus.cc:1631] Marking committed up to 5
0809 15:46:42.078126 (+ 29us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:42.078173 (+ 47us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:42.078258 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:46:42.026595 26506 consensus_queue.cc:1225] T d3be1e356b254d22b995be32f1c12f6f P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Peer 61e69695258f48baa984c9d395e8ce3f log is divergent from this leader: its last log entry 4.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260809 15:46:42.106300 23273 raft_consensus.cc:1226] T cee02fff2c9c48a19f43eb358a89e7b0 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 3. Current term is 4. Ops: []
I20260809 15:46:42.106891 23394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3924b648461e463b9d7925e5671d68ae" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:42.107501 23394 raft_consensus.cc:3037] T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Advancing to term 6
I20260809 15:46:42.049335 26488 raft_consensus.cc:482] T 763e58d6ba644eea830a6c1664d39c9b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:46:42.044384 24453 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38734 (request call id 242) took 1320 ms. Trace:
I20260809 15:46:42.123744 23399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0acc47ce21c3422698d1ef7277c3ed63" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:42.076375 26420 raft_consensus.cc:482] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:42.131006 23529 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44018 (request call id 160) took 1455 ms. Trace:
I20260809 15:46:42.129781 23399 raft_consensus.cc:3032] T 953eea268e17491ca76f3a555860a512 P c0f38c1908414566aa9115bb2cbdd85d [term 3 LEADER]: Stepping down as leader of term 3
I20260809 15:46:42.132613 23404 raft_consensus.cc:1226] T d3be1e356b254d22b995be32f1c12f6f P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Rejecting Update request from peer 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 3. Current term is 4. Ops: []
I20260809 15:46:42.132889 23399 raft_consensus.cc:729] T 953eea268e17491ca76f3a555860a512 P c0f38c1908414566aa9115bb2cbdd85d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:46:42.131390 23529 rpcz_store.cc:276] 0809 15:46:40.675204 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:40.684949 (+ 9745us) service_pool.cc:227] Handling call
0809 15:46:40.695058 (+ 10109us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:40.737072 (+ 42014us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:46:40.737103 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 15:46:40.737140 (+ 37us) raft_consensus.cc:1631] Marking committed up to 5
0809 15:46:40.737169 (+ 29us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:42.130778 (+1393609us) spinlock_profiling.cc:243] Waited 41.8 ms on lock 0x7b54000753d4. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e9220f51 00007f34e921f99e 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772 00007f34e764a94c 00007f34e7649d86
0809 15:46:42.130825 (+ 47us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:42.130949 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":96190720}
I20260809 15:46:42.133652 23399 consensus_queue.cc:260] T 953eea268e17491ca76f3a555860a512 P c0f38c1908414566aa9115bb2cbdd85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:42.134375 23273 raft_consensus.cc:1226] T 0c48a3049897452383257aa757834fb5 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Rejecting Update request from peer 61e69695258f48baa984c9d395e8ce3f for earlier term 2. Current term is 3. Ops: []
I20260809 15:46:42.134706 23399 raft_consensus.cc:3037] T 953eea268e17491ca76f3a555860a512 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:46:42.088536 26447 raft_consensus.cc:686] T 0c48a3049897452383257aa757834fb5 P 65c36d164b0540f099d592f38307ee6c [term 3 LEADER]: Becoming Leader. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:46:42.002004 23518 rpcz_store.cc:276] 0809 15:46:40.707339 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:40.709944 (+ 2605us) service_pool.cc:227] Handling call
0809 15:46:40.755048 (+ 45104us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:41.355328 (+600280us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:41.988750 (+633422us) spinlock_profiling.cc:243] Waited 476 ms on lock 0x7b540038d454. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e9220f51 00007f34e921f99e 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772 00007f34e764a94c 00007f34e7649d86
0809 15:46:41.988927 (+ 177us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1095623424}
W20260809 15:46:42.127331 26463 consensus_meta.cc:287] T 0c29937a80724ddb9d07a27534a146f3 P c0f38c1908414566aa9115bb2cbdd85d: Time spent flushing consensus metadata: real 0.667s user 0.005s sys 0.007s
W20260809 15:46:42.143637 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 267554us
W20260809 15:46:42.140426 26555 log.cc:924] Time spent T 294d4bcc1e7a477cb56455581277a60f P 61e69695258f48baa984c9d395e8ce3f: Append to log took a long time: real 0.134s user 0.001s sys 0.000s
W20260809 15:46:42.124552 25863 consensus_meta.cc:287] T 998c6cee72cf40228c6dba064c0db2da P 65c36d164b0540f099d592f38307ee6c: Time spent flushing consensus metadata: real 0.660s user 0.009s sys 0.000s
I20260809 15:46:42.118368 25899 raft_consensus.cc:482] T ba56725e42f74a1baa4c04d9660727f9 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:42.119798 26436 leader_election.cc:288] T a3622f6e3e3641cea1960e3a31498bb7 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:42.119988 26488 raft_consensus.cc:3037] T 763e58d6ba644eea830a6c1664d39c9b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:46:42.152555 23228 leader_election.cc:302] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:42.136346 25639 leader_election.cc:288] T 99bdb93503db4de4a1cfb0161ea0c07c P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:42.142974 23400 raft_consensus.cc:1261] T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Refusing update from remote peer 61e69695258f48baa984c9d395e8ce3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260809 15:46:42.131747 26386 raft_consensus.cc:482] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:42.153383 23273 raft_consensus.cc:1261] T 2424fd9bd7f14e508fd106cbfda786a4 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 5 index: 2. (index mismatch)
I20260809 15:46:42.128196 26381 leader_election.cc:288] T 297c82ed906d4b48b184985eed3de601 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 6 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:42.162460 24455 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "907bacfa79d04a55a68c8dbe60056462" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f"
I20260809 15:46:42.163370 24455 raft_consensus.cc:2379] T 907bacfa79d04a55a68c8dbe60056462 P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d in current term 4: Already voted for candidate 61e69695258f48baa984c9d395e8ce3f in this term.
I20260809 15:46:42.143736 26442 leader_election.cc:288] T 907bacfa79d04a55a68c8dbe60056462 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 4 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:42.140611 26464 leader_election.cc:288] T 05ad7d8fdca64e3da527280ca5be9280 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:42.162083 26386 raft_consensus.cc:504] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:46:42.162269 26552 log.cc:924] Time spent T 294d4bcc1e7a477cb56455581277a60f P 65c36d164b0540f099d592f38307ee6c: Append to log took a long time: real 0.076s user 0.000s sys 0.000s
I20260809 15:46:42.168356 23228 leader_election.cc:302] T f7715d71dbac447b9351fc5eb36c5e80 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c
I20260809 15:46:42.169320 23397 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55320 (request call id 1210) took 1957 ms. Trace:
I20260809 15:46:42.131513 25793 raft_consensus.cc:504] T 73f6f44c259547f1b05a3a434616fa54 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:42.152302 25929 leader_election.cc:288] T e14d118b2eaa42aa8a7cd4025aba7128 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:42.171944 23397 rpcz_store.cc:276] 0809 15:46:40.212213 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:41.222327 (+1010114us) service_pool.cc:227] Handling call
0809 15:46:41.229000 (+ 6673us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 15:46:41.229184 (+ 184us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:46:41.229198 (+ 14us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 15:46:41.232588 (+ 3390us) log.cc:842] Serialized 26 byte log entry
0809 15:46:41.388236 (+155648us) raft_consensus.cc:1631] Marking committed up to 1
0809 15:46:41.388269 (+ 33us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:41.388329 (+ 60us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 15:46:42.169135 (+780806us) raft_consensus.cc:1671] finished
0809 15:46:42.169154 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:42.169265 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":130886,"threads_started":1,"wal-append.queue_time_us":280909}
W20260809 15:46:42.195961 23470 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 321706us
I20260809 15:46:42.196645 23228 leader_election.cc:302] T 2db057fd10e149a9895dff6ef8c9f038 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:42.139612 26420 raft_consensus.cc:504] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:42.197916 23228 leader_election.cc:302] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:42.196674 23470 leader_election.cc:302] T 763e58d6ba644eea830a6c1664d39c9b P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:46:42.146824 26463 raft_consensus.cc:504] T 0c29937a80724ddb9d07a27534a146f3 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:42.203704 23527 raft_consensus.cc:3037] T 73f6f44c259547f1b05a3a434616fa54 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Advancing to term 5
W20260809 15:46:42.204437 24410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 124775us
I20260809 15:46:42.167217 26398 raft_consensus.cc:686] T f9e7cadcd24d45fcbfa84209225e8f2d P 65c36d164b0540f099d592f38307ee6c [term 4 LEADER]: Becoming Leader. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:46:42.156257 25863 raft_consensus.cc:504] T 998c6cee72cf40228c6dba064c0db2da P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:42.203177 23470 leader_election.cc:302] T 0c29937a80724ddb9d07a27534a146f3 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:46:42.158815 25899 raft_consensus.cc:504] T ba56725e42f74a1baa4c04d9660727f9 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:42.201064 25929 raft_consensus.cc:2781] T f7715d71dbac447b9351fc5eb36c5e80 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260809 15:46:42.170208 26436 consensus_queue.cc:1046] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 67b3aeadfa7a432abe8af4d292e04ab9 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.039s
I20260809 15:46:42.214658 23470 leader_election.cc:302] T 7baa2492df0340c4a5ac241959ce1ac3 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:42.205796 26560 raft_consensus.cc:2781] T 763e58d6ba644eea830a6c1664d39c9b P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260809 15:46:42.218502 23527 raft_consensus.cc:2454] T 73f6f44c259547f1b05a3a434616fa54 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 5.
I20260809 15:46:42.219254 23396 raft_consensus.cc:2379] T 0c29937a80724ddb9d07a27534a146f3 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c in current term 5: Already voted for candidate c0f38c1908414566aa9115bb2cbdd85d in this term.
I20260809 15:46:42.217061 23399 raft_consensus.cc:1261] T 953eea268e17491ca76f3a555860a512 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 3. (term mismatch)
I20260809 15:46:42.229310 23353 leader_election.cc:302] T c5c2e425a74b4e64ae70b5def750434c P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:46:42.230494 23399 pending_rounds.cc:76] T 953eea268e17491ca76f3a555860a512 P c0f38c1908414566aa9115bb2cbdd85d: Aborting all ops after (but not including) 2
I20260809 15:46:42.209507 26465 raft_consensus.cc:2726] T 2db057fd10e149a9895dff6ef8c9f038 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260809 15:46:42.230809 23399 pending_rounds.cc:97] T 953eea268e17491ca76f3a555860a512 P c0f38c1908414566aa9115bb2cbdd85d: Aborting uncommitted NO_OP operation due to leader change: 3.3
I20260809 15:46:42.234246 23399 raft_consensus.cc:2866] T 953eea268e17491ca76f3a555860a512 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260809 15:46:42.208690 26279 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.574s user 0.012s sys 0.004s
I20260809 15:46:42.231561 23353 leader_election.cc:302] T 4bcac4b24e7a4448abdc934bba5a624e P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c
I20260809 15:46:42.213726 26509 raft_consensus.cc:2781] T 0c29937a80724ddb9d07a27534a146f3 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260809 15:46:42.240144 23353 leader_election.cc:302] T 763e58d6ba644eea830a6c1664d39c9b P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:46:42.242496 26228 raft_consensus.cc:2726] T 7baa2492df0340c4a5ac241959ce1ac3 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260809 15:46:42.224491 23223 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.68:34689, user_credentials={real_user=slave}} blocked reactor thread for 248108us
I20260809 15:46:42.232653 26560 raft_consensus.cc:482] T 763e58d6ba644eea830a6c1664d39c9b P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:46:42.229933 26436 consensus_queue.cc:1225] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 67b3aeadfa7a432abe8af4d292e04ab9 [NON_LEADER]: Peer c0f38c1908414566aa9115bb2cbdd85d log is divergent from this leader: its last log entry 3.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260809 15:46:42.224566 25929 raft_consensus.cc:482] T f7715d71dbac447b9351fc5eb36c5e80 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:46:42.236498 26148 raft_consensus.cc:2781] T c5c2e425a74b4e64ae70b5def750434c P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Leader election won for term 4
I20260809 15:46:42.276748 23522 raft_consensus.cc:1226] T 7660dfa2ef694927b8eda4100a2bcd9b P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Rejecting Update request from peer 61e69695258f48baa984c9d395e8ce3f for earlier term 4. Current term is 5. Ops: []
W20260809 15:46:42.240548 26279 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.606s user 0.013s sys 0.004s
I20260809 15:46:42.241752 26405 raft_consensus.cc:2726] T 4bcac4b24e7a4448abdc934bba5a624e P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 15:46:42.203831 26447 consensus_queue.cc:237] T 0c48a3049897452383257aa757834fb5 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:42.245406 26509 raft_consensus.cc:482] T 0c29937a80724ddb9d07a27534a146f3 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:46:42.305581 23517 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56766 (request call id 1766) took 2707 ms. Trace:
I20260809 15:46:42.246655 26512 raft_consensus.cc:2781] T 763e58d6ba644eea830a6c1664d39c9b P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260809 15:46:42.273625 26560 raft_consensus.cc:3037] T 763e58d6ba644eea830a6c1664d39c9b P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Advancing to term 4
W20260809 15:46:42.303026 25875 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.557s user 0.001s sys 0.022s
I20260809 15:46:42.272317 26507 leader_election.cc:288] T 6ae14a74a26f434db08c652e097385c8 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:42.270454 24455 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e14d118b2eaa42aa8a7cd4025aba7128" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:42.275959 25929 raft_consensus.cc:3037] T f7715d71dbac447b9351fc5eb36c5e80 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Advancing to term 5
I20260809 15:46:42.249912 26563 raft_consensus.cc:2726] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260809 15:46:42.351923 23404 raft_consensus.cc:1261] T f7715d71dbac447b9351fc5eb36c5e80 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260809 15:46:42.249262 26228 consensus_queue.cc:1046] T cee02fff2c9c48a19f43eb358a89e7b0 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.067s
I20260809 15:46:42.358016 23274 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53562 (request call id 1613) took 2014 ms. Trace:
I20260809 15:46:42.416462 23274 rpcz_store.cc:276] 0809 15:46:40.344053 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:40.767839 (+423786us) service_pool.cc:227] Handling call
0809 15:46:42.357845 (+1590006us) spinlock_profiling.cc:243] Waited 430 ms on lock 0x7b54000bf754. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e92242ae 00007f34ee161c27 00007f34e7849c04 00007f34e7849b92 00007f34e7849ac1 00007f34e7849a3c 00007f34e7848cf2 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:46:42.357960 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":989588352}
I20260809 15:46:42.417492 23274 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "538a228ddfb54bc6a36528d5ff07597b"
dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
mode: GRACEFUL
new_leader_uuid: "65c36d164b0540f099d592f38307ee6c"
from {username='slave'} at 127.0.0.1:33468
I20260809 15:46:42.418020 23274 raft_consensus.cc:595] T 538a228ddfb54bc6a36528d5ff07597b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Received request to transfer leadership to TS 65c36d164b0540f099d592f38307ee6c
I20260809 15:46:42.302879 26231 raft_consensus.cc:482] T 7660dfa2ef694927b8eda4100a2bcd9b P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:46:42.312611 23517 rpcz_store.cc:276] 0809 15:46:39.570423 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:39.570561 (+ 138us) service_pool.cc:227] Handling call
0809 15:46:39.578144 (+ 7583us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:39.621121 (+ 42977us) raft_consensus.cc:1520] Early marking committed up to index 6
0809 15:46:40.290812 (+669691us) spinlock_profiling.cc:243] Waited 3.37 ms on lock 0x7b180016ada0. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34ef1978b1 00007f34ef1a4661 00007f34e626f4ca 00007f34ef950f99 00007f34e6133990 00007f34e62f5864 00007f34e62f5155 00007f34ecbf5167 00007f34e922a807 00007f34e9239f0f 00007f34e9239e88
0809 15:46:40.361581 (+ 70769us) log.cc:842] Serialized 14 byte log entry
0809 15:46:40.736878 (+375297us) log.cc:842] Serialized 14 byte log entry
0809 15:46:41.232163 (+495285us) spinlock_profiling.cc:243] Waited 8.55 ms on lock 0x7b180016ada0. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34ef1978b1 00007f34ef1a4661 00007f34e62c5f96 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34ef1978b1 00007f34ef1a4661 00007f34e62f8638 00007f34e62fbb11
0809 15:46:42.042369 (+810206us) spinlock_profiling.cc:243] Waited 2.81 ms on lock 0x7b180016ada0. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34ef1978b1 00007f34ef1a4661 00007f34e62da345 00007f34ef111a39 00007f34e62f78bb 00007f34e62f5cbf 00007f34e62f5155 00007f34e92183d0 00007f34e922a576 00007f34e923b0d7 00007f34e923b058
0809 15:46:42.276308 (+233939us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 15:46:42.276362 (+ 54us) raft_consensus.cc:1631] Marking committed up to 6
0809 15:46:42.276403 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:42.276485 (+ 82us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:42.277372 (+ 887us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":79268,"prepare.queue_time_us":672388,"prepare.run_cpu_time_us":27,"prepare.run_wall_time_us":28,"raft.queue_time_us":33153,"raft.run_cpu_time_us":316,"raft.run_wall_time_us":17558,"spinlock_wait_cycles":42030208,"thread_start_us":626351,"threads_started":3,"wal-append.queue_time_us":528762,"wal-append.run_cpu_time_us":2198,"wal-append.run_wall_time_us":1326428}
I20260809 15:46:42.308225 26148 raft_consensus.cc:686] T c5c2e425a74b4e64ae70b5def750434c P c0f38c1908414566aa9115bb2cbdd85d [term 4 LEADER]: Becoming Leader. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
W20260809 15:46:42.365592 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 167722us
I20260809 15:46:42.315515 26509 raft_consensus.cc:3037] T 0c29937a80724ddb9d07a27534a146f3 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Advancing to term 5
W20260809 15:46:42.321810 25875 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.601s user 0.002s sys 0.022s
W20260809 15:46:42.409258 23472 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 297557us
I20260809 15:46:42.316015 26512 raft_consensus.cc:482] T 763e58d6ba644eea830a6c1664d39c9b P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:46:41.995009 23395 rpcz_store.cc:276] 0809 15:46:40.211504 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:41.219825 (+1008321us) service_pool.cc:227] Handling call
0809 15:46:41.220211 (+ 386us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 15:46:41.309172 (+ 88961us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:46:41.309205 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 15:46:41.309612 (+ 407us) log.cc:842] Serialized 26 byte log entry
0809 15:46:41.417791 (+108179us) raft_consensus.cc:1631] Marking committed up to 2
0809 15:46:41.787555 (+369764us) log.cc:842] Serialized 14 byte log entry
0809 15:46:41.864806 (+ 77251us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:41.877829 (+ 13023us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 15:46:41.883065 (+ 5236us) raft_consensus.cc:1671] finished
0809 15:46:41.883079 (+ 14us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:41.921641 (+ 38562us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":54540,"prepare.queue_time_us":82894,"prepare.run_cpu_time_us":27,"prepare.run_wall_time_us":28,"raft.queue_time_us":66506,"raft.run_cpu_time_us":122,"raft.run_wall_time_us":8330,"thread_start_us":400848,"threads_started":2,"wal-append.queue_time_us":183906}
I20260809 15:46:42.304278 26442 consensus_queue.cc:1035] T 0acc47ce21c3422698d1ef7277c3ed63 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.056s
I20260809 15:46:42.250571 26554 consensus_queue.cc:1046] T 4564dfc6d4094c1b9240b367ccecf3b1 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.069s
I20260809 15:46:42.247049 26559 consensus_queue.cc:1035] T f7715d71dbac447b9351fc5eb36c5e80 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.159s
I20260809 15:46:42.418354 23274 raft_consensus.cc:603] T 538a228ddfb54bc6a36528d5ff07597b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260809 15:46:42.277577 26523 consensus_queue.cc:1046] T 0c48a3049897452383257aa757834fb5 P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.122s
W20260809 15:46:42.434193 23470 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 195425us
I20260809 15:46:42.436980 23517 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56770 (request call id 1245) took 1058 ms. Trace:
I20260809 15:46:42.488507 23517 rpcz_store.cc:276] 0809 15:46:41.378533 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:42.419332 (+1040799us) service_pool.cc:227] Handling call
0809 15:46:42.436529 (+ 17197us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:42.436709 (+ 180us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:46:42.436733 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 15:46:42.436767 (+ 34us) raft_consensus.cc:1631] Marking committed up to 4
0809 15:46:42.436792 (+ 25us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:42.436836 (+ 44us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:42.436925 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:46:42.440079 23228 leader_election.cc:302] T c5c2e425a74b4e64ae70b5def750434c P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:42.451977 23277 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53554 (request call id 1293) took 1180 ms. Trace:
I20260809 15:46:42.451686 26420 leader_election.cc:288] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:42.455541 26386 leader_election.cc:288] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
W20260809 15:46:42.495136 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 55151.5us
I20260809 15:46:42.449405 26563 raft_consensus.cc:482] T 0da4d5283c8a46878bd39b85e5c71f66 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:42.443048 26512 raft_consensus.cc:3037] T 763e58d6ba644eea830a6c1664d39c9b P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:46:42.495999 26563 raft_consensus.cc:504] T 0da4d5283c8a46878bd39b85e5c71f66 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:46:42.497458 26557 log.cc:924] Time spent T 3924b648461e463b9d7925e5671d68ae P 65c36d164b0540f099d592f38307ee6c: Append to log took a long time: real 0.419s user 0.001s sys 0.001s
I20260809 15:46:42.497969 26563 leader_election.cc:288] T 0da4d5283c8a46878bd39b85e5c71f66 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:42.498473 26563 raft_consensus.cc:2726] T c5c2e425a74b4e64ae70b5def750434c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260809 15:46:42.433204 24455 raft_consensus.cc:2454] T e14d118b2eaa42aa8a7cd4025aba7128 P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 5.
I20260809 15:46:42.420466 26570 raft_consensus.cc:482] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
I20260809 15:46:42.499611 24451 raft_consensus.cc:1226] T d3be1e356b254d22b995be32f1c12f6f P 61e69695258f48baa984c9d395e8ce3f [term 4 LEADER]: Rejecting Update request from peer 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 3. Current term is 4. Ops: [3.5-3.5]
I20260809 15:46:42.499745 26570 raft_consensus.cc:504] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:42.501698 26570 leader_election.cc:288] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:42.502149 26570 raft_consensus.cc:3032] T 4564dfc6d4094c1b9240b367ccecf3b1 P 65c36d164b0540f099d592f38307ee6c [term 3 LEADER]: Stepping down as leader of term 3
I20260809 15:46:42.502486 26570 raft_consensus.cc:729] T 4564dfc6d4094c1b9240b367ccecf3b1 P 65c36d164b0540f099d592f38307ee6c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:46:42.307744 26398 consensus_queue.cc:237] T f9e7cadcd24d45fcbfa84209225e8f2d P 65c36d164b0540f099d592f38307ee6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:42.447671 26464 raft_consensus.cc:2726] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260809 15:46:42.321962 26569 consensus_queue.cc:1035] T 2424fd9bd7f14e508fd106cbfda786a4 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.131s
I20260809 15:46:42.423935 25793 leader_election.cc:288] T 73f6f44c259547f1b05a3a434616fa54 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 4 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:42.502260 23228 leader_election.cc:302] T d0721a4403a440f383318a3cea9d1442 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:42.490433 23277 rpcz_store.cc:276] 0809 15:46:41.271109 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:41.921736 (+650627us) service_pool.cc:227] Handling call
0809 15:46:41.922023 (+ 287us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:41.922185 (+ 162us) raft_consensus.cc:1520] Early marking committed up to index 2
0809 15:46:42.213246 (+291061us) log.cc:842] Serialized 14 byte log entry
0809 15:46:42.451458 (+238212us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 15:46:42.451583 (+ 125us) raft_consensus.cc:1631] Marking committed up to 2
0809 15:46:42.451644 (+ 61us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:42.451726 (+ 82us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:42.451872 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1058,"raft.queue_time_us":6018,"thread_start_us":512481,"threads_started":2}
I20260809 15:46:42.506076 23277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "297c82ed906d4b48b184985eed3de601" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:42.506649 23277 raft_consensus.cc:3037] T 297c82ed906d4b48b184985eed3de601 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Advancing to term 6
I20260809 15:46:42.507396 23223 leader_election.cc:302] T e14d118b2eaa42aa8a7cd4025aba7128 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:46:42.508510 26523 raft_consensus.cc:3032] T 0c48a3049897452383257aa757834fb5 P 61e69695258f48baa984c9d395e8ce3f [term 2 LEADER]: Stepping down as leader of term 2
I20260809 15:46:42.508862 26523 raft_consensus.cc:729] T 0c48a3049897452383257aa757834fb5 P 61e69695258f48baa984c9d395e8ce3f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 61e69695258f48baa984c9d395e8ce3f, State: Running, Role: LEADER
I20260809 15:46:42.509636 26523 consensus_queue.cc:260] T 0c48a3049897452383257aa757834fb5 P 61e69695258f48baa984c9d395e8ce3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:42.510824 26523 raft_consensus.cc:3037] T 0c48a3049897452383257aa757834fb5 P 61e69695258f48baa984c9d395e8ce3f [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:42.493129 26450 raft_consensus.cc:3032] T d3be1e356b254d22b995be32f1c12f6f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 LEADER]: Stepping down as leader of term 3
I20260809 15:46:42.511592 26450 raft_consensus.cc:729] T d3be1e356b254d22b995be32f1c12f6f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Running, Role: LEADER
I20260809 15:46:42.512277 26450 consensus_queue.cc:260] T d3be1e356b254d22b995be32f1c12f6f P 67b3aeadfa7a432abe8af4d292e04ab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:42.513293 26450 raft_consensus.cc:3037] T d3be1e356b254d22b995be32f1c12f6f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:46:42.411442 25885 raft_consensus.cc:504] T 563daf42d7784db195ea99c3cb21b370 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:42.515957 25885 leader_election.cc:288] T 563daf42d7784db195ea99c3cb21b370 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:42.475303 26148 consensus_queue.cc:237] T c5c2e425a74b4e64ae70b5def750434c P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:42.516615 26570 consensus_queue.cc:260] T 4564dfc6d4094c1b9240b367ccecf3b1 P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:42.517343 23285 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53554 (request call id 1296) took 1244 ms. Trace:
I20260809 15:46:42.517746 26570 raft_consensus.cc:3037] T 4564dfc6d4094c1b9240b367ccecf3b1 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:46:42.520186 25899 leader_election.cc:288] T ba56725e42f74a1baa4c04d9660727f9 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:42.121417 24453 rpcz_store.cc:276] 0809 15:46:40.702305 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:40.703510 (+ 1205us) service_pool.cc:227] Handling call
0809 15:46:41.161388 (+457878us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 15:46:41.161825 (+ 437us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:46:41.161884 (+ 59us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 15:46:41.162353 (+ 469us) log.cc:842] Serialized 26 byte log entry
0809 15:46:41.204753 (+ 42400us) raft_consensus.cc:1631] Marking committed up to 5
0809 15:46:41.204807 (+ 54us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:41.204887 (+ 80us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 15:46:41.211912 (+ 7025us) raft_consensus.cc:1671] finished
0809 15:46:41.215194 (+ 3282us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:41.989962 (+774768us) spinlock_profiling.cc:243] Waited 457 ms on lock 0x7b540047ead0. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e921fa09 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:46:42.023069 (+ 33107us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":9076,"raft.queue_time_us":5332,"raft.run_cpu_time_us":99,"raft.run_wall_time_us":100,"spinlock_wait_cycles":1052209408,"thread_start_us":42029,"threads_started":1,"wal-append.queue_time_us":42384,"wal-append.run_cpu_time_us":1413,"wal-append.run_wall_time_us":1003706}
I20260809 15:46:42.521739 26562 consensus_queue.cc:1035] T 953eea268e17491ca76f3a555860a512 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260809 15:46:42.525251 25875 consensus_queue.cc:1035] T 2424fd9bd7f14e508fd106cbfda786a4 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260809 15:46:42.527525 26574 raft_consensus.cc:2781] T e14d118b2eaa42aa8a7cd4025aba7128 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260809 15:46:42.527964 26574 raft_consensus.cc:482] T e14d118b2eaa42aa8a7cd4025aba7128 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:42.528468 26574 raft_consensus.cc:3037] T e14d118b2eaa42aa8a7cd4025aba7128 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Advancing to term 6
I20260809 15:46:42.517697 23285 rpcz_store.cc:276] 0809 15:46:41.272606 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:41.923490 (+650884us) service_pool.cc:227] Handling call
0809 15:46:41.923737 (+ 247us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 15:46:41.924006 (+ 269us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:46:41.924031 (+ 25us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 15:46:41.924386 (+ 355us) log.cc:842] Serialized 26 byte log entry
0809 15:46:42.221151 (+296765us) raft_consensus.cc:1631] Marking committed up to 2
0809 15:46:42.221211 (+ 60us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:42.221300 (+ 89us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 15:46:42.517155 (+295855us) raft_consensus.cc:1671] finished
0809 15:46:42.517174 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:42.517288 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":101690,"raft.run_cpu_time_us":74,"raft.run_wall_time_us":75,"thread_start_us":229337,"threads_started":1,"wal-append.queue_time_us":480273}
I20260809 15:46:42.531450 23285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99bdb93503db4de4a1cfb0161ea0c07c" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:42.532533 23285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0acc47ce21c3422698d1ef7277c3ed63" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:42.516024 23274 raft_consensus.cc:1226] T d1a134ba33ea4076bbaac0ae460c17ba P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 2. Current term is 3. Ops: []
I20260809 15:46:42.533048 23285 raft_consensus.cc:2454] T 0acc47ce21c3422698d1ef7277c3ed63 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 4.
I20260809 15:46:42.534041 23285 raft_consensus.cc:1226] T 05ad7d8fdca64e3da527280ca5be9280 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 3. Current term is 4. Ops: []
I20260809 15:46:42.444052 25639 raft_consensus.cc:3032] T cee02fff2c9c48a19f43eb358a89e7b0 P 65c36d164b0540f099d592f38307ee6c [term 3 LEADER]: Stepping down as leader of term 3
I20260809 15:46:42.488026 23394 raft_consensus.cc:2454] T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 6.
I20260809 15:46:42.443529 26463 leader_election.cc:288] T 0c29937a80724ddb9d07a27534a146f3 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:42.541534 23397 raft_consensus.cc:1226] T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 5. Current term is 6. Ops: [3.6-5.7]
W20260809 15:46:42.541674 24410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 321382us
I20260809 15:46:42.542109 23517 raft_consensus.cc:1226] T 99bdb93503db4de4a1cfb0161ea0c07c P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Rejecting Update request from peer 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260809 15:46:42.542712 23517 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56770 (request call id 1246) took 1163 ms. Trace:
I20260809 15:46:42.542994 23517 rpcz_store.cc:276] 0809 15:46:41.378812 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:42.490124 (+1111312us) service_pool.cc:227] Handling call
0809 15:46:42.538920 (+ 48796us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 15:46:42.542483 (+ 3563us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:42.542663 (+ 180us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:46:42.540460 23285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73f6f44c259547f1b05a3a434616fa54" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:42.544641 23285 raft_consensus.cc:2379] T 73f6f44c259547f1b05a3a434616fa54 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c in current term 4: Already voted for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in this term.
I20260809 15:46:42.545174 23394 raft_consensus.cc:1226] T 0acc47ce21c3422698d1ef7277c3ed63 P c0f38c1908414566aa9115bb2cbdd85d [term 4 LEADER]: Rejecting Update request from peer 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 3. Current term is 4. Ops: [1.2-3.3]
W20260809 15:46:42.545580 23353 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 294531us
I20260809 15:46:42.550431 24406 leader_election.cc:302] T 294d4bcc1e7a477cb56455581277a60f P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:42.554703 24406 leader_election.cc:302] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:42.458850 25863 leader_election.cc:288] T 998c6cee72cf40228c6dba064c0db2da P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 7 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:42.560814 24410 leader_election.cc:302] T 3924b648461e463b9d7925e5671d68ae P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c
I20260809 15:46:42.561765 23285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998c6cee72cf40228c6dba064c0db2da" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:42.562331 23285 raft_consensus.cc:3037] T 998c6cee72cf40228c6dba064c0db2da P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Advancing to term 7
I20260809 15:46:42.564642 23517 raft_consensus.cc:1261] T d3e94b7e50754a81b3d72a07476cb7b9 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Refusing update from remote peer 61e69695258f48baa984c9d395e8ce3f: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260809 15:46:42.565205 23517 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44018 (request call id 167) took 1078 ms. Trace:
I20260809 15:46:42.565501 23517 rpcz_store.cc:276] 0809 15:46:41.486566 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:42.543902 (+1057336us) service_pool.cc:227] Handling call
0809 15:46:42.564351 (+ 20449us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:42.565001 (+ 650us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:42.565159 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:46:42.422847 26231 raft_consensus.cc:504] T 7660dfa2ef694927b8eda4100a2bcd9b P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:42.421674 25941 consensus_queue.cc:1046] T 3924b648461e463b9d7925e5671d68ae P 65c36d164b0540f099d592f38307ee6c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 5, Time since last communication: 0.133s
I20260809 15:46:42.572157 23470 leader_election.cc:302] T 0acc47ce21c3422698d1ef7277c3ed63 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters: c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:42.573490 23404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4564dfc6d4094c1b9240b367ccecf3b1" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:42.573983 23394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1a134ba33ea4076bbaac0ae460c17ba" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:42.541733 25639 raft_consensus.cc:729] T cee02fff2c9c48a19f43eb358a89e7b0 P 65c36d164b0540f099d592f38307ee6c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:46:42.615438 23277 raft_consensus.cc:2454] T 297c82ed906d4b48b184985eed3de601 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 6.
I20260809 15:46:42.569123 26185 raft_consensus.cc:482] T 16ca412a0dae4feebe85dd01bf1a9173 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:42.561286 26594 raft_consensus.cc:3037] T d0721a4403a440f383318a3cea9d1442 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Advancing to term 5
I20260809 15:46:42.619457 23274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ced718c1a5c4da79cb8c1ebd1435b87" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:42.624245 26581 raft_consensus.cc:2726] T 294d4bcc1e7a477cb56455581277a60f P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260809 15:46:42.624166 23521 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56766 (request call id 1769) took 1847 ms. Trace:
I20260809 15:46:42.626557 26559 raft_consensus.cc:3032] T 3924b648461e463b9d7925e5671d68ae P 65c36d164b0540f099d592f38307ee6c [term 5 LEADER]: Stepping down as leader of term 5
I20260809 15:46:42.584986 26582 raft_consensus.cc:482] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:42.631724 23521 rpcz_store.cc:276] 0809 15:46:40.776623 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:40.776738 (+ 115us) service_pool.cc:227] Handling call
0809 15:46:40.777752 (+ 1014us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:41.311624 (+533872us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:42.623934 (+1312310us) spinlock_profiling.cc:243] Waited 533 ms on lock 0x7b540038de54. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e9220f51 00007f34e921f99e 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772 00007f34e764a94c 00007f34e7649d86
0809 15:46:42.624114 (+ 180us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1226202368}
I20260809 15:46:42.621604 23395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "563daf42d7784db195ea99c3cb21b370" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:42.648705 23521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05ad7d8fdca64e3da527280ca5be9280" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:42.648490 26582 raft_consensus.cc:504] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:42.649761 23521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a88421dba80d4d3388aa5eb62e23853b" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:42.650884 23521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4564dfc6d4094c1b9240b367ccecf3b1" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:42.561636 26507 consensus_queue.cc:1035] T 99bdb93503db4de4a1cfb0161ea0c07c P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.057s
I20260809 15:46:42.538362 26560 raft_consensus.cc:504] T 763e58d6ba644eea830a6c1664d39c9b P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:46:42.653249 23193 auto_rebalancer.cc:254] failed to send replica move request: Aborted: Op aborted by new leader
I20260809 15:46:42.654731 26560 leader_election.cc:288] T 763e58d6ba644eea830a6c1664d39c9b P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 4 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:42.564325 26258 consensus_queue.cc:1035] T 6ced718c1a5c4da79cb8c1ebd1435b87 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.057s
I20260809 15:46:42.658102 23274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16ca412a0dae4feebe85dd01bf1a9173" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:42.659376 23274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "907bacfa79d04a55a68c8dbe60056462" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:42.659925 23274 raft_consensus.cc:3037] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:46:42.660172 23394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a88421dba80d4d3388aa5eb62e23853b" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:42.661329 23394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "563daf42d7784db195ea99c3cb21b370" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:42.662148 23520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "297c82ed906d4b48b184985eed3de601" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:42.662397 23394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ae14a74a26f434db08c652e097385c8" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:42.662927 23520 raft_consensus.cc:2379] T 297c82ed906d4b48b184985eed3de601 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d in current term 6: Already voted for candidate 65c36d164b0540f099d592f38307ee6c in this term.
I20260809 15:46:42.662854 26512 raft_consensus.cc:504] T 763e58d6ba644eea830a6c1664d39c9b P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:42.665346 24468 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38734 (request call id 245) took 1816 ms. Trace:
I20260809 15:46:42.665809 24468 rpcz_store.cc:276] 0809 15:46:40.849210 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:41.194313 (+345103us) service_pool.cc:227] Handling call
0809 15:46:41.203614 (+ 9301us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:41.395294 (+191680us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:46:41.419135 (+ 23841us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 15:46:41.422385 (+ 3250us) raft_consensus.cc:1631] Marking committed up to 5
0809 15:46:41.425370 (+ 2985us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:42.664929 (+1239559us) spinlock_profiling.cc:243] Waited 181 ms on lock 0x7b54006667d4. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e9220f51 00007f34e921f99e 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772 00007f34e764a94c 00007f34e7649d86
0809 15:46:42.664979 (+ 50us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:42.665241 (+ 262us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":417423616}
W20260809 15:46:42.667826 26488 consensus_meta.cc:287] T 763e58d6ba644eea830a6c1664d39c9b P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent flushing consensus metadata: real 0.504s user 0.007s sys 0.002s
I20260809 15:46:42.667953 23527 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56766 (request call id 1790) took 1031 ms. Trace:
I20260809 15:46:42.668200 25929 raft_consensus.cc:504] T f7715d71dbac447b9351fc5eb36c5e80 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:42.670320 23526 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56766 (request call id 1785) took 1314 ms. Trace:
I20260809 15:46:42.670733 23526 rpcz_store.cc:276] 0809 15:46:41.356203 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:42.120414 (+764211us) service_pool.cc:227] Handling call
0809 15:46:42.120717 (+ 303us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 15:46:42.130882 (+ 10165us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:46:42.130910 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 15:46:42.138525 (+ 7615us) log.cc:842] Serialized 26 byte log entry
0809 15:46:42.398719 (+260194us) raft_consensus.cc:1631] Marking committed up to 1
0809 15:46:42.398789 (+ 70us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:42.398898 (+ 109us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 15:46:42.670122 (+271224us) raft_consensus.cc:1671] finished
0809 15:46:42.670143 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:42.670265 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":11303,"raft.queue_time_us":48021,"raft.run_cpu_time_us":142,"raft.run_wall_time_us":21666,"thread_start_us":237594,"threads_started":1,"wal-append.queue_time_us":482134}
I20260809 15:46:42.672402 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae6d2c08a7a94f58ac8dd698e3fd456c" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:42.673435 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e14d118b2eaa42aa8a7cd4025aba7128" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:42.673689 23400 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55320 (request call id 1230) took 1216 ms. Trace:
I20260809 15:46:42.674428 23518 raft_consensus.cc:1226] T 0da4d5283c8a46878bd39b85e5c71f66 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Rejecting Update request from peer 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260809 15:46:42.674160 23400 rpcz_store.cc:276] 0809 15:46:41.457496 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:42.160190 (+702694us) service_pool.cc:227] Handling call
0809 15:46:42.172879 (+ 12689us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 15:46:42.199322 (+ 26443us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:46:42.199350 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 15:46:42.199696 (+ 346us) log.cc:842] Serialized 26 byte log entry
0809 15:46:42.538044 (+338348us) raft_consensus.cc:1631] Marking committed up to 1
0809 15:46:42.538078 (+ 34us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:42.538143 (+ 65us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 15:46:42.673494 (+135351us) raft_consensus.cc:1671] finished
0809 15:46:42.673515 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:42.673636 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":6752,"raft.run_cpu_time_us":121,"raft.run_wall_time_us":4688,"thread_start_us":306738,"threads_started":1,"wal-append.queue_time_us":465773}
I20260809 15:46:42.621621 26584 consensus_queue.cc:1035] T d3e94b7e50754a81b3d72a07476cb7b9 P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.055s
I20260809 15:46:42.677861 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00a1881267a14ba8b78c81040f17a014" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:42.677930 26598 raft_consensus.cc:2726] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260809 15:46:42.678668 23526 raft_consensus.cc:2354] T 00a1881267a14ba8b78c81040f17a014 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 4. Current term is 5.
I20260809 15:46:42.679483 26485 raft_consensus.cc:3037] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:42.632932 26228 raft_consensus.cc:2781] T 0acc47ce21c3422698d1ef7277c3ed63 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260809 15:46:42.632776 26559 raft_consensus.cc:729] T 3924b648461e463b9d7925e5671d68ae P 65c36d164b0540f099d592f38307ee6c [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:46:42.622881 24453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "563daf42d7784db195ea99c3cb21b370" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f"
I20260809 15:46:42.686812 24453 raft_consensus.cc:3037] T 563daf42d7784db195ea99c3cb21b370 P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:46:42.687053 26559 consensus_queue.cc:260] T 3924b648461e463b9d7925e5671d68ae P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:42.688300 26559 raft_consensus.cc:3037] T 3924b648461e463b9d7925e5671d68ae P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Advancing to term 6
I20260809 15:46:42.688678 24463 raft_consensus.cc:1261] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260809 15:46:42.647190 26185 raft_consensus.cc:504] T 16ca412a0dae4feebe85dd01bf1a9173 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:42.641114 26523 raft_consensus.cc:2781] T 3924b648461e463b9d7925e5671d68ae P 61e69695258f48baa984c9d395e8ce3f [term 6 FOLLOWER]: Leader election won for term 6
I20260809 15:46:42.697036 26523 raft_consensus.cc:686] T 3924b648461e463b9d7925e5671d68ae P 61e69695258f48baa984c9d395e8ce3f [term 6 LEADER]: Becoming Leader. State: Replica: 61e69695258f48baa984c9d395e8ce3f, State: Running, Role: LEADER
I20260809 15:46:42.697896 26523 consensus_queue.cc:237] T 3924b648461e463b9d7925e5671d68ae P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:42.649674 23395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae6d2c08a7a94f58ac8dd698e3fd456c" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:42.692494 23519 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56770 (request call id 1248) took 1313 ms. Trace:
I20260809 15:46:42.700062 23395 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6ced718c1a5c4da79cb8c1ebd1435b87"
dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
mode: GRACEFUL
new_leader_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
from {username='slave'} at 127.0.0.1:51280
I20260809 15:46:42.562513 25885 consensus_queue.cc:1046] T f9e7cadcd24d45fcbfa84209225e8f2d P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.061s
I20260809 15:46:42.701053 23395 raft_consensus.cc:595] T 6ced718c1a5c4da79cb8c1ebd1435b87 P c0f38c1908414566aa9115bb2cbdd85d [term 3 LEADER]: Received request to transfer leadership to TS 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:42.701390 26185 leader_election.cc:288] T 16ca412a0dae4feebe85dd01bf1a9173 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:42.702277 23400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3622f6e3e3641cea1960e3a31498bb7" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:42.680774 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1a134ba33ea4076bbaac0ae460c17ba" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:42.641348 25639 consensus_queue.cc:260] T cee02fff2c9c48a19f43eb358a89e7b0 P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:42.703641 23393 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55320 (request call id 1213) took 2490 ms. Trace:
I20260809 15:46:42.704329 25639 raft_consensus.cc:3037] T cee02fff2c9c48a19f43eb358a89e7b0 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:46:42.682245 23520 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56766 (request call id 1784) took 1405 ms. Trace:
I20260809 15:46:42.683451 26488 raft_consensus.cc:504] T 763e58d6ba644eea830a6c1664d39c9b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:42.683907 26509 raft_consensus.cc:504] T 0c29937a80724ddb9d07a27534a146f3 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:42.684757 26228 raft_consensus.cc:482] T 0acc47ce21c3422698d1ef7277c3ed63 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260809 15:46:42.686292 26143 consensus_meta.cc:287] T ae6d2c08a7a94f58ac8dd698e3fd456c P 65c36d164b0540f099d592f38307ee6c: Time spent flushing consensus metadata: real 0.770s user 0.005s sys 0.005s
I20260809 15:46:42.692517 23394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0da4d5283c8a46878bd39b85e5c71f66" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:42.635391 23404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05ad7d8fdca64e3da527280ca5be9280" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
W20260809 15:46:42.687009 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 1.23772e+06us
I20260809 15:46:42.713315 23351 leader_election.cc:302] T df20671a7b874dbb98c0ef28371c1308 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:42.668356 23527 rpcz_store.cc:276] 0809 15:46:41.636323 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:41.649790 (+ 13467us) service_pool.cc:227] Handling call
0809 15:46:42.667803 (+1018013us) spinlock_profiling.cc:243] Waited 415 ms on lock 0x7b54000f82d4. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e92242ae 00007f34ee161c27 00007f34e7849c04 00007f34e7849b92 00007f34e7849ac1 00007f34e7849a3c 00007f34e7848cf2 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:46:42.667900 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":954557824}
I20260809 15:46:42.691952 26582 leader_election.cc:288] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:42.694526 26608 consensus_queue.cc:1035] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20260809 15:46:42.587972 25863 consensus_queue.cc:1046] T d1a134ba33ea4076bbaac0ae460c17ba P 65c36d164b0540f099d592f38307ee6c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.092s
I20260809 15:46:42.705717 23520 rpcz_store.cc:276] 0809 15:46:41.277005 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:41.543238 (+266233us) service_pool.cc:227] Handling call
0809 15:46:42.661974 (+1118736us) spinlock_profiling.cc:243] Waited 4.89 ms on lock 0x7b5800004298. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e613cf11 00007f34e62c0851 00007f34e62c0702 00007f34ef1b0b09 00007f34ee0ec730 00007f34ee1b279b 00007f34ee1b2821 00007f34ee1621b0 00007f34ee161a94 00007f34e7849c04 00007f34e7849b92
0809 15:46:42.682182 (+ 20208us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":11239808}
I20260809 15:46:42.699908 23519 rpcz_store.cc:276] 0809 15:46:41.379323 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:41.458475 (+ 79152us) service_pool.cc:227] Handling call
0809 15:46:42.692304 (+1233829us) spinlock_profiling.cc:243] Waited 906 ms on lock 0x7b54001fc554. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e92242ae 00007f34ee161c27 00007f34e7849c04 00007f34e7849b92 00007f34e7849ac1 00007f34e7849a3c 00007f34e7848cf2 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:46:42.692428 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2083777920}
I20260809 15:46:42.708088 23285 raft_consensus.cc:2454] T 998c6cee72cf40228c6dba064c0db2da P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 7.
I20260809 15:46:42.722021 23285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3be1e356b254d22b995be32f1c12f6f" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:42.710875 23274 raft_consensus.cc:2454] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 4.
I20260809 15:46:42.715255 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ae14a74a26f434db08c652e097385c8" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:42.665768 26512 leader_election.cc:288] T 763e58d6ba644eea830a6c1664d39c9b P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 4 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:42.715322 23351 leader_election.cc:302] T 30c1740ef6874df0a332fbd6fd4b03ce P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:42.727041 23351 leader_election.cc:302] T 6ae14a74a26f434db08c652e097385c8 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:42.728495 23351 leader_election.cc:302] T 0a51c0baaa6e432997c9d3552af4f254 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters: 61e69695258f48baa984c9d395e8ce3f
I20260809 15:46:42.712083 26228 raft_consensus.cc:3037] T 0acc47ce21c3422698d1ef7277c3ed63 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Advancing to term 5
I20260809 15:46:42.730409 23274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0741ab44d99439abc3aa3bdd27ee160" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:42.723104 23285 raft_consensus.cc:2396] T d3be1e356b254d22b995be32f1c12f6f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 61e69695258f48baa984c9d395e8ce3f for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260809 15:46:42.732187 23274 raft_consensus.cc:2379] T c0741ab44d99439abc3aa3bdd27ee160 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 61e69695258f48baa984c9d395e8ce3f in current term 4: Already voted for candidate 65c36d164b0540f099d592f38307ee6c in this term.
I20260809 15:46:42.716594 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3622f6e3e3641cea1960e3a31498bb7" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:42.719798 26527 raft_consensus.cc:482] T 998c6cee72cf40228c6dba064c0db2da P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:42.733309 23520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0da4d5283c8a46878bd39b85e5c71f66" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:42.720175 26144 raft_consensus.cc:2726] T df20671a7b874dbb98c0ef28371c1308 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260809 15:46:42.721938 23468 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.68:34689, user_credentials={real_user=slave}} blocked reactor thread for 469374us
I20260809 15:46:42.721175 26608 consensus_queue.cc:1225] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Peer 61e69695258f48baa984c9d395e8ce3f log is divergent from this leader: its last log entry 4.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260809 15:46:42.743765 23529 raft_consensus.cc:1226] T f9e7cadcd24d45fcbfa84209225e8f2d P 65c36d164b0540f099d592f38307ee6c [term 4 LEADER]: Rejecting Update request from peer 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 3. Current term is 4. Ops: []
W20260809 15:46:42.742684 23472 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 298410us
I20260809 15:46:42.783010 23519 raft_consensus.cc:1261] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260809 15:46:42.717303 26143 raft_consensus.cc:504] T ae6d2c08a7a94f58ac8dd698e3fd456c P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:42.816789 23519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba56725e42f74a1baa4c04d9660727f9" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:42.817404 23526 raft_consensus.cc:1261] T cef3e7277d7d4760824a3436ed4a847d P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
W20260809 15:46:42.818686 24410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 264646us
W20260809 15:46:42.829793 24406 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 275149us
I20260809 15:46:42.831295 24406 leader_election.cc:302] T 16ca412a0dae4feebe85dd01bf1a9173 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
W20260809 15:46:42.860836 23472 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 47705.5us
I20260809 15:46:42.861624 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee02fff2c9c48a19f43eb358a89e7b0" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:42.731698 26136 raft_consensus.cc:2726] T 30c1740ef6874df0a332fbd6fd4b03ce P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260809 15:46:42.735323 26512 raft_consensus.cc:2726] T 6ae14a74a26f434db08c652e097385c8 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260809 15:46:42.813325 21289 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260809 15:46:42.872749 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c29937a80724ddb9d07a27534a146f3" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:42.873399 23527 raft_consensus.cc:2379] T 0c29937a80724ddb9d07a27534a146f3 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d in current term 5: Already voted for candidate 65c36d164b0540f099d592f38307ee6c in this term.
I20260809 15:46:42.874064 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16ca412a0dae4feebe85dd01bf1a9173" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:42.869879 23274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee02fff2c9c48a19f43eb358a89e7b0" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:42.865614 23519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a823d35f4f348d1b52ba9e6d80b00d2" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:42.711812 26574 raft_consensus.cc:504] T e14d118b2eaa42aa8a7cd4025aba7128 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:42.742476 26490 leader_election.cc:288] T cee02fff2c9c48a19f43eb358a89e7b0 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 election: Requested vote from peers 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:42.878675 23528 raft_consensus.cc:1226] T ae6d2c08a7a94f58ac8dd698e3fd456c P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Rejecting Update request from peer c0f38c1908414566aa9115bb2cbdd85d for earlier term 4. Current term is 5. Ops: []
I20260809 15:46:42.731595 26258 consensus_queue.cc:1035] T ae6d2c08a7a94f58ac8dd698e3fd456c P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.141s
I20260809 15:46:42.888916 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "763e58d6ba644eea830a6c1664d39c9b" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
W20260809 15:46:42.893507 24406 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 62305.6us
I20260809 15:46:42.894207 23527 raft_consensus.cc:2379] T 763e58d6ba644eea830a6c1664d39c9b P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d in current term 4: Already voted for candidate 65c36d164b0540f099d592f38307ee6c in this term.
I20260809 15:46:42.894474 24406 leader_election.cc:302] T 907bacfa79d04a55a68c8dbe60056462 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 67b3aeadfa7a432abe8af4d292e04ab9; no voters: c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:42.734377 26594 raft_consensus.cc:2726] T d0721a4403a440f383318a3cea9d1442 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260809 15:46:42.875480 23274 raft_consensus.cc:2379] T cee02fff2c9c48a19f43eb358a89e7b0 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 61e69695258f48baa984c9d395e8ce3f in current term 4: Already voted for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in this term.
I20260809 15:46:42.875445 23519 raft_consensus.cc:2454] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 5.
I20260809 15:46:42.878664 26136 raft_consensus.cc:482] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:42.867057 26527 raft_consensus.cc:504] T 998c6cee72cf40228c6dba064c0db2da P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:42.911409 26144 raft_consensus.cc:2781] T 0a51c0baaa6e432997c9d3552af4f254 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260809 15:46:42.914861 23402 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52856 (request call id 147) took 2340 ms (client timeout 1806 ms). Trace:
I20260809 15:46:42.915849 23274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3be1e356b254d22b995be32f1c12f6f" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:42.910837 26488 leader_election.cc:288] T 763e58d6ba644eea830a6c1664d39c9b P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:42.911206 26584 raft_consensus.cc:2726] T 16ca412a0dae4feebe85dd01bf1a9173 P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260809 15:46:42.916244 24410 leader_election.cc:302] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c; no voters:
I20260809 15:46:42.923887 23394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99bdb93503db4de4a1cfb0161ea0c07c" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:42.924258 23400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73f6f44c259547f1b05a3a434616fa54" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:42.932737 23282 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55434 (request call id 159) took 1573 ms. Trace:
I20260809 15:46:42.704041 23393 rpcz_store.cc:276] 0809 15:46:40.212971 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:41.313121 (+1100150us) service_pool.cc:227] Handling call
0809 15:46:41.313446 (+ 325us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 15:46:41.313658 (+ 212us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:46:41.313685 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 15:46:41.314075 (+ 390us) log.cc:842] Serialized 26 byte log entry
0809 15:46:41.426520 (+112445us) raft_consensus.cc:1631] Marking committed up to 1
0809 15:46:41.428666 (+ 2146us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:41.433173 (+ 4507us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 15:46:42.701865 (+1268692us) spinlock_profiling.cc:243] Waited 10.7 ms on lock 0x7b240014b358. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e613cf11 00007f34e6150cf4 00007f34e614edfb 00007f34e61dd559 00007f34e61dd1e5 00007f34e61dc740 00007f34e8ecf578 00007f34e8ecfcd8 00007f34e8ea76e8 00007f34e8ea2aea 00007f34e8ea1f36
0809 15:46:42.703348 (+ 1483us) raft_consensus.cc:1671] finished
0809 15:46:42.703373 (+ 25us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:42.703582 (+ 209us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":24661888,"thread_start_us":103361,"threads_started":1,"wal-append.queue_time_us":164029}
I20260809 15:46:42.978626 23274 raft_consensus.cc:2396] T d3be1e356b254d22b995be32f1c12f6f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 61e69695258f48baa984c9d395e8ce3f for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260809 15:46:42.978881 23400 raft_consensus.cc:2354] T 73f6f44c259547f1b05a3a434616fa54 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c for earlier term 4. Current term is 5.
I20260809 15:46:42.885537 25929 leader_election.cc:288] T f7715d71dbac447b9351fc5eb36c5e80 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:42.979413 23394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "763e58d6ba644eea830a6c1664d39c9b" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:42.876773 26399 consensus_queue.cc:1035] T f7715d71dbac447b9351fc5eb36c5e80 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.041s
I20260809 15:46:42.979835 23472 leader_election.cc:302] T 99bdb93503db4de4a1cfb0161ea0c07c P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d
W20260809 15:46:42.906472 26447 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.558s user 0.006s sys 0.018s
I20260809 15:46:42.949870 24463 pending_rounds.cc:76] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 61e69695258f48baa984c9d395e8ce3f: Aborting all ops after (but not including) 4
I20260809 15:46:42.876417 26581 consensus_queue.cc:1035] T d3e94b7e50754a81b3d72a07476cb7b9 P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.077s
I20260809 15:46:43.002995 23286 raft_consensus.cc:2396] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260809 15:46:43.003427 23397 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55320 (request call id 1231) took 1545 ms. Trace:
W20260809 15:46:43.004155 23286 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53562 (request call id 1618) took 2656 ms (client timeout 2081 ms). Trace:
I20260809 15:46:42.986095 23394 raft_consensus.cc:2379] T 763e58d6ba644eea830a6c1664d39c9b P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c in current term 4: Already voted for candidate c0f38c1908414566aa9115bb2cbdd85d in this term.
W20260809 15:46:42.973316 23402 rpcz_store.cc:269] 0809 15:46:40.559903 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:40.742771 (+182868us) service_pool.cc:227] Handling call
0809 15:46:41.259819 (+517048us) spinlock_profiling.cc:243] Waited 3 ms on lock 0x7b58001d7c98. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e613cf11 00007f34e62c0851 00007f34e62c0702 00007f34ef1b0b09 00007f34ee0ec730 00007f34ee1b279b 00007f34ee1b2821 00007f34ee1621b0 00007f34ee161a94 00007f34e7849c04 00007f34e7849b92
0809 15:46:42.899834 (+1640015us) spinlock_profiling.cc:243] Waited 398 ms on lock 0x7b5400140ad4. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e92242ae 00007f34ee161c27 00007f34e7849c04 00007f34e7849b92 00007f34e7849ac1 00007f34e7849a3c 00007f34e7848cf2 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:46:42.899934 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":921666176}
I20260809 15:46:43.003770 23397 rpcz_store.cc:276] 0809 15:46:41.458051 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:42.199322 (+741271us) service_pool.cc:227] Handling call
0809 15:46:42.541283 (+341961us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 15:46:42.542072 (+ 789us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:43.003226 (+461154us) spinlock_profiling.cc:243] Waited 331 ms on lock 0x7b54002206d0. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e921fa09 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:46:43.003370 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":760482048}
I20260809 15:46:43.008262 23280 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53562 (request call id 1626) took 2633 ms. Trace:
I20260809 15:46:43.008589 23280 rpcz_store.cc:276] 0809 15:46:40.374953 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:41.880338 (+1505385us) service_pool.cc:227] Handling call
0809 15:46:42.358582 (+478244us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:42.398674 (+ 40092us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:46:42.399558 (+ 884us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 15:46:42.399604 (+ 46us) raft_consensus.cc:1631] Marking committed up to 1
0809 15:46:42.399637 (+ 33us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:42.399697 (+ 60us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:42.469110 (+ 69413us) spinlock_profiling.cc:243] Waited 9.26 ms on lock 0x7b18003b1b20. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34ef1978b1 00007f34ef1a4661 00007f34e62c5f96 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e921fa09 00007f34ee1612e9 00007f34e7847684
0809 15:46:43.008108 (+538998us) spinlock_profiling.cc:243] Waited 478 ms on lock 0x7b54000bf750. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e921fa09 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:46:43.008209 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":40031,"raft.run_cpu_time_us":131,"raft.run_wall_time_us":7499,"spinlock_wait_cycles":1120468224}
I20260809 15:46:43.009990 23280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "907bacfa79d04a55a68c8dbe60056462" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:43.010102 26527 leader_election.cc:288] T 998c6cee72cf40228c6dba064c0db2da P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:42.973239 26144 raft_consensus.cc:482] T 0a51c0baaa6e432997c9d3552af4f254 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting leader election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
I20260809 15:46:43.010509 23280 raft_consensus.cc:2362] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 4. Re-sending same reply.
I20260809 15:46:43.010807 26144 raft_consensus.cc:3037] T 0a51c0baaa6e432997c9d3552af4f254 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:43.004953 26436 raft_consensus.cc:482] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:43.011685 26436 raft_consensus.cc:504] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:46:43.004515 23286 rpcz_store.cc:269] 0809 15:46:40.348134 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:40.701294 (+353160us) service_pool.cc:227] Handling call
0809 15:46:43.003907 (+2302613us) spinlock_profiling.cc:243] Waited 2.3 s on lock 0x7b540030eb54. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e92242ae 00007f34ee161c27 00007f34e7849c04 00007f34e7849b92 00007f34e7849ac1 00007f34e7849a3c 00007f34e7848cf2 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:46:43.004060 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5291177344}
I20260809 15:46:42.975471 23282 rpcz_store.cc:276] 0809 15:46:41.358900 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:42.028827 (+669927us) service_pool.cc:227] Handling call
0809 15:46:42.029099 (+ 272us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 15:46:42.029411 (+ 312us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:46:42.029436 (+ 25us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 15:46:42.029850 (+ 414us) log.cc:842] Serialized 26 byte log entry
0809 15:46:42.231197 (+201347us) raft_consensus.cc:1631] Marking committed up to 6
0809 15:46:42.443097 (+211900us) log.cc:842] Serialized 14 byte log entry
0809 15:46:42.447443 (+ 4346us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:42.447709 (+ 266us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 15:46:42.928783 (+481074us) spinlock_profiling.cc:243] Waited 25.8 ms on lock 0x7b18002ea260. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34ef1978b1 00007f34ef1a4661 00007f34e62f8638 00007f34e62fbb11 00007f34e62fbac9 00007f34e62fba59 00007f34e62fba21 00007f34e62fad1d 00007f34ef55c204 00007f34ef55c039 00007f34e62dadc6
0809 15:46:42.932465 (+ 3682us) raft_consensus.cc:1671] finished
0809 15:46:42.932569 (+ 104us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:42.932684 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":9039,"prepare.queue_time_us":280675,"raft.queue_time_us":26873,"raft.run_cpu_time_us":376,"raft.run_wall_time_us":55098,"spinlock_wait_cycles":103066112,"thread_start_us":394233,"threads_started":2,"wal-append.queue_time_us":478957}
I20260809 15:46:43.016336 23521 raft_consensus.cc:2454] T 4564dfc6d4094c1b9240b367ccecf3b1 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 4.
I20260809 15:46:43.017160 23280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e14d118b2eaa42aa8a7cd4025aba7128" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:42.986999 23274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d880de657e544468c82157769622a26" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:42.987574 23400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998c6cee72cf40228c6dba064c0db2da" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:42.987912 26574 leader_election.cc:288] T e14d118b2eaa42aa8a7cd4025aba7128 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 6 election: Requested vote from peers 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:42.988468 26509 leader_election.cc:288] T 0c29937a80724ddb9d07a27534a146f3 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 5 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:42.990602 26279 raft_consensus.cc:2781] T 907bacfa79d04a55a68c8dbe60056462 P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Leader election won for term 4
I20260809 15:46:42.994937 23396 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55320 (request call id 1232) took 1536 ms. Trace:
I20260809 15:46:43.037770 26279 raft_consensus.cc:686] T 907bacfa79d04a55a68c8dbe60056462 P 61e69695258f48baa984c9d395e8ce3f [term 4 LEADER]: Becoming Leader. State: Replica: 61e69695258f48baa984c9d395e8ce3f, State: Running, Role: LEADER
I20260809 15:46:43.038018 23396 rpcz_store.cc:276] 0809 15:46:41.458582 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:41.873805 (+415223us) service_pool.cc:227] Handling call
0809 15:46:42.994776 (+1120971us) spinlock_profiling.cc:243] Waited 219 ms on lock 0x7b54002ffb54. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e92242ae 00007f34ee161c27 00007f34e7849c04 00007f34e7849b92 00007f34e7849ac1 00007f34e7849a3c 00007f34e7848cf2 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:46:42.994887 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":503284608}
I20260809 15:46:43.038707 26279 consensus_queue.cc:237] T 907bacfa79d04a55a68c8dbe60056462 P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:43.039139 23396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ced718c1a5c4da79cb8c1ebd1435b87" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:43.028923 23400 raft_consensus.cc:3037] T 998c6cee72cf40228c6dba064c0db2da P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Advancing to term 7
I20260809 15:46:42.992585 24463 pending_rounds.cc:97] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 61e69695258f48baa984c9d395e8ce3f: Aborting uncommitted NO_OP operation due to leader change: 4.5
I20260809 15:46:43.038977 26559 raft_consensus.cc:482] T d3e94b7e50754a81b3d72a07476cb7b9 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:43.043674 23397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad8a3b25d7b49e5a2142d5709edd0c7" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:43.033046 23472 leader_election.cc:398] T 73f6f44c259547f1b05a3a434616fa54 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 4 election: Vote denied by peer c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263) with higher term. Message: Invalid argument: T 73f6f44c259547f1b05a3a434616fa54 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c for earlier term 4. Current term is 5.
W20260809 15:46:42.992810 26447 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.645s user 0.006s sys 0.018s
I20260809 15:46:42.953027 26143 leader_election.cc:288] T ae6d2c08a7a94f58ac8dd698e3fd456c P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 5 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:43.041364 23282 raft_consensus.cc:1226] T f7715d71dbac447b9351fc5eb36c5e80 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260809 15:46:42.923544 23395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "297c82ed906d4b48b184985eed3de601" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:43.045372 23472 leader_election.cc:401] T 73f6f44c259547f1b05a3a434616fa54 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
I20260809 15:46:43.045969 23274 raft_consensus.cc:1261] T d3be1e356b254d22b995be32f1c12f6f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Refusing update from remote peer 61e69695258f48baa984c9d395e8ce3f: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 5. (term mismatch)
I20260809 15:46:43.056190 23395 raft_consensus.cc:3037] T 297c82ed906d4b48b184985eed3de601 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Advancing to term 6
I20260809 15:46:43.044991 23394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c29937a80724ddb9d07a27534a146f3" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
W20260809 15:46:43.057582 23353 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 198691us
I20260809 15:46:43.058461 23394 raft_consensus.cc:2379] T 0c29937a80724ddb9d07a27534a146f3 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c in current term 5: Already voted for candidate c0f38c1908414566aa9115bb2cbdd85d in this term.
I20260809 15:46:43.060482 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae6d2c08a7a94f58ac8dd698e3fd456c" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:43.056420 23274 pending_rounds.cc:76] T d3be1e356b254d22b995be32f1c12f6f P 67b3aeadfa7a432abe8af4d292e04ab9: Aborting all ops after (but not including) 4
I20260809 15:46:43.063674 23274 pending_rounds.cc:97] T d3be1e356b254d22b995be32f1c12f6f P 67b3aeadfa7a432abe8af4d292e04ab9: Aborting uncommitted NO_OP operation due to leader change: 3.5
I20260809 15:46:43.063917 23274 raft_consensus.cc:2866] T d3be1e356b254d22b995be32f1c12f6f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260809 15:46:43.047796 23470 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 475687us
I20260809 15:46:43.060616 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998c6cee72cf40228c6dba064c0db2da" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:43.070716 23527 raft_consensus.cc:2379] T 998c6cee72cf40228c6dba064c0db2da P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d in current term 7: Already voted for candidate 65c36d164b0540f099d592f38307ee6c in this term.
I20260809 15:46:43.037689 26399 raft_consensus.cc:2726] T 99bdb93503db4de4a1cfb0161ea0c07c P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260809 15:46:43.052167 24463 raft_consensus.cc:2866] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 15:46:42.982031 26136 raft_consensus.cc:504] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
W20260809 15:46:43.060542 24448 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.565s user 0.017s sys 0.055s
I20260809 15:46:43.085978 23399 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55320 (request call id 1236) took 1614 ms. Trace:
I20260809 15:46:43.087882 23399 rpcz_store.cc:276] 0809 15:46:41.471869 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:42.239385 (+767516us) service_pool.cc:227] Handling call
0809 15:46:42.239705 (+ 320us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:42.239880 (+ 175us) raft_consensus.cc:1520] Early marking committed up to index 2
0809 15:46:42.249795 (+ 9915us) log.cc:842] Serialized 14 byte log entry
0809 15:46:42.670914 (+421119us) spinlock_profiling.cc:243] Waited 20.6 ms on lock 0x7b18000f9ee0. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34ef1978b1 00007f34ef1a4661 00007f34e62da345 00007f34ef111a39 00007f34e62f78bb 00007f34e62f5cbf 00007f34e62f7e22 00007f34e8ea15bb 00007f34e8ea7063 00007f34e8ea757b 00007f34e922a88a
0809 15:46:43.085671 (+414757us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 15:46:43.085715 (+ 44us) raft_consensus.cc:1631] Marking committed up to 2
0809 15:46:43.085751 (+ 36us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:43.085805 (+ 54us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:43.085904 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":8998,"prepare.run_cpu_time_us":57,"prepare.run_wall_time_us":27911,"spinlock_wait_cycles":47445248,"thread_start_us":414579,"threads_started":1}
I20260809 15:46:43.090116 23399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472816994d814cb198b69e5248697440" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:43.090715 23399 raft_consensus.cc:2454] T 472816994d814cb198b69e5248697440 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 4.
I20260809 15:46:43.093318 23400 raft_consensus.cc:2454] T 998c6cee72cf40228c6dba064c0db2da P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 7.
I20260809 15:46:43.094987 23400 raft_consensus.cc:1261] T df20671a7b874dbb98c0ef28371c1308 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
W20260809 15:46:43.081053 23273 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.543s user 0.000s sys 0.038s
W20260809 15:46:43.095677 23273 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.563s user 0.000s sys 0.038s
I20260809 15:46:43.069998 26509 raft_consensus.cc:2726] T 73f6f44c259547f1b05a3a434616fa54 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263) with higher term. Message: Invalid argument: T 73f6f44c259547f1b05a3a434616fa54 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c for earlier term 4. Current term is 5.
I20260809 15:46:43.096500 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a823d35f4f348d1b52ba9e6d80b00d2" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:43.086261 23353 leader_election.cc:302] T 73f6f44c259547f1b05a3a434616fa54 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:46:43.096819 23470 leader_election.cc:302] T 297c82ed906d4b48b184985eed3de601 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:46:43.073540 23393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3aa6a9a52804df0ba9b0ca3c82f38c3" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
W20260809 15:46:43.089668 26469 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.857s user 0.016s sys 0.013s
I20260809 15:46:43.087318 26214 consensus_queue.cc:1046] T 05ad7d8fdca64e3da527280ca5be9280 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.025s
W20260809 15:46:43.115562 26469 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.885s user 0.016s sys 0.013s
I20260809 15:46:43.113142 23472 leader_election.cc:302] T 998c6cee72cf40228c6dba064c0db2da P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
W20260809 15:46:43.092864 24448 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.597s user 0.018s sys 0.056s
I20260809 15:46:43.116819 24448 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38724 (request call id 457) took 1067 ms. Trace:
W20260809 15:46:43.123286 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 394814us
I20260809 15:46:43.054786 26559 raft_consensus.cc:504] T d3e94b7e50754a81b3d72a07476cb7b9 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:43.124135 23351 leader_election.cc:302] T 4b30f28311964a0b9149a50495e83d8f P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:43.116680 26527 raft_consensus.cc:2781] T 73f6f44c259547f1b05a3a434616fa54 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Leader election won for term 5
I20260809 15:46:43.125662 24475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad8a3b25d7b49e5a2142d5709edd0c7" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
W20260809 15:46:43.131927 23283 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55434 (request call id 162) took 1769 ms (client timeout 2236 ms). Trace:
W20260809 15:46:43.132313 23283 rpcz_store.cc:269] 0809 15:46:41.362234 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:41.648153 (+285919us) service_pool.cc:227] Handling call
0809 15:46:43.131721 (+1483568us) spinlock_profiling.cc:243] Waited 273 ms on lock 0x7b54000bf254. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e92242ae 00007f34ee161c27 00007f34e7849c04 00007f34e7849b92 00007f34e7849ac1 00007f34e7849a3c 00007f34e7848cf2 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:46:43.131859 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":627109248}
I20260809 15:46:43.087432 26608 consensus_queue.cc:1035] T 7baa2492df0340c4a5ac241959ce1ac3 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.034s
I20260809 15:46:43.111966 24453 raft_consensus.cc:2454] T 563daf42d7784db195ea99c3cb21b370 P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 4.
W20260809 15:46:43.138340 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 312862us
I20260809 15:46:43.138700 23468 leader_election.cc:302] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:43.140583 24475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16ca412a0dae4feebe85dd01bf1a9173" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:43.141186 24475 raft_consensus.cc:2454] T 16ca412a0dae4feebe85dd01bf1a9173 P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 3.
W20260809 15:46:43.140610 23223 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.68:34689, user_credentials={real_user=slave}} blocked reactor thread for 244071us
I20260809 15:46:43.129738 26477 raft_consensus.cc:2781] T 297c82ed906d4b48b184985eed3de601 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Leader election won for term 6
I20260809 15:46:43.135735 23276 raft_consensus.cc:1203] T e14d118b2eaa42aa8a7cd4025aba7128 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Deduplicated request from leader. Original: 2.5->[4.6-4.6] Dedup: 4.6->[]
I20260809 15:46:43.147132 26136 leader_election.cc:288] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:43.147990 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3aa6a9a52804df0ba9b0ca3c82f38c3" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:43.129633 26563 raft_consensus.cc:482] T 30c1740ef6874df0a332fbd6fd4b03ce P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:43.124300 23393 raft_consensus.cc:2454] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 2.
I20260809 15:46:43.153517 24475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3aa6a9a52804df0ba9b0ca3c82f38c3" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:43.156451 25863 raft_consensus.cc:2726] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260809 15:46:43.156852 25959 consensus_queue.cc:1035] T cef3e7277d7d4760824a3436ed4a847d P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 15:46:43.157298 23470 leader_election.cc:302] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:43.158327 23276 raft_consensus.cc:1226] T e14d118b2eaa42aa8a7cd4025aba7128 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Rejecting Update request from peer 61e69695258f48baa984c9d395e8ce3f for earlier term 4. Current term is 6. Ops: [4.6-4.6]
I20260809 15:46:43.159564 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e14d118b2eaa42aa8a7cd4025aba7128" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f"
I20260809 15:46:43.161607 23223 leader_election.cc:302] T 16ca412a0dae4feebe85dd01bf1a9173 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:46:43.270032 25959 raft_consensus.cc:482] T e71a05ba1d4d42bd95c3f85f9313399e P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:43.162487 23228 leader_election.cc:398] T 00a1881267a14ba8b78c81040f17a014 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 pre-election: Vote denied by peer 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553) with higher term. Message: Invalid argument: T 00a1881267a14ba8b78c81040f17a014 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 4. Current term is 5.
I20260809 15:46:43.210999 23273 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53554 (request call id 1315) took 1662 ms. Trace:
I20260809 15:46:43.217378 26231 leader_election.cc:288] T 7660dfa2ef694927b8eda4100a2bcd9b P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:43.188087 26594 raft_consensus.cc:482] T 28e17dcb5efe45b8b6a8c42da03b4819 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:43.149312 24455 raft_consensus.cc:1261] T c0741ab44d99439abc3aa3bdd27ee160 P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20260809 15:46:43.223702 23193 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 0acc47ce21c3422698d1ef7277c3ed63, TS c0f38c1908414566aa9115bb2cbdd85d -> TS 61e69695258f48baa984c9d395e8ce3f move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:43.161576 26477 raft_consensus.cc:686] T 297c82ed906d4b48b184985eed3de601 P 65c36d164b0540f099d592f38307ee6c [term 6 LEADER]: Becoming Leader. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:46:43.192691 26487 consensus_queue.cc:1046] T 99bdb93503db4de4a1cfb0161ea0c07c P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.032s
I20260809 15:46:43.153119 26574 raft_consensus.cc:482] T c0741ab44d99439abc3aa3bdd27ee160 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:43.117314 26627 consensus_queue.cc:1035] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.045s
I20260809 15:46:43.216215 26611 raft_consensus.cc:2781] T 998c6cee72cf40228c6dba064c0db2da P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Leader election won for term 7
I20260809 15:46:43.150235 26527 raft_consensus.cc:686] T 73f6f44c259547f1b05a3a434616fa54 P c0f38c1908414566aa9115bb2cbdd85d [term 5 LEADER]: Becoming Leader. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:46:43.144145 26214 consensus_queue.cc:1035] T df20671a7b874dbb98c0ef28371c1308 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.053s
W20260809 15:46:43.247992 23468 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.68:34689, user_credentials={real_user=slave}} blocked reactor thread for 87671.4us
I20260809 15:46:43.248373 26598 raft_consensus.cc:482] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:46:43.402102 26598 raft_consensus.cc:504] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:43.403636 24455 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7660dfa2ef694927b8eda4100a2bcd9b" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
W20260809 15:46:43.252787 26610 log.cc:924] Time spent T 0c48a3049897452383257aa757834fb5 P 65c36d164b0540f099d592f38307ee6c: Append to log took a long time: real 0.114s user 0.000s sys 0.000s
I20260809 15:46:43.405267 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3e94b7e50754a81b3d72a07476cb7b9" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:43.405339 26598 leader_election.cc:288] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:43.252693 26118 raft_consensus.cc:2726] T 4b30f28311964a0b9149a50495e83d8f P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 15:46:43.255369 26582 raft_consensus.cc:3032] T d1a134ba33ea4076bbaac0ae460c17ba P 65c36d164b0540f099d592f38307ee6c [term 2 LEADER]: Stepping down as leader of term 2
I20260809 15:46:43.407510 26582 raft_consensus.cc:729] T d1a134ba33ea4076bbaac0ae460c17ba P 65c36d164b0540f099d592f38307ee6c [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:46:43.408572 26582 consensus_queue.cc:260] T d1a134ba33ea4076bbaac0ae460c17ba P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:43.410055 26582 raft_consensus.cc:3037] T d1a134ba33ea4076bbaac0ae460c17ba P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:43.256749 25650 raft_consensus.cc:3032] T 05ad7d8fdca64e3da527280ca5be9280 P 65c36d164b0540f099d592f38307ee6c [term 3 LEADER]: Stepping down as leader of term 3
W20260809 15:46:43.262980 26633 log.cc:924] Time spent T 00a1881267a14ba8b78c81040f17a014 P c0f38c1908414566aa9115bb2cbdd85d: Append to log took a long time: real 0.060s user 0.000s sys 0.000s
I20260809 15:46:43.264567 23518 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56770 (request call id 1243) took 1886 ms. Trace:
I20260809 15:46:43.414204 23518 rpcz_store.cc:276] 0809 15:46:41.378032 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:42.151222 (+773190us) service_pool.cc:227] Handling call
0809 15:46:42.674095 (+522873us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 15:46:42.675015 (+ 920us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:43.264202 (+589187us) spinlock_profiling.cc:243] Waited 477 ms on lock 0x7b54001081d0. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e921fa09 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:46:43.264478 (+ 276us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1096985856}
I20260809 15:46:43.265559 26559 leader_election.cc:288] T d3e94b7e50754a81b3d72a07476cb7b9 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:43.267333 23393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7660dfa2ef694927b8eda4100a2bcd9b" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:43.416316 26527 consensus_queue.cc:237] T 73f6f44c259547f1b05a3a434616fa54 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:43.268350 23472 leader_election.cc:302] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
W20260809 15:46:43.270710 23223 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.68:34689, user_credentials={real_user=slave}} blocked reactor thread for 109166us
W20260809 15:46:43.272768 23470 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 115564us
I20260809 15:46:43.274215 23517 raft_consensus.cc:1226] T 953eea268e17491ca76f3a555860a512 P 65c36d164b0540f099d592f38307ee6c [term 4 LEADER]: Rejecting Update request from peer c0f38c1908414566aa9115bb2cbdd85d for earlier term 3. Current term is 4. Ops: []
I20260809 15:46:43.272332 23273 rpcz_store.cc:276] 0809 15:46:41.548497 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:42.208082 (+659585us) service_pool.cc:227] Handling call
0809 15:46:42.215378 (+ 7296us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:42.215556 (+ 178us) raft_consensus.cc:1520] Early marking committed up to index 6
0809 15:46:42.532920 (+317364us) log.cc:842] Serialized 14 byte log entry
0809 15:46:43.210679 (+677759us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 15:46:43.210729 (+ 50us) raft_consensus.cc:1631] Marking committed up to 6
0809 15:46:43.210770 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:43.210830 (+ 60us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:43.210945 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":114076,"prepare.queue_time_us":296128,"prepare.run_cpu_time_us":32,"prepare.run_wall_time_us":33,"raft.queue_time_us":115385,"raft.run_cpu_time_us":78,"raft.run_wall_time_us":78,"spinlock_wait_cycles":1127168,"thread_start_us":861436,"threads_started":2,"wal-append.queue_time_us":656773}
I20260809 15:46:43.337715 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee02fff2c9c48a19f43eb358a89e7b0" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f"
I20260809 15:46:43.420298 24472 raft_consensus.cc:3037] T cee02fff2c9c48a19f43eb358a89e7b0 P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:46:43.117457 24448 rpcz_store.cc:276] 0809 15:46:42.049432 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:42.077819 (+ 28387us) service_pool.cc:227] Handling call
0809 15:46:42.147921 (+ 70102us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:42.161817 (+ 13896us) raft_consensus.cc:1520] Early marking committed up to index 7
0809 15:46:42.466713 (+304896us) log.cc:842] Serialized 14 byte log entry
0809 15:46:43.116369 (+649656us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 15:46:43.116429 (+ 60us) raft_consensus.cc:1631] Marking committed up to 7
0809 15:46:43.116475 (+ 46us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:43.116545 (+ 70us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:43.116742 (+ 197us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":460019,"prepare.run_cpu_time_us":32,"prepare.run_wall_time_us":33,"raft.queue_time_us":21718,"raft.run_cpu_time_us":95,"raft.run_wall_time_us":95,"thread_start_us":865877,"threads_started":2}
I20260809 15:46:43.345837 26594 raft_consensus.cc:504] T 28e17dcb5efe45b8b6a8c42da03b4819 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:43.345827 26608 raft_consensus.cc:482] T d3be1e356b254d22b995be32f1c12f6f P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 15:46:43.347762 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 185403us
I20260809 15:46:43.433722 23228 leader_election.cc:302] T df20671a7b874dbb98c0ef28371c1308 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:43.274403 26506 raft_consensus.cc:2781] T 16ca412a0dae4feebe85dd01bf1a9173 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260809 15:46:43.272044 25959 raft_consensus.cc:504] T e71a05ba1d4d42bd95c3f85f9313399e P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
W20260809 15:46:43.329594 23193 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 0acc47ce21c3422698d1ef7277c3ed63, TS c0f38c1908414566aa9115bb2cbdd85d -> TS 61e69695258f48baa984c9d395e8ce3f move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:43.438341 26386 raft_consensus.cc:2781] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260809 15:46:43.439160 26386 raft_consensus.cc:482] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Starting leader election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
I20260809 15:46:43.440078 26580 consensus_queue.cc:1046] T f7715d71dbac447b9351fc5eb36c5e80 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260809 15:46:43.440793 26386 raft_consensus.cc:3037] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:43.442500 25959 leader_election.cc:288] T e71a05ba1d4d42bd95c3f85f9313399e P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:43.442932 24472 raft_consensus.cc:2454] T cee02fff2c9c48a19f43eb358a89e7b0 P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 4.
I20260809 15:46:43.443771 23392 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52856 (request call id 140) took 3854 ms. Trace:
I20260809 15:46:43.444198 23392 rpcz_store.cc:276] 0809 15:46:39.589775 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:41.173984 (+1584209us) service_pool.cc:227] Handling call
0809 15:46:41.193124 (+ 19140us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:41.632469 (+439345us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:43.429805 (+1797336us) spinlock_profiling.cc:243] Waited 301 ms on lock 0x7b54001cedd4. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e9220f51 00007f34e921f99e 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772 00007f34e764a94c 00007f34e7649d86
0809 15:46:43.443663 (+ 13858us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":694447360}
I20260809 15:46:43.437033 26506 raft_consensus.cc:482] T 16ca412a0dae4feebe85dd01bf1a9173 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:43.449395 26506 raft_consensus.cc:3037] T 16ca412a0dae4feebe85dd01bf1a9173 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:46:43.453150 23228 leader_election.cc:302] T d1a134ba33ea4076bbaac0ae460c17ba P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:43.455302 23228 leader_election.cc:302] T a88421dba80d4d3388aa5eb62e23853b P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:43.461050 26469 raft_consensus.cc:2726] T d1a134ba33ea4076bbaac0ae460c17ba P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260809 15:46:43.462136 23274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "763e58d6ba644eea830a6c1664d39c9b" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:43.462841 23274 raft_consensus.cc:2379] T 763e58d6ba644eea830a6c1664d39c9b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c in current term 4: Already voted for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in this term.
I20260809 15:46:43.463455 23282 raft_consensus.cc:1261] T 2db057fd10e149a9895dff6ef8c9f038 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260809 15:46:43.463940 26580 raft_consensus.cc:3032] T f7715d71dbac447b9351fc5eb36c5e80 P 65c36d164b0540f099d592f38307ee6c [term 4 LEADER]: Stepping down as leader of term 4
I20260809 15:46:43.464330 26580 raft_consensus.cc:729] T f7715d71dbac447b9351fc5eb36c5e80 P 65c36d164b0540f099d592f38307ee6c [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:46:43.250298 26563 raft_consensus.cc:504] T 30c1740ef6874df0a332fbd6fd4b03ce P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:43.274596 25863 raft_consensus.cc:2726] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260809 15:46:43.358120 23404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3e94b7e50754a81b3d72a07476cb7b9" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:43.382293 21289 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
W20260809 15:46:43.389014 23444 heartbeater.cc:644] Failed to heartbeat to 127.20.202.126:43895 (0 consecutive failures): Timed out: Failed to send heartbeat to master: TSHeartbeat RPC to 127.20.202.126:43895 timed out after 15.000s (SENT)
I20260809 15:46:43.411432 25650 raft_consensus.cc:729] T 05ad7d8fdca64e3da527280ca5be9280 P 65c36d164b0540f099d592f38307ee6c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
W20260809 15:46:43.391433 25639 consensus_meta.cc:287] T cee02fff2c9c48a19f43eb358a89e7b0 P 65c36d164b0540f099d592f38307ee6c: Time spent flushing consensus metadata: real 0.687s user 0.007s sys 0.003s
I20260809 15:46:43.422593 23517 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56766 (request call id 1789) took 1925 ms. Trace:
I20260809 15:46:43.484186 23517 rpcz_store.cc:276] 0809 15:46:41.497576 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:42.566127 (+1068551us) service_pool.cc:227] Handling call
0809 15:46:42.586320 (+ 20193us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:43.271884 (+685564us) spinlock_profiling.cc:243] Waited 34.8 ms on lock 0x7b54002349a0. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e91b0730 00007f34e9222300 00007f34e9223371 00007f34e921fedf 00007f34e921f99e 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc
0809 15:46:43.419881 (+147997us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:43.422538 (+ 2657us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":79993728}
W20260809 15:46:43.418938 23472 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 150666us
I20260809 15:46:43.398856 26436 leader_election.cc:288] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:43.423233 23223 leader_election.cc:302] T 563daf42d7784db195ea99c3cb21b370 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:46:43.467046 26185 raft_consensus.cc:3032] T f9e7cadcd24d45fcbfa84209225e8f2d P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 LEADER]: Stepping down as leader of term 3
I20260809 15:46:43.467145 26485 raft_consensus.cc:2726] T a88421dba80d4d3388aa5eb62e23853b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260809 15:46:43.470810 23286 raft_consensus.cc:1226] T e14d118b2eaa42aa8a7cd4025aba7128 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 5. Current term is 6. Ops: []
I20260809 15:46:43.471119 23522 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44018 (request call id 153) took 3345 ms. Trace:
I20260809 15:46:43.474175 26611 raft_consensus.cc:686] T 998c6cee72cf40228c6dba064c0db2da P 65c36d164b0540f099d592f38307ee6c [term 7 LEADER]: Becoming Leader. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:46:43.392138 26574 raft_consensus.cc:504] T c0741ab44d99439abc3aa3bdd27ee160 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:43.438087 26487 raft_consensus.cc:2726] T df20671a7b874dbb98c0ef28371c1308 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20260809 15:46:43.485797 23223 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.68:34689, user_credentials={real_user=slave}} blocked reactor thread for 62644.7us
I20260809 15:46:43.484066 25863 raft_consensus.cc:482] T 472816994d814cb198b69e5248697440 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:43.503489 23470 leader_election.cc:302] T 763e58d6ba644eea830a6c1664d39c9b P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d
W20260809 15:46:43.505970 26228 consensus_meta.cc:287] T 0acc47ce21c3422698d1ef7277c3ed63 P 65c36d164b0540f099d592f38307ee6c: Time spent flushing consensus metadata: real 0.646s user 0.011s sys 0.000s
I20260809 15:46:43.456156 26144 raft_consensus.cc:504] T 0a51c0baaa6e432997c9d3552af4f254 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:43.532122 23286 raft_consensus.cc:1261] T df20671a7b874dbb98c0ef28371c1308 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20260809 15:46:43.508092 26386 raft_consensus.cc:504] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:43.511833 26594 leader_election.cc:288] T 28e17dcb5efe45b8b6a8c42da03b4819 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:43.515798 25650 consensus_queue.cc:260] T 05ad7d8fdca64e3da527280ca5be9280 P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:43.466761 26580 consensus_queue.cc:260] T f7715d71dbac447b9351fc5eb36c5e80 P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:43.474886 26608 raft_consensus.cc:504] T d3be1e356b254d22b995be32f1c12f6f P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:43.490290 23522 rpcz_store.cc:276] 0809 15:46:40.125831 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:40.125954 (+ 123us) service_pool.cc:227] Handling call
0809 15:46:42.276591 (+2150637us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:42.302614 (+ 26023us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:43.470937 (+1168323us) spinlock_profiling.cc:243] Waited 2.08 s on lock 0x7b54000f87d0. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e921fa09 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:46:43.471063 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4793988352}
I20260809 15:46:43.523530 23223 leader_election.cc:302] T cee02fff2c9c48a19f43eb358a89e7b0 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:46:43.481436 21289 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 15:46:43.491050 26469 raft_consensus.cc:2781] T 563daf42d7784db195ea99c3cb21b370 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader election won for term 4
W20260809 15:46:43.533864 24406 leader_election.cc:334] T cee02fff2c9c48a19f43eb358a89e7b0 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459): Timed out: RequestConsensusVote RPC to 127.20.202.65:35459 timed out after 1.595s (SENT)
I20260809 15:46:43.541082 24406 leader_election.cc:302] T cee02fff2c9c48a19f43eb358a89e7b0 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:43.489555 26185 raft_consensus.cc:729] T f9e7cadcd24d45fcbfa84209225e8f2d P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Running, Role: LEADER
I20260809 15:46:43.536401 23528 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56766 (request call id 1773) took 2692 ms. Trace:
I20260809 15:46:43.544111 26386 leader_election.cc:288] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:43.545316 24453 raft_consensus.cc:1226] T 563daf42d7784db195ea99c3cb21b370 P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Rejecting Update request from peer c0f38c1908414566aa9115bb2cbdd85d for earlier term 3. Current term is 4. Ops: []
I20260809 15:46:43.545975 24448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e71a05ba1d4d42bd95c3f85f9313399e" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:43.548933 23283 raft_consensus.cc:1261] T c0741ab44d99439abc3aa3bdd27ee160 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260809 15:46:43.546905 23528 rpcz_store.cc:276] 0809 15:46:40.844297 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:42.024671 (+1180374us) service_pool.cc:227] Handling call
0809 15:46:42.072698 (+ 48027us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:42.884829 (+812131us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:43.536192 (+651363us) spinlock_profiling.cc:243] Waited 804 ms on lock 0x7b54001dfbd4. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e9220f51 00007f34e921f99e 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772 00007f34e764a94c 00007f34e7649d86
0809 15:46:43.536345 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1848801664}
W20260809 15:46:43.552294 23470 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 48895.8us
I20260809 15:46:43.545212 26608 leader_election.cc:288] T d3be1e356b254d22b995be32f1c12f6f P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:43.562003 26627 raft_consensus.cc:482] T 28e17dcb5efe45b8b6a8c42da03b4819 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:43.565080 26627 raft_consensus.cc:504] T 28e17dcb5efe45b8b6a8c42da03b4819 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:43.565744 26559 consensus_queue.cc:1046] T e14d118b2eaa42aa8a7cd4025aba7128 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260809 15:46:43.475488 26477 consensus_queue.cc:237] T 297c82ed906d4b48b184985eed3de601 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:43.566926 26559 raft_consensus.cc:3032] T e14d118b2eaa42aa8a7cd4025aba7128 P 65c36d164b0540f099d592f38307ee6c [term 5 LEADER]: Stepping down as leader of term 5
I20260809 15:46:43.567412 26559 raft_consensus.cc:729] T e14d118b2eaa42aa8a7cd4025aba7128 P 65c36d164b0540f099d592f38307ee6c [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
W20260809 15:46:43.569598 23116 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:56218 (request call id 241) took 15171 ms (client timeout 15000 ms). Trace:
I20260809 15:46:43.567950 26627 leader_election.cc:288] T 28e17dcb5efe45b8b6a8c42da03b4819 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:43.574461 23276 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55434 (request call id 154) took 2257 ms. Trace:
I20260809 15:46:43.552038 25639 raft_consensus.cc:2726] T 763e58d6ba644eea830a6c1664d39c9b P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260809 15:46:43.562665 24448 raft_consensus.cc:2454] T e71a05ba1d4d42bd95c3f85f9313399e P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 4.
W20260809 15:46:43.576220 23468 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.68:34689, user_credentials={real_user=slave}} blocked reactor thread for 172123us
I20260809 15:46:43.577191 23468 leader_election.cc:302] T 7660dfa2ef694927b8eda4100a2bcd9b P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:43.577535 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3aa6a9a52804df0ba9b0ca3c82f38c3" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:43.577941 25793 consensus_queue.cc:1035] T c0741ab44d99439abc3aa3bdd27ee160 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260809 15:46:43.578229 23468 leader_election.cc:302] T d3e94b7e50754a81b3d72a07476cb7b9 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:43.579675 23529 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56770 (request call id 1241) took 2202 ms. Trace:
I20260809 15:46:43.548483 26487 raft_consensus.cc:2781] T cee02fff2c9c48a19f43eb358a89e7b0 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader election won for term 4
I20260809 15:46:43.549181 26420 raft_consensus.cc:2726] T cee02fff2c9c48a19f43eb358a89e7b0 P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260809 15:46:43.551498 26594 raft_consensus.cc:482] T 00a1881267a14ba8b78c81040f17a014 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:43.593006 26594 raft_consensus.cc:504] T 00a1881267a14ba8b78c81040f17a014 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:43.596112 23282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c29937a80724ddb9d07a27534a146f3" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
W20260809 15:46:43.600862 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 128066us
I20260809 15:46:43.603250 23521 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56770 (request call id 1252) took 1021 ms. Trace:
I20260809 15:46:43.603652 23521 rpcz_store.cc:276] 0809 15:46:42.582294 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:42.650668 (+ 68374us) service_pool.cc:227] Handling call
0809 15:46:43.603108 (+952440us) spinlock_profiling.cc:243] Waited 224 ms on lock 0x7b540020ec54. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e92242ae 00007f34ee161c27 00007f34e7849c04 00007f34e7849b92 00007f34e7849ac1 00007f34e7849a3c 00007f34e7848cf2 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:46:43.603197 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":516288768}
I20260809 15:46:43.561537 23526 raft_consensus.cc:2454] T cee02fff2c9c48a19f43eb358a89e7b0 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 4.
I20260809 15:46:43.580050 23529 rpcz_store.cc:276] 0809 15:46:41.377452 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:42.134467 (+757015us) service_pool.cc:227] Handling call
0809 15:46:42.149110 (+ 14643us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:42.786928 (+637818us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:43.579480 (+792552us) spinlock_profiling.cc:243] Waited 586 ms on lock 0x7b54000915d4. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e9220f51 00007f34e921f99e 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772 00007f34e764a94c 00007f34e7649d86
0809 15:46:43.579622 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1348714752}
I20260809 15:46:43.605544 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3aa6a9a52804df0ba9b0ca3c82f38c3" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f"
I20260809 15:46:43.555258 26469 raft_consensus.cc:686] T 563daf42d7784db195ea99c3cb21b370 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 LEADER]: Becoming Leader. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Running, Role: LEADER
I20260809 15:46:43.563488 26580 raft_consensus.cc:3037] T f7715d71dbac447b9351fc5eb36c5e80 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Advancing to term 5
I20260809 15:46:43.557842 26609 raft_consensus.cc:482] T cef3e7277d7d4760824a3436ed4a847d P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:43.572721 26485 raft_consensus.cc:482] T 0c29937a80724ddb9d07a27534a146f3 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:43.583516 26619 raft_consensus.cc:2726] T 7660dfa2ef694927b8eda4100a2bcd9b P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260809 15:46:43.533558 26228 raft_consensus.cc:504] T 0acc47ce21c3422698d1ef7277c3ed63 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:43.625465 26228 leader_election.cc:288] T 0acc47ce21c3422698d1ef7277c3ed63 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 5 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
W20260809 15:46:43.584079 26618 log.cc:924] Time spent T f9e7cadcd24d45fcbfa84209225e8f2d P 65c36d164b0540f099d592f38307ee6c: Append to log took a long time: real 0.494s user 0.014s sys 0.000s
I20260809 15:46:43.588969 26398 raft_consensus.cc:2726] T d3e94b7e50754a81b3d72a07476cb7b9 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260809 15:46:43.589660 26185 consensus_queue.cc:260] T f9e7cadcd24d45fcbfa84209225e8f2d P 67b3aeadfa7a432abe8af4d292e04ab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:43.636266 26399 consensus_queue.cc:1035] T df20671a7b874dbb98c0ef28371c1308 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260809 15:46:43.636606 24448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3be1e356b254d22b995be32f1c12f6f" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:43.608525 23228 leader_election.cc:302] T 6ae14a74a26f434db08c652e097385c8 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:43.638207 23228 leader_election.cc:302] T ae6d2c08a7a94f58ac8dd698e3fd456c P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:43.637862 26609 raft_consensus.cc:504] T cef3e7277d7d4760824a3436ed4a847d P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:43.639338 23228 leader_election.cc:302] T a3622f6e3e3641cea1960e3a31498bb7 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:43.640384 23228 leader_election.cc:302] T 0da4d5283c8a46878bd39b85e5c71f66 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:43.582737 23345 leader_election.cc:302] T e71a05ba1d4d42bd95c3f85f9313399e P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:46:43.533263 25863 raft_consensus.cc:504] T 472816994d814cb198b69e5248697440 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:43.552501 26563 leader_election.cc:288] T 30c1740ef6874df0a332fbd6fd4b03ce P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:43.606760 24455 raft_consensus.cc:1261] T 0c48a3049897452383257aa757834fb5 P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260809 15:46:43.609988 26431 raft_consensus.cc:3032] T 99bdb93503db4de4a1cfb0161ea0c07c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 LEADER]: Stepping down as leader of term 2
I20260809 15:46:43.654547 26431 raft_consensus.cc:729] T 99bdb93503db4de4a1cfb0161ea0c07c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Running, Role: LEADER
I20260809 15:46:43.606215 26611 consensus_queue.cc:237] T 998c6cee72cf40228c6dba064c0db2da P 65c36d164b0540f099d592f38307ee6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:43.611873 23274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae6d2c08a7a94f58ac8dd698e3fd456c" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:43.655555 26431 consensus_queue.cc:260] T 99bdb93503db4de4a1cfb0161ea0c07c P 67b3aeadfa7a432abe8af4d292e04ab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:43.660269 26431 raft_consensus.cc:3037] T 99bdb93503db4de4a1cfb0161ea0c07c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:43.576323 26214 raft_consensus.cc:482] T 28e17dcb5efe45b8b6a8c42da03b4819 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 15:46:43.626614 24406 leader_election.cc:334] T c0741ab44d99439abc3aa3bdd27ee160 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459): Timed out: RequestConsensusVote RPC to 127.20.202.65:35459 timed out after 1.990s (SENT)
I20260809 15:46:43.661356 24406 leader_election.cc:302] T c0741ab44d99439abc3aa3bdd27ee160 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:43.661062 26214 raft_consensus.cc:504] T 28e17dcb5efe45b8b6a8c42da03b4819 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:46:43.631481 23193 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 0da4d5283c8a46878bd39b85e5c71f66, TS 65c36d164b0540f099d592f38307ee6c -> TS 61e69695258f48baa984c9d395e8ce3f move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:43.572197 26386 consensus_queue.cc:1035] T 2db057fd10e149a9895dff6ef8c9f038 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.055s
I20260809 15:46:43.635026 23468 leader_election.cc:302] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:43.616074 26487 raft_consensus.cc:686] T cee02fff2c9c48a19f43eb358a89e7b0 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 LEADER]: Becoming Leader. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Running, Role: LEADER
I20260809 15:46:43.660590 26179 raft_consensus.cc:482] T 7660dfa2ef694927b8eda4100a2bcd9b P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:46:43.668373 26179 raft_consensus.cc:504] T 7660dfa2ef694927b8eda4100a2bcd9b P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:43.668705 26399 consensus_queue.cc:1035] T 0c48a3049897452383257aa757834fb5 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260809 15:46:43.669656 26609 leader_election.cc:288] T cef3e7277d7d4760824a3436ed4a847d P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
W20260809 15:46:43.635797 26473 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.519s user 0.000s sys 0.030s
I20260809 15:46:43.574146 26559 consensus_queue.cc:260] T e14d118b2eaa42aa8a7cd4025aba7128 P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
W20260809 15:46:43.642357 23345 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.68:34689, user_credentials={real_user=slave}} blocked reactor thread for 59699us
I20260809 15:46:43.621361 26506 raft_consensus.cc:504] T 16ca412a0dae4feebe85dd01bf1a9173 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:43.641976 23228 leader_election.cc:302] T 05ad7d8fdca64e3da527280ca5be9280 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
W20260809 15:46:43.696415 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 54507.8us
I20260809 15:46:43.644767 26185 raft_consensus.cc:3037] T f9e7cadcd24d45fcbfa84209225e8f2d P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:46:43.645074 26465 raft_consensus.cc:2726] T a3622f6e3e3641cea1960e3a31498bb7 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260809 15:46:43.645581 24452 raft_consensus.cc:1261] T e71a05ba1d4d42bd95c3f85f9313399e P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Refusing update from remote peer 67b3aeadfa7a432abe8af4d292e04ab9: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260809 15:46:43.613057 26469 consensus_queue.cc:237] T 563daf42d7784db195ea99c3cb21b370 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:43.647759 25863 leader_election.cc:288] T 472816994d814cb198b69e5248697440 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:43.648247 23283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0acc47ce21c3422698d1ef7277c3ed63" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:43.703505 23283 raft_consensus.cc:3037] T 0acc47ce21c3422698d1ef7277c3ed63 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Advancing to term 5
I20260809 15:46:43.704181 23522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e17dcb5efe45b8b6a8c42da03b4819" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:43.704772 23522 raft_consensus.cc:2454] T 28e17dcb5efe45b8b6a8c42da03b4819 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 4.
I20260809 15:46:43.704815 23521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e14d118b2eaa42aa8a7cd4025aba7128" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:43.704996 23529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "763e58d6ba644eea830a6c1664d39c9b" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:43.705502 23519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00a1881267a14ba8b78c81040f17a014" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:43.648268 26490 raft_consensus.cc:2726] T 6ae14a74a26f434db08c652e097385c8 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260809 15:46:43.706071 23519 raft_consensus.cc:2454] T 00a1881267a14ba8b78c81040f17a014 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 5.
I20260809 15:46:43.706554 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0741ab44d99439abc3aa3bdd27ee160" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:43.701923 26488 raft_consensus.cc:2726] T 05ad7d8fdca64e3da527280ca5be9280 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260809 15:46:43.705744 23529 raft_consensus.cc:2379] T 763e58d6ba644eea830a6c1664d39c9b P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 in current term 4: Already voted for candidate 65c36d164b0540f099d592f38307ee6c in this term.
I20260809 15:46:43.707674 23528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ced718c1a5c4da79cb8c1ebd1435b87" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:43.709087 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7715d71dbac447b9351fc5eb36c5e80" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:43.648718 26608 raft_consensus.cc:2781] T e71a05ba1d4d42bd95c3f85f9313399e P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260809 15:46:43.649209 26436 raft_consensus.cc:2726] T 0da4d5283c8a46878bd39b85e5c71f66 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260809 15:46:43.632175 26574 leader_election.cc:288] T c0741ab44d99439abc3aa3bdd27ee160 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:43.574970 23276 rpcz_store.cc:276] 0809 15:46:41.316727 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:41.942013 (+625286us) service_pool.cc:227] Handling call
0809 15:46:43.134268 (+1192255us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 15:46:43.269254 (+134986us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:43.574229 (+304975us) spinlock_profiling.cc:243] Waited 1.13 s on lock 0x7b54000bf250. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e921fa09 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:46:43.574392 (+ 163us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2606655232}
I20260809 15:46:43.644690 25885 raft_consensus.cc:2726] T ae6d2c08a7a94f58ac8dd698e3fd456c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260809 15:46:43.615098 26144 leader_election.cc:288] T 0a51c0baaa6e432997c9d3552af4f254 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
W20260809 15:46:43.570076 23116 rpcz_store.cc:269] 0809 15:46:28.386160 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:28.386278 (+ 118us) service_pool.cc:227] Handling call
0809 15:46:43.557185 (+15170907us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0809 15:46:38.704784 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0809 15:46:38.849362 (+144578us) write_op.cc:432] Acquiring schema lock in shared mode
0809 15:46:38.863035 (+ 13673us) write_op.cc:435] Acquired schema lock
0809 15:46:38.866312 (+ 3277us) tablet.cc:660] Decoding operations
0809 15:46:38.950618 (+ 84306us) write_op.cc:620] Acquiring the partition lock for write op
0809 15:46:38.971298 (+ 20680us) write_op.cc:641] Partition lock acquired for write op
0809 15:46:38.979616 (+ 8318us) tablet.cc:683] Acquiring locks for 1 operations
0809 15:46:39.094792 (+115176us) tablet.cc:699] Row locks acquired
0809 15:46:39.094832 (+ 40us) write_op.cc:260] PREPARE: finished
0809 15:46:39.095018 (+ 186us) write_op.cc:270] Start()
0809 15:46:39.095156 (+ 138us) write_op.cc:276] Timestamp: P: 1786290399094931 usec, L: 0
0809 15:46:39.095191 (+ 35us) op_driver.cc:331] REPLICATION: starting
0809 15:46:39.095860 (+ 669us) log.cc:842] Serialized 541 byte log entry
0809 15:46:40.133103 (+1037243us) op_driver.cc:444] REPLICATION: finished
0809 15:46:40.686080 (+552977us) write_op.cc:301] APPLY: starting
0809 15:46:40.686176 (+ 96us) tablet.cc:1365] starting BulkCheckPresence
0809 15:46:40.686301 (+ 125us) tablet.cc:1368] finished BulkCheckPresence
0809 15:46:40.686426 (+ 125us) tablet.cc:1370] starting ApplyRowOperation cycle
0809 15:46:40.686731 (+ 305us) tablet.cc:1381] finished ApplyRowOperation cycle
0809 15:46:40.686943 (+ 212us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0809 15:46:40.686998 (+ 55us) write_op.cc:312] APPLY: finished
0809 15:46:40.694291 (+ 7293us) log.cc:842] Serialized 23 byte log entry
0809 15:46:40.694442 (+ 151us) write_op.cc:489] Releasing partition, row and schema locks
0809 15:46:40.694536 (+ 94us) write_op.cc:454] Released schema lock
0809 15:46:40.694572 (+ 36us) write_op.cc:341] FINISH: Updating metrics
0809 15:46:40.694824 (+ 252us) write_op.cc:489] Releasing partition, row and schema locks
0809 15:46:40.694849 (+ 25us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":6496,"reported_tablets":10,"child_traces":[["op",{"apply.queue_time_us":489431,"apply.run_cpu_time_us":2018,"apply.run_wall_time_us":9019,"num_ops":1,"prepare.queue_time_us":460484,"prepare.run_cpu_time_us":10357,"prepare.run_wall_time_us":559600,"raft.queue_time_us":286032,"raft.run_cpu_time_us":4713,"raft.run_wall_time_us":236589,"replication_time_us":1033612,"thread_start_us":686677,"threads_started":4,"wal-append.queue_time_us":302546,"wal-append.run_cpu_time_us":40671,"wal-append.run_wall_time_us":2369053}]]}
W20260809 15:46:43.664768 23193 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 0da4d5283c8a46878bd39b85e5c71f66, TS 65c36d164b0540f099d592f38307ee6c -> TS 61e69695258f48baa984c9d395e8ce3f move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:43.666538 23115 catalog_manager.cc:5495] T 0da4d5283c8a46878bd39b85e5c71f66 P 67b3aeadfa7a432abe8af4d292e04ab9 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } health_report { overall_health: UNKNOWN } } }
I20260809 15:46:43.667423 26386 raft_consensus.cc:2726] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260809 15:46:43.610612 26594 leader_election.cc:288] T 00a1881267a14ba8b78c81040f17a014 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:43.668740 26487 consensus_queue.cc:237] T cee02fff2c9c48a19f43eb358a89e7b0 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:43.669764 24453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472816994d814cb198b69e5248697440" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:43.726715 24453 raft_consensus.cc:2454] T 472816994d814cb198b69e5248697440 P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 4.
I20260809 15:46:43.677491 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a51c0baaa6e432997c9d3552af4f254" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f"
I20260809 15:46:43.727978 24471 raft_consensus.cc:2379] T 0a51c0baaa6e432997c9d3552af4f254 P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d in current term 3: Already voted for candidate 61e69695258f48baa984c9d395e8ce3f in this term.
W20260809 15:46:43.677789 25639 consensus_queue.cc:1159] T e14d118b2eaa42aa8a7cd4025aba7128 P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "61e69695258f48baa984c9d395e8ce3f" responder_term: 5 status { last_received { term: 5 index: 7 } last_committed_idx: 7 last_received_current_leader { term: 5 index: 7 } }
I20260809 15:46:43.677192 26559 raft_consensus.cc:3037] T e14d118b2eaa42aa8a7cd4025aba7128 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Advancing to term 6
I20260809 15:46:43.677101 26405 consensus_queue.cc:1046] T 563daf42d7784db195ea99c3cb21b370 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.002s
W20260809 15:46:43.675000 26473 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.559s user 0.000s sys 0.030s
I20260809 15:46:43.731736 26405 raft_consensus.cc:3032] T 563daf42d7784db195ea99c3cb21b370 P c0f38c1908414566aa9115bb2cbdd85d [term 3 LEADER]: Stepping down as leader of term 3
I20260809 15:46:43.678789 24455 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0741ab44d99439abc3aa3bdd27ee160" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:43.732157 26405 raft_consensus.cc:729] T 563daf42d7784db195ea99c3cb21b370 P c0f38c1908414566aa9115bb2cbdd85d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
W20260809 15:46:43.680042 23395 consensus_meta.cc:287] T 297c82ed906d4b48b184985eed3de601 P c0f38c1908414566aa9115bb2cbdd85d: Time spent flushing consensus metadata: real 0.620s user 0.008s sys 0.004s
I20260809 15:46:43.733095 23395 raft_consensus.cc:2454] T 297c82ed906d4b48b184985eed3de601 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 6.
I20260809 15:46:43.733289 26405 consensus_queue.cc:260] T 563daf42d7784db195ea99c3cb21b370 P c0f38c1908414566aa9115bb2cbdd85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:43.734342 23395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0acc47ce21c3422698d1ef7277c3ed63" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:43.734853 26405 raft_consensus.cc:3037] T 563daf42d7784db195ea99c3cb21b370 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:46:43.680819 23273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cef3e7277d7d4760824a3436ed4a847d" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:43.661723 26398 consensus_queue.cc:1035] T c0741ab44d99439abc3aa3bdd27ee160 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.020s
W20260809 15:46:43.736927 23193 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 763e58d6ba644eea830a6c1664d39c9b, TS 67b3aeadfa7a432abe8af4d292e04ab9 -> TS 61e69695258f48baa984c9d395e8ce3f move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:46:43.737237 23193 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 763e58d6ba644eea830a6c1664d39c9b, TS 67b3aeadfa7a432abe8af4d292e04ab9 -> TS 61e69695258f48baa984c9d395e8ce3f move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:43.686946 26214 leader_election.cc:288] T 28e17dcb5efe45b8b6a8c42da03b4819 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:43.632457 26485 raft_consensus.cc:504] T 0c29937a80724ddb9d07a27534a146f3 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:43.675496 25650 raft_consensus.cc:3037] T 05ad7d8fdca64e3da527280ca5be9280 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:46:43.742698 23277 raft_consensus.cc:1226] T 16ca412a0dae4feebe85dd01bf1a9173 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 3. Current term is 4. Ops: []
I20260809 15:46:43.671196 26179 leader_election.cc:288] T 7660dfa2ef694927b8eda4100a2bcd9b P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:43.705824 23517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30c1740ef6874df0a332fbd6fd4b03ce" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:43.707201 23520 raft_consensus.cc:1226] T 99bdb93503db4de4a1cfb0161ea0c07c P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Rejecting Update request from peer 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260809 15:46:43.707209 26624 raft_consensus.cc:482] T 0da4d5283c8a46878bd39b85e5c71f66 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:46:43.711748 26608 raft_consensus.cc:482] T e71a05ba1d4d42bd95c3f85f9313399e P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:46:43.716580 23274 raft_consensus.cc:3037] T ae6d2c08a7a94f58ac8dd698e3fd456c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Advancing to term 5
I20260809 15:46:43.724154 23283 raft_consensus.cc:2454] T 0acc47ce21c3422698d1ef7277c3ed63 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 5.
I20260809 15:46:43.738245 23273 raft_consensus.cc:2454] T cef3e7277d7d4760824a3436ed4a847d P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 5.
I20260809 15:46:43.725508 23115 catalog_manager.cc:5495] T 16ca412a0dae4feebe85dd01bf1a9173 P 67b3aeadfa7a432abe8af4d292e04ab9 reported cstate change: term changed from 2 to 3, leader changed from 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) to 65c36d164b0540f099d592f38307ee6c (127.20.202.67). New cstate: current_term: 3 leader_uuid: "65c36d164b0540f099d592f38307ee6c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:43.742417 23282 raft_consensus.cc:3037] T 0c29937a80724ddb9d07a27534a146f3 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Advancing to term 5
I20260809 15:46:43.699529 24406 leader_election.cc:302] T e14d118b2eaa42aa8a7cd4025aba7128 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:43.743937 25885 raft_consensus.cc:482] T d3be1e356b254d22b995be32f1c12f6f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:43.736129 23276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e17dcb5efe45b8b6a8c42da03b4819" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
W20260809 15:46:43.755757 24406 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 56293.9us
I20260809 15:46:43.755105 23115 catalog_manager.cc:5495] T f9e7cadcd24d45fcbfa84209225e8f2d P 67b3aeadfa7a432abe8af4d292e04ab9 reported cstate change: term changed from 2 to 3, leader changed from c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66) to 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65). New cstate: current_term: 3 leader_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } health_report { overall_health: UNKNOWN } } }
I20260809 15:46:43.759367 24406 leader_election.cc:302] T 3d880de657e544468c82157769622a26 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:43.759889 23115 catalog_manager.cc:5495] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 67b3aeadfa7a432abe8af4d292e04ab9 reported cstate change: term changed from 1 to 2, leader changed from c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66) to 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65). New cstate: current_term: 2 leader_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260809 15:46:43.761660 23115 catalog_manager.cc:5495] T 0acc47ce21c3422698d1ef7277c3ed63 P 67b3aeadfa7a432abe8af4d292e04ab9 reported cstate change: term changed from 1 to 3, now has a pending config: VOTER 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65), VOTER c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66), VOTER 65c36d164b0540f099d592f38307ee6c (127.20.202.67), NON_VOTER 61e69695258f48baa984c9d395e8ce3f (127.20.202.68). New cstate: current_term: 3 leader_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } health_report { overall_health: UNKNOWN } } } pending_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } attrs { replace: true } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } } }
I20260809 15:46:43.763833 23470 leader_election.cc:302] T cef3e7277d7d4760824a3436ed4a847d P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:46:43.763916 23115 catalog_manager.cc:5495] T 4564dfc6d4094c1b9240b367ccecf3b1 P 67b3aeadfa7a432abe8af4d292e04ab9 reported cstate change: term changed from 1 to 3, leader changed from c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66) to 65c36d164b0540f099d592f38307ee6c (127.20.202.67). New cstate: current_term: 3 leader_uuid: "65c36d164b0540f099d592f38307ee6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } }
I20260809 15:46:43.753840 26608 raft_consensus.cc:3037] T e71a05ba1d4d42bd95c3f85f9313399e P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Advancing to term 5
I20260809 15:46:43.752946 26624 raft_consensus.cc:504] T 0da4d5283c8a46878bd39b85e5c71f66 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:43.767194 23470 leader_election.cc:302] T 0acc47ce21c3422698d1ef7277c3ed63 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters: c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:43.767738 26161 raft_consensus.cc:2781] T cef3e7277d7d4760824a3436ed4a847d P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260809 15:46:43.768603 23283 raft_consensus.cc:1261] T 0c48a3049897452383257aa757834fb5 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260809 15:46:43.791472 23283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f9d6565aa4c45a3b3f01210d0eb71f7" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:43.792025 23283 raft_consensus.cc:2454] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 3.
I20260809 15:46:43.767038 26594 raft_consensus.cc:482] T 0a51c0baaa6e432997c9d3552af4f254 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:43.769793 23276 raft_consensus.cc:2454] T 28e17dcb5efe45b8b6a8c42da03b4819 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 4.
I20260809 15:46:43.765748 25885 raft_consensus.cc:504] T d3be1e356b254d22b995be32f1c12f6f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:43.771361 25746 raft_consensus.cc:2726] T 3d880de657e544468c82157769622a26 P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 15:46:43.777660 26598 consensus_queue.cc:1035] T d3be1e356b254d22b995be32f1c12f6f P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260809 15:46:43.780990 23468 leader_election.cc:302] T 472816994d814cb198b69e5248697440 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c; no voters:
I20260809 15:46:43.782083 26488 consensus_queue.cc:1035] T e71a05ba1d4d42bd95c3f85f9313399e P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260809 15:46:43.775211 23228 leader_election.cc:302] T 4564dfc6d4094c1b9240b367ccecf3b1 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters: c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:43.784945 23274 raft_consensus.cc:2454] T ae6d2c08a7a94f58ac8dd698e3fd456c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 5.
I20260809 15:46:43.758792 26399 consensus_queue.cc:1046] T 16ca412a0dae4feebe85dd01bf1a9173 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.055s
I20260809 15:46:43.758373 26144 raft_consensus.cc:482] T a88421dba80d4d3388aa5eb62e23853b P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:43.766090 26179 raft_consensus.cc:2726] T e14d118b2eaa42aa8a7cd4025aba7128 P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260809 15:46:43.823426 26214 raft_consensus.cc:2781] T 472816994d814cb198b69e5248697440 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260809 15:46:43.810484 26485 leader_election.cc:288] T 0c29937a80724ddb9d07a27534a146f3 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:43.752003 23395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "763e58d6ba644eea830a6c1664d39c9b" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:43.825824 23470 leader_election.cc:302] T 28e17dcb5efe45b8b6a8c42da03b4819 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:46:43.828598 26574 raft_consensus.cc:2781] T 4564dfc6d4094c1b9240b367ccecf3b1 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader election won for term 4
I20260809 15:46:43.788764 26161 raft_consensus.cc:482] T cef3e7277d7d4760824a3436ed4a847d P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:46:43.864130 23470 leader_election.cc:302] T ae6d2c08a7a94f58ac8dd698e3fd456c P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters: c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:43.838980 25885 leader_election.cc:288] T d3be1e356b254d22b995be32f1c12f6f P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:43.840783 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3be1e356b254d22b995be32f1c12f6f" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:43.810233 26563 raft_consensus.cc:482] T 0c48a3049897452383257aa757834fb5 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 15:46:43.825934 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 50794.8us
I20260809 15:46:43.813079 26143 raft_consensus.cc:2781] T 0acc47ce21c3422698d1ef7277c3ed63 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Leader election won for term 5
I20260809 15:46:43.843804 26214 raft_consensus.cc:482] T 472816994d814cb198b69e5248697440 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:46:43.845400 26144 raft_consensus.cc:504] T a88421dba80d4d3388aa5eb62e23853b P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:43.866413 26398 raft_consensus.cc:2781] T 28e17dcb5efe45b8b6a8c42da03b4819 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260809 15:46:43.817085 26594 raft_consensus.cc:504] T 0a51c0baaa6e432997c9d3552af4f254 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
W20260809 15:46:43.878286 23399 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.762s user 0.003s sys 0.076s
I20260809 15:46:43.871234 23228 leader_election.cc:302] T 00a1881267a14ba8b78c81040f17a014 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:46:43.885582 23228 leader_election.cc:302] T c0741ab44d99439abc3aa3bdd27ee160 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c
I20260809 15:46:43.867074 26161 raft_consensus.cc:3037] T cef3e7277d7d4760824a3436ed4a847d P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Advancing to term 6
I20260809 15:46:43.819646 26609 consensus_queue.cc:1035] T 0c48a3049897452383257aa757834fb5 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.049s
I20260809 15:46:43.872072 25863 raft_consensus.cc:2781] T ae6d2c08a7a94f58ac8dd698e3fd456c P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Leader election won for term 5
I20260809 15:46:43.891523 23282 raft_consensus.cc:2454] T 0c29937a80724ddb9d07a27534a146f3 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 5.
I20260809 15:46:43.892505 23228 leader_election.cc:302] T 28e17dcb5efe45b8b6a8c42da03b4819 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:46:43.878733 26214 raft_consensus.cc:3037] T 472816994d814cb198b69e5248697440 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Advancing to term 5
I20260809 15:46:43.872645 26574 raft_consensus.cc:686] T 4564dfc6d4094c1b9240b367ccecf3b1 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 LEADER]: Becoming Leader. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Running, Role: LEADER
I20260809 15:46:43.872807 23276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0da4d5283c8a46878bd39b85e5c71f66" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:43.884107 26143 raft_consensus.cc:686] T 0acc47ce21c3422698d1ef7277c3ed63 P 65c36d164b0540f099d592f38307ee6c [term 5 LEADER]: Becoming Leader. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
W20260809 15:46:43.898370 23399 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.783s user 0.003s sys 0.076s
I20260809 15:46:43.888090 23395 raft_consensus.cc:2379] T 763e58d6ba644eea830a6c1664d39c9b P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 in current term 4: Already voted for candidate c0f38c1908414566aa9115bb2cbdd85d in this term.
I20260809 15:46:43.884558 26398 raft_consensus.cc:482] T 28e17dcb5efe45b8b6a8c42da03b4819 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:46:43.882360 26624 leader_election.cc:288] T 0da4d5283c8a46878bd39b85e5c71f66 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:43.884289 26436 raft_consensus.cc:482] T 7baa2492df0340c4a5ac241959ce1ac3 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:46:43.889482 26485 raft_consensus.cc:2781] T 00a1881267a14ba8b78c81040f17a014 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260809 15:46:43.902696 25863 raft_consensus.cc:686] T ae6d2c08a7a94f58ac8dd698e3fd456c P 65c36d164b0540f099d592f38307ee6c [term 5 LEADER]: Becoming Leader. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:46:43.877456 26563 raft_consensus.cc:504] T 0c48a3049897452383257aa757834fb5 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:43.921053 24406 leader_election.cc:302] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:46:43.924351 23520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16ca412a0dae4feebe85dd01bf1a9173" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:43.918484 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16ca412a0dae4feebe85dd01bf1a9173" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f"
I20260809 15:46:43.904493 26506 leader_election.cc:288] T 16ca412a0dae4feebe85dd01bf1a9173 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 election: Requested vote from peers 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:43.926882 23228 leader_election.cc:302] T 16ca412a0dae4feebe85dd01bf1a9173 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c
I20260809 15:46:43.915786 26488 raft_consensus.cc:2781] T 28e17dcb5efe45b8b6a8c42da03b4819 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260809 15:46:43.927510 23517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c29937a80724ddb9d07a27534a146f3" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:43.929265 23470 leader_election.cc:302] T 0c29937a80724ddb9d07a27534a146f3 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters: c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:43.920657 26398 raft_consensus.cc:3037] T 28e17dcb5efe45b8b6a8c42da03b4819 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Advancing to term 5
I20260809 15:46:43.933866 23517 raft_consensus.cc:2379] T 0c29937a80724ddb9d07a27534a146f3 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 in current term 5: Already voted for candidate 65c36d164b0540f099d592f38307ee6c in this term.
W20260809 15:46:43.926630 23280 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.777s user 0.020s sys 0.074s
I20260809 15:46:43.922204 26485 raft_consensus.cc:482] T 00a1881267a14ba8b78c81040f17a014 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:46:43.920904 23276 raft_consensus.cc:2454] T 0da4d5283c8a46878bd39b85e5c71f66 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 4.
I20260809 15:46:43.928107 26598 raft_consensus.cc:2781] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260809 15:46:43.931074 26490 raft_consensus.cc:2726] T 16ca412a0dae4feebe85dd01bf1a9173 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260809 15:46:43.931509 26488 raft_consensus.cc:482] T 28e17dcb5efe45b8b6a8c42da03b4819 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260809 15:46:43.955379 23280 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.806s user 0.020s sys 0.074s
I20260809 15:46:43.925055 26436 raft_consensus.cc:504] T 7baa2492df0340c4a5ac241959ce1ac3 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:43.935832 26399 raft_consensus.cc:2781] T 0c29937a80724ddb9d07a27534a146f3 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Leader election won for term 5
I20260809 15:46:43.968145 23470 leader_election.cc:302] T 0da4d5283c8a46878bd39b85e5c71f66 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:46:43.970037 23228 leader_election.cc:302] T 763e58d6ba644eea830a6c1664d39c9b P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:43.957763 26485 raft_consensus.cc:3037] T 00a1881267a14ba8b78c81040f17a014 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Advancing to term 6
W20260809 15:46:43.974970 23397 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.594s user 0.014s sys 0.011s
W20260809 15:46:43.975286 23397 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.594s user 0.014s sys 0.011s
I20260809 15:46:43.961036 23395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30c1740ef6874df0a332fbd6fd4b03ce" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:43.963449 26598 raft_consensus.cc:482] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Starting leader election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:46:43.967074 26488 raft_consensus.cc:3037] T 28e17dcb5efe45b8b6a8c42da03b4819 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Advancing to term 5
I20260809 15:46:44.009573 26608 raft_consensus.cc:504] T e71a05ba1d4d42bd95c3f85f9313399e P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:43.973981 26619 raft_consensus.cc:2781] T 0da4d5283c8a46878bd39b85e5c71f66 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260809 15:46:44.015507 26619 raft_consensus.cc:482] T 0da4d5283c8a46878bd39b85e5c71f66 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Starting leader election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:46:44.015882 26619 raft_consensus.cc:3037] T 0da4d5283c8a46878bd39b85e5c71f66 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Advancing to term 5
W20260809 15:46:44.027452 26654 log.cc:924] Time spent T 2424fd9bd7f14e508fd106cbfda786a4 P 67b3aeadfa7a432abe8af4d292e04ab9: Append to log took a long time: real 0.051s user 0.001s sys 0.000s
I20260809 15:46:44.028517 23280 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53554 (request call id 1331) took 1465 ms. Trace:
I20260809 15:46:44.029491 26436 leader_election.cc:288] T 7baa2492df0340c4a5ac241959ce1ac3 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:44.028919 23280 rpcz_store.cc:276] 0809 15:46:42.563542 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:43.021882 (+458340us) service_pool.cc:227] Handling call
0809 15:46:43.063000 (+ 41118us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 15:46:43.148609 (+ 85609us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:46:43.148637 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 15:46:43.148991 (+ 354us) log.cc:842] Serialized 26 byte log entry
0809 15:46:43.967501 (+818510us) raft_consensus.cc:1631] Marking committed up to 1
0809 15:46:43.967541 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:43.968906 (+ 1365us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 15:46:44.028301 (+ 59395us) raft_consensus.cc:1671] finished
0809 15:46:44.028324 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:44.028462 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":107841,"raft.queue_time_us":96349,"raft.run_cpu_time_us":167,"raft.run_wall_time_us":28348,"thread_start_us":817885,"threads_started":1,"wal-append.queue_time_us":821206}
I20260809 15:46:44.031204 23517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7baa2492df0340c4a5ac241959ce1ac3" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:44.031843 23517 raft_consensus.cc:2454] T 7baa2492df0340c4a5ac241959ce1ac3 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 2.
I20260809 15:46:44.033589 23228 leader_election.cc:302] T 7baa2492df0340c4a5ac241959ce1ac3 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:46:44.034404 26436 raft_consensus.cc:2781] T 7baa2492df0340c4a5ac241959ce1ac3 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260809 15:46:44.034751 26436 raft_consensus.cc:482] T 7baa2492df0340c4a5ac241959ce1ac3 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Starting leader election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:46:44.035061 26436 raft_consensus.cc:3037] T 7baa2492df0340c4a5ac241959ce1ac3 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:44.039678 26144 leader_election.cc:288] T a88421dba80d4d3388aa5eb62e23853b P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:44.041738 26436 raft_consensus.cc:504] T 7baa2492df0340c4a5ac241959ce1ac3 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:44.045974 26563 leader_election.cc:288] T 0c48a3049897452383257aa757834fb5 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:43.975092 26399 raft_consensus.cc:686] T 0c29937a80724ddb9d07a27534a146f3 P 65c36d164b0540f099d592f38307ee6c [term 5 LEADER]: Becoming Leader. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:46:44.048281 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c48a3049897452383257aa757834fb5" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:44.049058 26399 consensus_queue.cc:237] T 0c29937a80724ddb9d07a27534a146f3 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:44.051556 26594 leader_election.cc:288] T 0a51c0baaa6e432997c9d3552af4f254 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:44.051712 26436 leader_election.cc:288] T 7baa2492df0340c4a5ac241959ce1ac3 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 3 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:43.989943 26598 raft_consensus.cc:3037] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:46:44.065717 23286 raft_consensus.cc:1261] T f9e7cadcd24d45fcbfa84209225e8f2d P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260809 15:46:43.971952 26490 raft_consensus.cc:2726] T 763e58d6ba644eea830a6c1664d39c9b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260809 15:46:43.972649 25863 consensus_queue.cc:237] T ae6d2c08a7a94f58ac8dd698e3fd456c P 65c36d164b0540f099d592f38307ee6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:43.978814 26143 consensus_queue.cc:237] T 0acc47ce21c3422698d1ef7277c3ed63 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:44.094286 23521 raft_consensus.cc:2454] T e14d118b2eaa42aa8a7cd4025aba7128 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 6.
W20260809 15:46:44.096913 23468 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.68:34689, user_credentials={real_user=slave}} blocked reactor thread for 90289.5us
I20260809 15:46:44.097582 23526 raft_consensus.cc:2396] T f7715d71dbac447b9351fc5eb36c5e80 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260809 15:46:44.148960 23517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c48a3049897452383257aa757834fb5" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:44.150712 23228 leader_election.cc:302] T 0c48a3049897452383257aa757834fb5 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c
I20260809 15:46:44.154987 23520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7baa2492df0340c4a5ac241959ce1ac3" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:44.155606 23520 raft_consensus.cc:3037] T 7baa2492df0340c4a5ac241959ce1ac3 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:44.159593 26580 raft_consensus.cc:482] T 3924b648461e463b9d7925e5671d68ae P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:44.004846 25639 raft_consensus.cc:3032] T 16ca412a0dae4feebe85dd01bf1a9173 P 65c36d164b0540f099d592f38307ee6c [term 3 LEADER]: Stepping down as leader of term 3
I20260809 15:46:44.159595 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a51c0baaa6e432997c9d3552af4f254" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:43.961309 26574 consensus_queue.cc:237] T 4564dfc6d4094c1b9240b367ccecf3b1 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:44.160069 26580 raft_consensus.cc:504] T 3924b648461e463b9d7925e5671d68ae P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
W20260809 15:46:44.163784 23470 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 130861us
I20260809 15:46:44.165145 24471 raft_consensus.cc:2454] T 0a51c0baaa6e432997c9d3552af4f254 P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 3.
I20260809 15:46:44.004675 23395 raft_consensus.cc:2454] T 30c1740ef6874df0a332fbd6fd4b03ce P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 3.
I20260809 15:46:44.171650 23223 leader_election.cc:302] T 0a51c0baaa6e432997c9d3552af4f254 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
W20260809 15:46:44.187479 26638 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.835s user 0.000s sys 0.029s
I20260809 15:46:44.193567 23228 leader_election.cc:302] T 30c1740ef6874df0a332fbd6fd4b03ce P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c
I20260809 15:46:44.164736 25639 raft_consensus.cc:729] T 16ca412a0dae4feebe85dd01bf1a9173 P 65c36d164b0540f099d592f38307ee6c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:46:44.172122 25746 raft_consensus.cc:482] T 0a51c0baaa6e432997c9d3552af4f254 P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 15:46:44.201849 23400 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.070s user 0.002s sys 0.033s
W20260809 15:46:44.208930 23400 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.078s user 0.002s sys 0.033s
I20260809 15:46:44.184818 26608 leader_election.cc:288] T e71a05ba1d4d42bd95c3f85f9313399e P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
W20260809 15:46:44.213698 23394 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.784s user 0.001s sys 0.044s
I20260809 15:46:44.191658 26185 raft_consensus.cc:2781] T 0a51c0baaa6e432997c9d3552af4f254 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260809 15:46:44.223855 23394 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.794s user 0.002s sys 0.044s
W20260809 15:46:44.191144 26523 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.238s user 0.005s sys 0.031s
W20260809 15:46:44.218192 23396 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.683s user 0.017s sys 0.010s
W20260809 15:46:44.226089 23396 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.691s user 0.017s sys 0.010s
W20260809 15:46:44.218536 23351 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.094s user 0.010s sys 0.022s
I20260809 15:46:44.209242 26436 raft_consensus.cc:2781] T 30c1740ef6874df0a332fbd6fd4b03ce P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260809 15:46:44.207636 26638 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.855s user 0.000s sys 0.029s
W20260809 15:46:44.226994 23351 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.102s user 0.011s sys 0.022s
W20260809 15:46:44.218429 23393 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.737s user 0.003s sys 0.028s
W20260809 15:46:44.225744 23353 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.105s user 0.008s sys 0.024s
W20260809 15:46:44.232437 23353 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.112s user 0.008s sys 0.024s
W20260809 15:46:44.218075 26279 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.099s user 0.026s sys 0.009s
W20260809 15:46:44.231595 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 1.10746e+06us
I20260809 15:46:44.236152 23351 leader_election.cc:302] T a3622f6e3e3641cea1960e3a31498bb7 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
W20260809 15:46:44.231078 26569 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.806s user 0.004s sys 0.020s
W20260809 15:46:44.230078 23404 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.741s user 0.011s sys 0.013s
W20260809 15:46:44.242944 23404 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.754s user 0.012s sys 0.013s
W20260809 15:46:44.227692 26523 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.274s user 0.005s sys 0.031s
I20260809 15:46:44.225584 26185 raft_consensus.cc:482] T 0a51c0baaa6e432997c9d3552af4f254 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:46:44.224370 26594 raft_consensus.cc:482] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
W20260809 15:46:44.232805 23353 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 1.11245e+06us
I20260809 15:46:44.253707 23520 raft_consensus.cc:2454] T 7baa2492df0340c4a5ac241959ce1ac3 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 3.
I20260809 15:46:44.257630 23351 leader_election.cc:302] T f9e7cadcd24d45fcbfa84209225e8f2d P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
W20260809 15:46:44.240324 23393 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.759s user 0.003s sys 0.029s
I20260809 15:46:44.226162 23395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7baa2492df0340c4a5ac241959ce1ac3" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:44.227711 26436 raft_consensus.cc:482] T 30c1740ef6874df0a332fbd6fd4b03ce P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:44.215512 25746 raft_consensus.cc:504] T 0a51c0baaa6e432997c9d3552af4f254 P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
W20260809 15:46:44.243854 26279 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.125s user 0.027s sys 0.009s
W20260809 15:46:44.267334 26527 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.748s user 0.022s sys 0.029s
W20260809 15:46:44.240718 26569 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.819s user 0.004s sys 0.020s
W20260809 15:46:44.295459 24410 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.321s user 0.023s sys 0.027s
W20260809 15:46:44.301548 24410 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.327s user 0.023s sys 0.027s
W20260809 15:46:44.301841 24410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 1.38558e+06us
I20260809 15:46:44.260579 26185 raft_consensus.cc:3037] T 0a51c0baaa6e432997c9d3552af4f254 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:46:44.260301 26144 raft_consensus.cc:2726] T f9e7cadcd24d45fcbfa84209225e8f2d P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260809 15:46:44.260311 26608 raft_consensus.cc:2726] T a3622f6e3e3641cea1960e3a31498bb7 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260809 15:46:44.301321 26527 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.782s user 0.023s sys 0.029s
I20260809 15:46:44.307952 24410 leader_election.cc:302] T 472816994d814cb198b69e5248697440 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c
I20260809 15:46:44.292179 26436 raft_consensus.cc:3037] T 30c1740ef6874df0a332fbd6fd4b03ce P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Advancing to term 4
W20260809 15:46:44.349784 26646 log.cc:924] Time spent T 07fcabb569414bddbd39df6cedd24e64 P c0f38c1908414566aa9115bb2cbdd85d: Append to log took a long time: real 0.118s user 0.002s sys 0.000s
I20260809 15:46:44.350903 23399 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55320 (request call id 1259) took 1323 ms. Trace:
I20260809 15:46:44.235647 26161 raft_consensus.cc:504] T cef3e7277d7d4760824a3436ed4a847d P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:44.350935 23517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3aa6a9a52804df0ba9b0ca3c82f38c3" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:44.352388 23517 raft_consensus.cc:2379] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d in current term 3: Already voted for candidate 65c36d164b0540f099d592f38307ee6c in this term.
I20260809 15:46:44.352667 26161 leader_election.cc:288] T cef3e7277d7d4760824a3436ed4a847d P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 6 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:44.351251 23399 rpcz_store.cc:276] 0809 15:46:43.027246 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:43.098952 (+ 71706us) service_pool.cc:227] Handling call
0809 15:46:43.099309 (+ 357us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 15:46:43.099509 (+ 200us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:46:43.099533 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 15:46:43.110394 (+ 10861us) log.cc:842] Serialized 26 byte log entry
0809 15:46:43.907957 (+797563us) raft_consensus.cc:1631] Marking committed up to 2
0809 15:46:43.907994 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:43.908835 (+ 841us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 15:46:44.349290 (+440455us) spinlock_profiling.cc:243] Waited 63.5 ms on lock 0x7b44000232cc. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e613cf11 00007f34e620aa95 00007f34e620b4b2 00007f34e8eb3511 00007f34e8eaee1a 00007f34e8ea2bee 00007f34e8ea1f36 00007f34e8ea1b58 00007f34e8eab551 00007f34e8eab509 00007f34e8eab499
0809 15:46:44.350560 (+ 1270us) raft_consensus.cc:1671] finished
0809 15:46:44.350647 (+ 87us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:44.350846 (+ 199us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":146069376,"thread_start_us":792027,"threads_started":1,"wal-append.queue_time_us":1095882}
I20260809 15:46:44.354714 23522 raft_consensus.cc:1226] T cef3e7277d7d4760824a3436ed4a847d P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Rejecting Update request from peer c0f38c1908414566aa9115bb2cbdd85d for earlier term 5. Current term is 6. Ops: [5.3-5.3]
W20260809 15:46:44.306886 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 47983.7us
I20260809 15:46:44.234949 25639 consensus_queue.cc:260] T 16ca412a0dae4feebe85dd01bf1a9173 P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:44.351559 23528 raft_consensus.cc:1226] T 7baa2492df0340c4a5ac241959ce1ac3 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Rejecting Update request from peer c0f38c1908414566aa9115bb2cbdd85d for earlier term 2. Current term is 3. Ops: [2.2-2.2]
W20260809 15:46:44.356528 23402 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.603s user 0.014s sys 0.019s
W20260809 15:46:44.355777 24410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 47904.6us
W20260809 15:46:44.368441 23402 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.617s user 0.015s sys 0.020s
I20260809 15:46:44.361800 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e17dcb5efe45b8b6a8c42da03b4819" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:44.284977 26594 raft_consensus.cc:504] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
W20260809 15:46:44.327885 23392 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.856s user 0.018s sys 0.040s
W20260809 15:46:44.370676 24475 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.758s user 0.009s sys 0.022s
I20260809 15:46:44.340444 26506 raft_consensus.cc:482] T c5c2e425a74b4e64ae70b5def750434c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:44.365036 25885 raft_consensus.cc:2726] T c0741ab44d99439abc3aa3bdd27ee160 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260809 15:46:44.367583 24448 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.665s user 0.015s sys 0.011s
W20260809 15:46:44.375602 24475 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.763s user 0.009s sys 0.022s
I20260809 15:46:44.378150 23528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a88421dba80d4d3388aa5eb62e23853b" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:44.365321 26580 leader_election.cc:288] T 3924b648461e463b9d7925e5671d68ae P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:44.366371 26279 raft_consensus.cc:2781] T 472816994d814cb198b69e5248697440 P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260809 15:46:44.379489 23353 leader_election.cc:302] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c
I20260809 15:46:44.333717 26650 consensus_queue.cc:1035] T 0acc47ce21c3422698d1ef7277c3ed63 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.042s
W20260809 15:46:44.378477 24448 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.682s user 0.016s sys 0.011s
I20260809 15:46:44.355054 26144 consensus_queue.cc:1046] T ae6d2c08a7a94f58ac8dd698e3fd456c P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.061s
W20260809 15:46:44.440191 23270 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.317s user 0.008s sys 0.046s
W20260809 15:46:44.451815 23270 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.328s user 0.009s sys 0.046s
I20260809 15:46:44.378859 26420 raft_consensus.cc:2726] T c0741ab44d99439abc3aa3bdd27ee160 P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260809 15:46:44.377290 23395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cef3e7277d7d4760824a3436ed4a847d" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
W20260809 15:46:44.375221 23392 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.903s user 0.018s sys 0.040s
I20260809 15:46:44.448688 26465 raft_consensus.cc:482] T 73f6f44c259547f1b05a3a434616fa54 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 15:46:44.444187 26477 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.751s user 0.014s sys 0.022s
W20260809 15:46:44.450764 26469 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.666s user 0.018s sys 0.018s
I20260809 15:46:44.373554 26214 raft_consensus.cc:504] T 472816994d814cb198b69e5248697440 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:44.380954 26527 consensus_queue.cc:1035] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.092s
I20260809 15:46:44.453395 26464 raft_consensus.cc:482] T 3296ef50be254fc29cfc77d22a8f7a23 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
I20260809 15:46:44.452488 24452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3924b648461e463b9d7925e5671d68ae" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:44.483188 23270 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55434 (request call id 157) took 3165 ms. Trace:
I20260809 15:46:44.380744 26506 raft_consensus.cc:504] T c5c2e425a74b4e64ae70b5def750434c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:46:44.496259 26663 log.cc:924] Time spent T 953eea268e17491ca76f3a555860a512 P c0f38c1908414566aa9115bb2cbdd85d: Append to log took a long time: real 0.188s user 0.000s sys 0.000s
I20260809 15:46:44.497081 23400 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55320 (request call id 1258) took 1501 ms. Trace:
I20260809 15:46:44.497560 23397 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55320 (request call id 1250) took 1575 ms. Trace:
I20260809 15:46:44.497879 23397 rpcz_store.cc:276] 0809 15:46:42.922178 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:43.044562 (+122384us) service_pool.cc:227] Handling call
0809 15:46:43.379218 (+334656us) spinlock_profiling.cc:243] Waited 964 us on lock 0x7b58001d7c98. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e613cf11 00007f34e62c0851 00007f34e62c0702 00007f34ef1b0b09 00007f34ee0ec730 00007f34ee1b279b 00007f34ee1b2821 00007f34ee1762b0 00007f34ee1617f9 00007f34ee1612a2 00007f34e7847684
0809 15:46:43.379626 (+ 408us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 15:46:43.380354 (+ 728us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:46:43.380384 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 15:46:43.380785 (+ 401us) log.cc:842] Serialized 26 byte log entry
0809 15:46:43.975637 (+594852us) raft_consensus.cc:1631] Marking committed up to 2
0809 15:46:43.975675 (+ 38us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:43.975742 (+ 67us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 15:46:44.497282 (+521540us) raft_consensus.cc:1671] finished
0809 15:46:44.497303 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:44.497507 (+ 204us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":118,"raft.run_cpu_time_us":77,"raft.run_wall_time_us":78,"spinlock_wait_cycles":2216832,"thread_start_us":594589,"threads_started":1,"wal-append.queue_time_us":866633}
I20260809 15:46:44.499568 23397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3924b648461e463b9d7925e5671d68ae" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:44.500140 23397 raft_consensus.cc:2454] T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 6.
I20260809 15:46:44.500451 23399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ced718c1a5c4da79cb8c1ebd1435b87" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:44.501022 26594 leader_election.cc:288] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:44.501490 23399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7715d71dbac447b9351fc5eb36c5e80" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:44.500871 23397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472816994d814cb198b69e5248697440" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:44.502736 23397 raft_consensus.cc:2454] T 472816994d814cb198b69e5248697440 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 4.
I20260809 15:46:44.495412 26506 leader_election.cc:288] T c5c2e425a74b4e64ae70b5def750434c P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
W20260809 15:46:44.481475 26398 consensus_meta.cc:287] T 28e17dcb5efe45b8b6a8c42da03b4819 P 65c36d164b0540f099d592f38307ee6c: Time spent flushing consensus metadata: real 0.531s user 0.007s sys 0.003s
I20260809 15:46:44.503787 26398 raft_consensus.cc:504] T 28e17dcb5efe45b8b6a8c42da03b4819 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:44.504256 26619 raft_consensus.cc:504] T 0da4d5283c8a46878bd39b85e5c71f66 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:44.505183 25639 raft_consensus.cc:3037] T 16ca412a0dae4feebe85dd01bf1a9173 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:46:44.505704 26398 leader_election.cc:288] T 28e17dcb5efe45b8b6a8c42da03b4819 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 5 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:44.506134 26619 leader_election.cc:288] T 0da4d5283c8a46878bd39b85e5c71f66 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 5 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:44.506282 23519 raft_consensus.cc:1226] T 0da4d5283c8a46878bd39b85e5c71f66 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Rejecting Update request from peer c0f38c1908414566aa9115bb2cbdd85d for earlier term 4. Current term is 5. Ops: []
I20260809 15:46:44.507977 23527 raft_consensus.cc:2379] T 28e17dcb5efe45b8b6a8c42da03b4819 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d in current term 5: Already voted for candidate 65c36d164b0540f099d592f38307ee6c in this term.
I20260809 15:46:44.502847 26527 consensus_queue.cc:1035] T 0da4d5283c8a46878bd39b85e5c71f66 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260809 15:46:44.482513 26469 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.700s user 0.018s sys 0.018s
I20260809 15:46:44.488541 23270 rpcz_store.cc:276] 0809 15:46:41.319231 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:42.024301 (+705070us) service_pool.cc:227] Handling call
0809 15:46:42.075045 (+ 50744us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 15:46:42.075399 (+ 354us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:46:42.075425 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 15:46:42.075793 (+ 368us) log.cc:842] Serialized 26 byte log entry
0809 15:46:42.446567 (+370774us) raft_consensus.cc:1631] Marking committed up to 5
0809 15:46:42.603864 (+157297us) log.cc:842] Serialized 14 byte log entry
0809 15:46:43.123427 (+519563us) spinlock_profiling.cc:243] Waited 1.64 ms on lock 0x7b180044c0e0. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34ef1978b1 00007f34ef1a4661 00007f34e626f4ca 00007f34ef950f99 00007f34e6133990 00007f34e62f5864 00007f34e62f5155 00007f34e92183d0 00007f34e922a576 00007f34e923b0d7 00007f34e923b058
0809 15:46:44.452617 (+1329190us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:44.452691 (+ 74us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 15:46:44.452744 (+ 53us) raft_consensus.cc:1671] finished
0809 15:46:44.483010 (+ 30266us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:44.483136 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":38095,"prepare.queue_time_us":151219,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":41,"raft.queue_time_us":535693,"raft.run_cpu_time_us":274,"raft.run_wall_time_us":20990,"spinlock_wait_cycles":3768064,"thread_start_us":1776769,"threads_started":3,"wal-append.queue_time_us":515336,"wal-append.run_cpu_time_us":2753,"wal-append.run_wall_time_us":1009120}
I20260809 15:46:44.510864 23399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0da4d5283c8a46878bd39b85e5c71f66" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
W20260809 15:46:44.480948 26477 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.788s user 0.014s sys 0.022s
I20260809 15:46:44.448777 26279 raft_consensus.cc:482] T 472816994d814cb198b69e5248697440 P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:46:44.497427 23400 rpcz_store.cc:276] 0809 15:46:42.995976 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:43.095744 (+ 99768us) service_pool.cc:227] Handling call
0809 15:46:43.115101 (+ 19357us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 15:46:43.123325 (+ 8224us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:46:43.123385 (+ 60us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 15:46:43.123844 (+ 459us) log.cc:842] Serialized 26 byte log entry
0809 15:46:44.223040 (+1099196us) raft_consensus.cc:1631] Marking committed up to 2
0809 15:46:44.223084 (+ 44us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:44.223147 (+ 63us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 15:46:44.496885 (+273738us) raft_consensus.cc:1671] finished
0809 15:46:44.496905 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:44.497026 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":33861,"raft.run_cpu_time_us":73,"raft.run_wall_time_us":73,"thread_start_us":1091604,"threads_started":1,"wal-append.queue_time_us":1363533}
W20260809 15:46:44.486348 26485 consensus_meta.cc:287] T 00a1881267a14ba8b78c81040f17a014 P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent flushing consensus metadata: real 0.509s user 0.006s sys 0.002s
W20260809 15:46:44.511577 24463 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.241s user 0.011s sys 0.038s
I20260809 15:46:44.516268 23397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f9d6565aa4c45a3b3f01210d0eb71f7" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
W20260809 15:46:44.517663 23273 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.744s user 0.015s sys 0.025s
W20260809 15:46:44.517952 23273 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.744s user 0.015s sys 0.025s
I20260809 15:46:44.518249 23400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a51c0baaa6e432997c9d3552af4f254" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:44.518826 23400 raft_consensus.cc:2454] T 0a51c0baaa6e432997c9d3552af4f254 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 3.
I20260809 15:46:44.519641 23400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cef3e7277d7d4760824a3436ed4a847d" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:44.508916 26598 raft_consensus.cc:504] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:44.514109 26279 raft_consensus.cc:3037] T 472816994d814cb198b69e5248697440 P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Advancing to term 5
I20260809 15:46:44.482054 26465 raft_consensus.cc:504] T 73f6f44c259547f1b05a3a434616fa54 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:44.509586 26464 raft_consensus.cc:504] T 3296ef50be254fc29cfc77d22a8f7a23 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
W20260809 15:46:44.513156 24455 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.674s user 0.012s sys 0.026s
I20260809 15:46:44.522271 23400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e17dcb5efe45b8b6a8c42da03b4819" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:44.523546 23395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7baa2492df0340c4a5ac241959ce1ac3" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:44.525046 23399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3be1e356b254d22b995be32f1c12f6f" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:44.534775 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e4ffcf83c234ae7910ddef3ec0b4bda" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:44.536427 23397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7660dfa2ef694927b8eda4100a2bcd9b" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:44.517858 26485 raft_consensus.cc:504] T 00a1881267a14ba8b78c81040f17a014 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:44.558966 23399 raft_consensus.cc:2454] T d3be1e356b254d22b995be32f1c12f6f P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 4.
I20260809 15:46:44.561686 23400 raft_consensus.cc:2396] T 28e17dcb5efe45b8b6a8c42da03b4819 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260809 15:46:44.566511 23397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e17dcb5efe45b8b6a8c42da03b4819" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:44.567504 23397 raft_consensus.cc:2396] T 28e17dcb5efe45b8b6a8c42da03b4819 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260809 15:46:44.568606 23399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c29937a80724ddb9d07a27534a146f3" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:44.569378 23399 raft_consensus.cc:2379] T 0c29937a80724ddb9d07a27534a146f3 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 in current term 5: Already voted for candidate c0f38c1908414566aa9115bb2cbdd85d in this term.
W20260809 15:46:44.558993 23274 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.627s user 0.021s sys 0.020s
I20260809 15:46:44.559777 23395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00a1881267a14ba8b78c81040f17a014" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
W20260809 15:46:44.569799 26611 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.861s user 0.004s sys 0.024s
I20260809 15:46:44.580888 23397 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55320 (request call id 1271) took 1090 ms. Trace:
I20260809 15:46:44.581342 23397 rpcz_store.cc:276] 0809 15:46:43.490068 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:44.568030 (+1077962us) service_pool.cc:227] Handling call
0809 15:46:44.580436 (+ 12406us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:44.580598 (+ 162us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:46:44.580623 (+ 25us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 15:46:44.580660 (+ 37us) raft_consensus.cc:1631] Marking committed up to 2
0809 15:46:44.580687 (+ 27us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:44.580735 (+ 48us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:44.580841 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:46:44.583370 23397 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55320 (request call id 1272) took 1078 ms. Trace:
I20260809 15:46:44.581712 23400 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55320 (request call id 1270) took 1091 ms. Trace:
I20260809 15:46:44.583753 23397 rpcz_store.cc:276] 0809 15:46:43.505059 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:44.582609 (+1077550us) service_pool.cc:227] Handling call
0809 15:46:44.582864 (+ 255us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:44.583010 (+ 146us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:46:44.583030 (+ 20us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 15:46:44.583064 (+ 34us) raft_consensus.cc:1631] Marking committed up to 3
0809 15:46:44.583088 (+ 24us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:44.583129 (+ 41us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:44.583320 (+ 191us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:46:44.584523 23400 rpcz_store.cc:276] 0809 15:46:43.489817 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:44.565955 (+1076138us) service_pool.cc:227] Handling call
0809 15:46:44.581170 (+ 15215us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:44.581325 (+ 155us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:46:44.581351 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 15:46:44.581388 (+ 37us) raft_consensus.cc:1631] Marking committed up to 2
0809 15:46:44.581423 (+ 35us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:44.581467 (+ 44us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:44.581559 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 15:46:44.569357 23285 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.971s user 0.000s sys 0.028s
W20260809 15:46:44.588160 23285 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.990s user 0.001s sys 0.028s
W20260809 15:46:44.560372 24455 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.721s user 0.012s sys 0.026s
W20260809 15:46:44.581789 23274 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.650s user 0.021s sys 0.021s
I20260809 15:46:44.596170 23399 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55320 (request call id 1273) took 1040 ms. Trace:
I20260809 15:46:44.596522 23399 rpcz_store.cc:276] 0809 15:46:43.555800 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:44.582951 (+1027151us) service_pool.cc:227] Handling call
0809 15:46:44.595765 (+ 12814us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:44.595911 (+ 146us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:46:44.595934 (+ 23us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 15:46:44.595967 (+ 33us) raft_consensus.cc:1631] Marking committed up to 2
0809 15:46:44.595995 (+ 28us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:44.596041 (+ 46us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:44.596125 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:46:44.565430 26667 raft_consensus.cc:3037] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Advancing to term 3
W20260809 15:46:44.569216 26487 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.790s user 0.020s sys 0.007s
W20260809 15:46:44.521839 24463 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.259s user 0.011s sys 0.038s
W20260809 15:46:44.585188 26611 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.876s user 0.004s sys 0.024s
I20260809 15:46:44.607430 24452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472816994d814cb198b69e5248697440" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f"
I20260809 15:46:44.607722 23270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f9d6565aa4c45a3b3f01210d0eb71f7" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:44.608731 23270 raft_consensus.cc:3037] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:46:44.601837 23397 raft_consensus.cc:1261] T f9e7cadcd24d45fcbfa84209225e8f2d P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
W20260809 15:46:44.603826 26488 consensus_meta.cc:287] T 28e17dcb5efe45b8b6a8c42da03b4819 P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent flushing consensus metadata: real 0.552s user 0.003s sys 0.007s
I20260809 15:46:44.600715 26214 leader_election.cc:288] T 472816994d814cb198b69e5248697440 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 5 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:44.605159 23399 raft_consensus.cc:1226] T e71a05ba1d4d42bd95c3f85f9313399e P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Rejecting Update request from peer 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 4. Current term is 5. Ops: []
W20260809 15:46:44.604905 26487 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.826s user 0.021s sys 0.007s
I20260809 15:46:44.612301 23397 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55320 (request call id 1275) took 1031 ms. Trace:
W20260809 15:46:44.633009 23282 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.652s user 0.004s sys 0.023s
W20260809 15:46:44.670148 23282 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.689s user 0.004s sys 0.024s
I20260809 15:46:44.670102 23397 rpcz_store.cc:276] 0809 15:46:43.581281 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:44.584976 (+1003695us) service_pool.cc:227] Handling call
0809 15:46:44.601484 (+ 16508us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:44.612033 (+ 10549us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:44.612258 (+ 225us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 15:46:44.672909 24404 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.594s user 0.002s sys 0.031s
W20260809 15:46:44.673287 24404 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.594s user 0.002s sys 0.031s
I20260809 15:46:44.678818 23270 raft_consensus.cc:2454] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 4.
I20260809 15:46:44.680621 24406 leader_election.cc:302] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
W20260809 15:46:44.681614 23115 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.790s user 0.009s sys 0.044s
W20260809 15:46:44.682063 23115 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.790s user 0.009s sys 0.044s
W20260809 15:46:44.674768 26143 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.596s user 0.020s sys 0.016s
I20260809 15:46:44.672770 26490 raft_consensus.cc:2726] T 0c48a3049897452383257aa757834fb5 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260809 15:46:44.696166 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e71a05ba1d4d42bd95c3f85f9313399e" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f"
W20260809 15:46:44.725787 23277 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.975s user 0.008s sys 0.031s
W20260809 15:46:44.735035 23277 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.984s user 0.009s sys 0.031s
I20260809 15:46:44.733852 23400 raft_consensus.cc:1261] T d3be1e356b254d22b995be32f1c12f6f P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Refusing update from remote peer 61e69695258f48baa984c9d395e8ce3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20260809 15:46:44.733554 25863 raft_consensus.cc:482] T 05ad7d8fdca64e3da527280ca5be9280 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:44.737401 23399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f9d6565aa4c45a3b3f01210d0eb71f7" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:44.631397 26488 raft_consensus.cc:504] T 28e17dcb5efe45b8b6a8c42da03b4819 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:46:44.695801 26143 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.617s user 0.020s sys 0.016s
I20260809 15:46:44.691956 26581 raft_consensus.cc:2781] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260809 15:46:44.739804 23397 raft_consensus.cc:1261] T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Refusing update from remote peer 61e69695258f48baa984c9d395e8ce3f: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260809 15:46:44.751884 23518 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:56760 (request call id 591) took 1012 ms. Trace:
I20260809 15:46:44.752296 23518 rpcz_store.cc:276] 0809 15:46:43.739073 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:43.743021 (+ 3948us) service_pool.cc:227] Handling call
0809 15:46:44.737081 (+994060us) spinlock_profiling.cc:243] Waited 736 ms on lock 0x7b5400108bd4. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5af0 00007f34e9228f2a 00007f34ee165c6e 00007f34e785c807 00007f34e785c792 00007f34e785c6c1 00007f34e785c63c 00007f34e785b8f2 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:46:44.751791 (+ 14710us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1692930304}
W20260809 15:46:44.754594 23193 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 297c82ed906d4b48b184985eed3de601, TS 67b3aeadfa7a432abe8af4d292e04ab9 -> TS 61e69695258f48baa984c9d395e8ce3f move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:46:44.754912 23193 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 297c82ed906d4b48b184985eed3de601, TS 67b3aeadfa7a432abe8af4d292e04ab9 -> TS 61e69695258f48baa984c9d395e8ce3f move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:46:44.745568 26670 log.cc:924] Time spent T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d: Append to log took a long time: real 0.152s user 0.000s sys 0.001s
I20260809 15:46:44.760664 26279 raft_consensus.cc:504] T 472816994d814cb198b69e5248697440 P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:44.761986 24452 raft_consensus.cc:2379] T 472816994d814cb198b69e5248697440 P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c in current term 5: Already voted for candidate 61e69695258f48baa984c9d395e8ce3f in this term.
I20260809 15:46:44.762718 26279 leader_election.cc:288] T 472816994d814cb198b69e5248697440 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 5 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:44.747236 26488 leader_election.cc:288] T 28e17dcb5efe45b8b6a8c42da03b4819 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:44.749265 23520 raft_consensus.cc:1226] T 0acc47ce21c3422698d1ef7277c3ed63 P 65c36d164b0540f099d592f38307ee6c [term 5 LEADER]: Rejecting Update request from peer c0f38c1908414566aa9115bb2cbdd85d for earlier term 4. Current term is 5. Ops: [4.2-4.2]
I20260809 15:46:44.750336 26485 leader_election.cc:288] T 00a1881267a14ba8b78c81040f17a014 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 6 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:44.692530 26598 leader_election.cc:288] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 4 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:44.724169 26523 raft_consensus.cc:2781] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Leader election won for term 4
I20260809 15:46:44.765686 26667 raft_consensus.cc:2726] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 15:46:44.738520 26611 raft_consensus.cc:482] T 4564dfc6d4094c1b9240b367ccecf3b1 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:44.734807 26465 leader_election.cc:288] T 73f6f44c259547f1b05a3a434616fa54 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:44.847914 23397 raft_consensus.cc:1226] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Rejecting Update request from peer 61e69695258f48baa984c9d395e8ce3f for earlier term 2. Current term is 3. Ops: [2.5-2.5]
I20260809 15:46:44.775827 25863 raft_consensus.cc:504] T 05ad7d8fdca64e3da527280ca5be9280 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:44.867292 23270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a51c0baaa6e432997c9d3552af4f254" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:44.837765 26523 raft_consensus.cc:686] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 61e69695258f48baa984c9d395e8ce3f [term 4 LEADER]: Becoming Leader. State: Replica: 61e69695258f48baa984c9d395e8ce3f, State: Running, Role: LEADER
I20260809 15:46:44.866639 25746 leader_election.cc:288] T 0a51c0baaa6e432997c9d3552af4f254 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:44.850452 26488 raft_consensus.cc:482] T 2424fd9bd7f14e508fd106cbfda786a4 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
W20260809 15:46:44.889500 23470 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.696s user 0.013s sys 0.049s
W20260809 15:46:44.889822 23470 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.696s user 0.013s sys 0.049s
W20260809 15:46:44.890077 23470 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 707091us
W20260809 15:46:44.890151 23396 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.663s user 0.018s sys 0.010s
W20260809 15:46:44.891552 23396 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.665s user 0.018s sys 0.010s
I20260809 15:46:44.894742 23396 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55320 (request call id 1249) took 1972 ms. Trace:
I20260809 15:46:44.898865 23286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cef3e7277d7d4760824a3436ed4a847d" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:44.848359 26611 raft_consensus.cc:504] T 4564dfc6d4094c1b9240b367ccecf3b1 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:44.899474 23286 raft_consensus.cc:3037] T cef3e7277d7d4760824a3436ed4a847d P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Advancing to term 6
I20260809 15:46:44.897260 23396 rpcz_store.cc:276] 0809 15:46:42.921930 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:43.040154 (+118224us) service_pool.cc:227] Handling call
0809 15:46:43.521275 (+481121us) spinlock_profiling.cc:243] Waited 2.34 ms on lock 0x7b58001d7c98. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e613cf11 00007f34e62c0851 00007f34e62c0702 00007f34ef1b0b09 00007f34ee0ec730 00007f34ee1b279b 00007f34ee1b2821 00007f34ee1762b0 00007f34ee1617f9 00007f34ee1612a2 00007f34e7847684
0809 15:46:43.521467 (+ 192us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:43.521649 (+ 182us) raft_consensus.cc:1520] Early marking committed up to index 2
0809 15:46:44.226460 (+704811us) log.cc:842] Serialized 14 byte log entry
0809 15:46:44.894435 (+667975us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 15:46:44.894487 (+ 52us) raft_consensus.cc:1631] Marking committed up to 2
0809 15:46:44.894527 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:44.894586 (+ 59us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:44.894690 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2484,"prepare.queue_time_us":978091,"prepare.run_cpu_time_us":24,"prepare.run_wall_time_us":25,"spinlock_wait_cycles":5378304,"thread_start_us":1356494,"threads_started":2}
I20260809 15:46:44.925982 23280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e17dcb5efe45b8b6a8c42da03b4819" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
W20260809 15:46:44.932704 23394 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.708s user 0.000s sys 0.030s
W20260809 15:46:44.935138 23394 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.711s user 0.000s sys 0.031s
I20260809 15:46:44.937227 23280 raft_consensus.cc:2379] T 28e17dcb5efe45b8b6a8c42da03b4819 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c in current term 5: Already voted for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in this term.
I20260809 15:46:44.941315 23394 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55320 (request call id 1260) took 1877 ms. Trace:
W20260809 15:46:44.939081 23529 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.590s user 0.008s sys 0.026s
W20260809 15:46:44.905652 26638 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.619s user 0.022s sys 0.052s
I20260809 15:46:44.890261 26488 raft_consensus.cc:504] T 2424fd9bd7f14e508fd106cbfda786a4 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:46:44.924969 26436 consensus_meta.cc:287] T 30c1740ef6874df0a332fbd6fd4b03ce P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent flushing consensus metadata: real 0.568s user 0.007s sys 0.006s
I20260809 15:46:44.979346 23286 raft_consensus.cc:2454] T cef3e7277d7d4760824a3436ed4a847d P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 6.
W20260809 15:46:44.941992 23393 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.586s user 0.012s sys 0.017s
W20260809 15:46:44.948383 23529 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.599s user 0.008s sys 0.026s
W20260809 15:46:44.959587 23469 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.670s user 0.015s sys 0.018s
W20260809 15:46:44.983186 23469 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.780s user 0.015s sys 0.018s
I20260809 15:46:44.983983 23228 leader_election.cc:302] T 7baa2492df0340c4a5ac241959ce1ac3 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:46:44.960039 23280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0da4d5283c8a46878bd39b85e5c71f66" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:44.984750 23280 raft_consensus.cc:3037] T 0da4d5283c8a46878bd39b85e5c71f66 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Advancing to term 5
W20260809 15:46:44.961341 26687 log.cc:924] Time spent T 0c48a3049897452383257aa757834fb5 P 61e69695258f48baa984c9d395e8ce3f: Append to log took a long time: real 0.092s user 0.000s sys 0.000s
W20260809 15:46:44.961894 26680 log.cc:924] Time spent T c0741ab44d99439abc3aa3bdd27ee160 P 61e69695258f48baa984c9d395e8ce3f: Append to log took a long time: real 0.132s user 0.000s sys 0.000s
W20260809 15:46:44.962456 26683 log.cc:924] Time spent T df20671a7b874dbb98c0ef28371c1308 P c0f38c1908414566aa9115bb2cbdd85d: Append to log took a long time: real 0.112s user 0.000s sys 0.000s
I20260809 15:46:44.968911 23273 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53554 (request call id 1357) took 1295 ms. Trace:
I20260809 15:46:44.992654 23273 rpcz_store.cc:276] 0809 15:46:43.673770 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:43.772470 (+ 98700us) service_pool.cc:227] Handling call
0809 15:46:43.772834 (+ 364us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 15:46:43.773135 (+ 301us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:46:43.773163 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 15:46:43.773594 (+ 431us) log.cc:842] Serialized 26 byte log entry
0809 15:46:44.518296 (+744702us) raft_consensus.cc:1631] Marking committed up to 2
0809 15:46:44.518332 (+ 36us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:44.518391 (+ 59us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 15:46:44.968744 (+450353us) raft_consensus.cc:1671] finished
0809 15:46:44.968763 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:44.968864 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":591,"raft.queue_time_us":3180,"raft.run_cpu_time_us":147,"raft.run_wall_time_us":148,"thread_start_us":744192,"threads_started":1,"wal-append.queue_time_us":1125237}
I20260809 15:46:44.995880 23273 raft_consensus.cc:1261] T 998c6cee72cf40228c6dba064c0db2da P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
W20260809 15:46:44.996773 26648 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.520s user 0.008s sys 0.016s
W20260809 15:46:44.997159 26648 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.520s user 0.008s sys 0.016s
I20260809 15:46:44.969714 26143 consensus_queue.cc:1035] T f9e7cadcd24d45fcbfa84209225e8f2d P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260809 15:46:44.971249 25885 raft_consensus.cc:482] T 538a228ddfb54bc6a36528d5ff07597b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
W20260809 15:46:44.945607 26677 log.cc:924] Time spent T 73f6f44c259547f1b05a3a434616fa54 P c0f38c1908414566aa9115bb2cbdd85d: Append to log took a long time: real 0.263s user 0.000s sys 0.001s
I20260809 15:46:44.985077 23228 leader_election.cc:302] T e14d118b2eaa42aa8a7cd4025aba7128 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 61e69695258f48baa984c9d395e8ce3f
W20260809 15:46:44.977102 23353 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.524s user 0.002s sys 0.025s
I20260809 15:46:44.945139 26605 raft_consensus.cc:482] T d1a134ba33ea4076bbaac0ae460c17ba P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:44.988014 23470 leader_election.cc:302] T cef3e7277d7d4760824a3436ed4a847d P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
W20260809 15:46:44.989120 23351 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.542s user 0.011s sys 0.020s
W20260809 15:46:45.008996 23351 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.562s user 0.011s sys 0.021s
I20260809 15:46:44.989820 21289 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20260809 15:46:44.981305 23276 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.605s user 0.014s sys 0.031s
W20260809 15:46:44.954814 26638 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.693s user 0.022s sys 0.052s
W20260809 15:46:44.982865 23393 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.635s user 0.012s sys 0.017s
W20260809 15:46:45.001129 23353 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.548s user 0.002s sys 0.025s
I20260809 15:46:45.000260 23527 raft_consensus.cc:1226] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Rejecting Update request from peer 61e69695258f48baa984c9d395e8ce3f for earlier term 2. Current term is 3. Ops: [2.5-2.5]
I20260809 15:46:45.002836 23521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7660dfa2ef694927b8eda4100a2bcd9b" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
W20260809 15:46:45.019167 23353 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 639728us
W20260809 15:46:45.003396 23517 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.528s user 0.007s sys 0.023s
W20260809 15:46:45.020116 23517 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.545s user 0.007s sys 0.023s
I20260809 15:46:45.007730 23286 raft_consensus.cc:1226] T 16ca412a0dae4feebe85dd01bf1a9173 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 3. Current term is 4. Ops: []
I20260809 15:46:44.981195 26431 raft_consensus.cc:482] T d0721a4403a440f383318a3cea9d1442 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 15:46:45.009339 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 628362us
I20260809 15:46:44.988826 26465 raft_consensus.cc:2781] T 7baa2492df0340c4a5ac241959ce1ac3 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader election won for term 3
I20260809 15:46:44.986459 26667 raft_consensus.cc:482] T 4bcac4b24e7a4448abdc934bba5a624e P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
I20260809 15:46:45.018139 23280 raft_consensus.cc:2454] T 0da4d5283c8a46878bd39b85e5c71f66 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 5.
W20260809 15:46:45.032680 23345 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.536s user 0.010s sys 0.024s
W20260809 15:46:45.038074 23345 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.541s user 0.010s sys 0.024s
I20260809 15:46:45.031890 23286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99bdb93503db4de4a1cfb0161ea0c07c" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
W20260809 15:46:45.038350 24410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 341231us
I20260809 15:46:45.039278 23286 raft_consensus.cc:2396] T 99bdb93503db4de4a1cfb0161ea0c07c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20260809 15:46:45.007997 26688 log.cc:924] Time spent T 297c82ed906d4b48b184985eed3de601 P 65c36d164b0540f099d592f38307ee6c: Append to log took a long time: real 0.229s user 0.001s sys 0.001s
W20260809 15:46:45.009377 26144 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.529s user 0.008s sys 0.017s
I20260809 15:46:45.040412 23286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "763e58d6ba644eea830a6c1664d39c9b" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
W20260809 15:46:44.988577 26569 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.619s user 0.011s sys 0.035s
W20260809 15:46:45.046468 26569 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.690s user 0.011s sys 0.035s
W20260809 15:46:45.003763 26386 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.524s user 0.003s sys 0.021s
W20260809 15:46:45.046995 26386 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.567s user 0.004s sys 0.022s
I20260809 15:46:45.012215 26485 raft_consensus.cc:2781] T e14d118b2eaa42aa8a7cd4025aba7128 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Leader election won for term 6
I20260809 15:46:45.053836 23397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472816994d814cb198b69e5248697440" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:45.054405 23397 raft_consensus.cc:3037] T 472816994d814cb198b69e5248697440 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Advancing to term 5
I20260809 15:46:45.033044 26469 raft_consensus.cc:482] T f9e7cadcd24d45fcbfa84209225e8f2d P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:45.044093 23280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "297c82ed906d4b48b184985eed3de601" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:45.059680 25746 consensus_queue.cc:1035] T d3be1e356b254d22b995be32f1c12f6f P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260809 15:46:45.062016 25746 consensus_queue.cc:1035] T 3924b648461e463b9d7925e5671d68ae P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.000s
W20260809 15:46:45.065052 26691 log.cc:924] Time spent T c0741ab44d99439abc3aa3bdd27ee160 P 67b3aeadfa7a432abe8af4d292e04ab9: Append to log took a long time: real 0.213s user 0.002s sys 0.000s
I20260809 15:46:45.065227 25746 consensus_queue.cc:1046] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260809 15:46:45.058560 23402 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55320 (request call id 1269) took 1569 ms. Trace:
I20260809 15:46:44.998169 26619 raft_consensus.cc:482] T 4bcac4b24e7a4448abdc934bba5a624e P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
I20260809 15:46:44.998330 26609 raft_consensus.cc:482] T d0721a4403a440f383318a3cea9d1442 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
W20260809 15:46:45.026054 23276 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.651s user 0.014s sys 0.031s
W20260809 15:46:45.043606 26144 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.563s user 0.009s sys 0.017s
W20260809 15:46:45.034245 26398 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.501s user 0.003s sys 0.016s
I20260809 15:46:45.048060 23282 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53554 (request call id 1364) took 1114 ms. Trace:
W20260809 15:46:45.013751 26608 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.536s user 0.004s sys 0.022s
I20260809 15:46:44.982435 26436 raft_consensus.cc:504] T 30c1740ef6874df0a332fbd6fd4b03ce P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:45.057441 24475 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38734 (request call id 262) took 1455 ms. Trace:
I20260809 15:46:45.089560 23397 raft_consensus.cc:2454] T 472816994d814cb198b69e5248697440 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 5.
I20260809 15:46:45.070492 24448 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38734 (request call id 265) took 1397 ms. Trace:
I20260809 15:46:44.948463 26523 consensus_queue.cc:237] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:45.067629 23402 rpcz_store.cc:276] 0809 15:46:43.489561 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:43.658334 (+168773us) service_pool.cc:227] Handling call
0809 15:46:43.675940 (+ 17606us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 15:46:43.695095 (+ 19155us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:46:43.695124 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 15:46:43.701983 (+ 6859us) log.cc:842] Serialized 26 byte log entry
0809 15:46:44.369436 (+667453us) raft_consensus.cc:1631] Marking committed up to 1
0809 15:46:44.369472 (+ 36us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:44.369531 (+ 59us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 15:46:45.058364 (+688833us) raft_consensus.cc:1671] finished
0809 15:46:45.058382 (+ 18us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:45.058509 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":2039,"raft.run_cpu_time_us":88,"raft.run_wall_time_us":88,"thread_start_us":618383,"threads_started":1,"wal-append.queue_time_us":981627}
W20260809 15:46:45.051354 23392 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.528s user 0.006s sys 0.013s
I20260809 15:46:45.045624 26465 raft_consensus.cc:686] T 7baa2492df0340c4a5ac241959ce1ac3 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 LEADER]: Becoming Leader. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Running, Role: LEADER
I20260809 15:46:45.011912 25885 raft_consensus.cc:504] T 538a228ddfb54bc6a36528d5ff07597b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:45.117525 25885 leader_election.cc:288] T 538a228ddfb54bc6a36528d5ff07597b P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:45.117681 23285 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53554 (request call id 1346) took 1548 ms. Trace:
I20260809 15:46:45.118046 23285 rpcz_store.cc:276] 0809 15:46:43.569182 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:43.596749 (+ 27567us) service_pool.cc:227] Handling call
0809 15:46:43.597058 (+ 309us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 15:46:43.597246 (+ 188us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:46:43.597269 (+ 23us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 15:46:43.597650 (+ 381us) log.cc:842] Serialized 26 byte log entry
0809 15:46:44.588566 (+990916us) raft_consensus.cc:1631] Marking committed up to 2
0809 15:46:44.588606 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:44.588669 (+ 63us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 15:46:45.117478 (+528809us) raft_consensus.cc:1671] finished
0809 15:46:45.117497 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:45.117629 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":990427,"threads_started":1,"wal-append.queue_time_us":1484028}
W20260809 15:46:45.090142 26608 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.614s user 0.004s sys 0.022s
I20260809 15:46:45.119367 23285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5c2e425a74b4e64ae70b5def750434c" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:45.072558 23400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a51c0baaa6e432997c9d3552af4f254" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:45.071686 25746 consensus_queue.cc:1046] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260809 15:46:45.025233 26605 raft_consensus.cc:504] T d1a134ba33ea4076bbaac0ae460c17ba P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:45.080370 26638 consensus_queue.cc:1046] T 7baa2492df0340c4a5ac241959ce1ac3 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 15:46:45.071419 26469 raft_consensus.cc:504] T f9e7cadcd24d45fcbfa84209225e8f2d P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:45.081226 23280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73f6f44c259547f1b05a3a434616fa54" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:45.036865 26431 raft_consensus.cc:504] T d0721a4403a440f383318a3cea9d1442 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:45.058562 26485 raft_consensus.cc:686] T e14d118b2eaa42aa8a7cd4025aba7128 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 LEADER]: Becoming Leader. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Running, Role: LEADER
I20260809 15:46:45.070344 24455 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38722 (request call id 410) took 1233 ms. Trace:
I20260809 15:46:45.070811 25863 leader_election.cc:288] T 05ad7d8fdca64e3da527280ca5be9280 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:45.044977 26667 raft_consensus.cc:504] T 4bcac4b24e7a4448abdc934bba5a624e P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:45.090301 24475 rpcz_store.cc:276] 0809 15:46:43.602268 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:43.602383 (+ 115us) service_pool.cc:227] Handling call
0809 15:46:43.602671 (+ 288us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 15:46:43.602994 (+ 323us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:46:43.603021 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 15:46:43.612704 (+ 9683us) log.cc:842] Serialized 26 byte log entry
0809 15:46:44.378065 (+765361us) raft_consensus.cc:1631] Marking committed up to 5
0809 15:46:44.378099 (+ 34us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:44.378218 (+ 119us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 15:46:45.057260 (+679042us) raft_consensus.cc:1671] finished
0809 15:46:45.057279 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:45.057388 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":93,"raft.run_cpu_time_us":217,"raft.run_wall_time_us":4700,"thread_start_us":765007,"threads_started":1,"wal-append.queue_time_us":1083108}
I20260809 15:46:45.044056 23286 raft_consensus.cc:2379] T 763e58d6ba644eea830a6c1664d39c9b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d in current term 4: Already voted for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in this term.
I20260809 15:46:45.237124 23286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e17dcb5efe45b8b6a8c42da03b4819" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:45.089118 23282 rpcz_store.cc:276] 0809 15:46:43.933990 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:43.934234 (+ 244us) service_pool.cc:227] Handling call
0809 15:46:43.934536 (+ 302us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 15:46:43.934851 (+ 315us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:46:43.934875 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0809 15:46:43.935344 (+ 469us) log.cc:842] Serialized 52 byte log entry
0809 15:46:44.670580 (+735236us) raft_consensus.cc:1631] Marking committed up to 4
0809 15:46:44.670617 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:44.670680 (+ 63us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 15:46:45.047844 (+377164us) raft_consensus.cc:1671] finished
0809 15:46:45.047861 (+ 17us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:45.048009 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1197,"raft.run_cpu_time_us":132,"raft.run_wall_time_us":11448,"thread_start_us":689301,"threads_started":1,"wal-append.queue_time_us":978965}
I20260809 15:46:45.075174 26619 raft_consensus.cc:504] T 4bcac4b24e7a4448abdc934bba5a624e P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:45.242599 23286 raft_consensus.cc:2379] T 28e17dcb5efe45b8b6a8c42da03b4819 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d in current term 5: Already voted for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in this term.
I20260809 15:46:45.243450 23286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "907bacfa79d04a55a68c8dbe60056462" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:45.077668 26611 leader_election.cc:288] T 4564dfc6d4094c1b9240b367ccecf3b1 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:45.094726 26488 leader_election.cc:288] T 2424fd9bd7f14e508fd106cbfda786a4 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:45.095459 26740 consensus_queue.cc:1046] T 0acc47ce21c3422698d1ef7277c3ed63 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260809 15:46:45.077831 24463 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38724 (request call id 460) took 2124 ms. Trace:
I20260809 15:46:45.072971 24410 leader_election.cc:302] T 7660dfa2ef694927b8eda4100a2bcd9b P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
W20260809 15:46:45.121650 26185 consensus_meta.cc:287] T 0a51c0baaa6e432997c9d3552af4f254 P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent flushing consensus metadata: real 0.787s user 0.008s sys 0.003s
W20260809 15:46:45.121927 26574 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.633s user 0.005s sys 0.020s
I20260809 15:46:45.135000 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e17dcb5efe45b8b6a8c42da03b4819" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
W20260809 15:46:45.257201 23353 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 234502us
I20260809 15:46:45.120177 23400 raft_consensus.cc:2454] T 0a51c0baaa6e432997c9d3552af4f254 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 3.
I20260809 15:46:45.141265 23283 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53554 (request call id 1360) took 1403 ms. Trace:
I20260809 15:46:45.258694 23402 raft_consensus.cc:3037] T 28e17dcb5efe45b8b6a8c42da03b4819 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Advancing to term 5
W20260809 15:46:45.100785 23392 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.586s user 0.007s sys 0.014s
I20260809 15:46:45.222098 23280 raft_consensus.cc:3037] T 73f6f44c259547f1b05a3a434616fa54 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Advancing to term 5
I20260809 15:46:45.092885 26728 consensus_queue.cc:1046] T cef3e7277d7d4760824a3436ed4a847d P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.048s
I20260809 15:46:45.078402 26609 raft_consensus.cc:504] T d0721a4403a440f383318a3cea9d1442 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:45.089265 26144 consensus_queue.cc:1046] T 0da4d5283c8a46878bd39b85e5c71f66 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.056s
I20260809 15:46:45.238809 23282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a51c0baaa6e432997c9d3552af4f254" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:45.240454 23228 leader_election.cc:302] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:45.244376 23286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a88421dba80d4d3388aa5eb62e23853b" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:45.266332 23282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c29937a80724ddb9d07a27534a146f3" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:45.267012 23282 raft_consensus.cc:2379] T 0c29937a80724ddb9d07a27534a146f3 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d in current term 5: Already voted for candidate 65c36d164b0540f099d592f38307ee6c in this term.
I20260809 15:46:45.267663 23282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998c6cee72cf40228c6dba064c0db2da" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
W20260809 15:46:45.240265 26699 log.cc:924] Time spent T d3be1e356b254d22b995be32f1c12f6f P 67b3aeadfa7a432abe8af4d292e04ab9: Append to log took a long time: real 0.288s user 0.000s sys 0.001s
I20260809 15:46:45.140050 23285 raft_consensus.cc:2379] T c5c2e425a74b4e64ae70b5def750434c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d in current term 4: Already voted for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in this term.
I20260809 15:46:45.270073 23228 leader_election.cc:302] T f7715d71dbac447b9351fc5eb36c5e80 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:45.270910 23285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e71a05ba1d4d42bd95c3f85f9313399e" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:45.273177 23521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e17dcb5efe45b8b6a8c42da03b4819" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
W20260809 15:46:45.258754 24410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 185851us
I20260809 15:46:45.258898 23283 rpcz_store.cc:276] 0809 15:46:43.737999 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:43.813203 (+ 75204us) service_pool.cc:227] Handling call
0809 15:46:43.813493 (+ 290us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 15:46:43.813815 (+ 322us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:46:43.813839 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 15:46:43.819538 (+ 5699us) log.cc:842] Serialized 26 byte log entry
0809 15:46:44.364860 (+545322us) raft_consensus.cc:1631] Marking committed up to 5
0809 15:46:44.364900 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:44.365063 (+ 163us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 15:46:45.064522 (+699459us) spinlock_profiling.cc:243] Waited 10.1 ms on lock 0x7b44000232cc. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e613cf11 00007f34e620aa95 00007f34e620b4b2 00007f34e8eb3511 00007f34e8eaee1a 00007f34e8ea2bee 00007f34e8ea1f36 00007f34e8ea1b58 00007f34e8eab551 00007f34e8eab509 00007f34e8eab499
0809 15:46:45.140985 (+ 76463us) raft_consensus.cc:1671] finished
0809 15:46:45.141006 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:45.141207 (+ 201us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":75,"raft.run_cpu_time_us":148,"raft.run_wall_time_us":7602,"spinlock_wait_cycles":23132544,"thread_start_us":496719,"threads_started":1,"wal-append.queue_time_us":892842}
I20260809 15:46:45.275081 23283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e71a05ba1d4d42bd95c3f85f9313399e" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:45.253899 24463 rpcz_store.cc:276] 0809 15:46:42.948734 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:42.948935 (+ 201us) service_pool.cc:227] Handling call
0809 15:46:42.949572 (+ 637us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 15:46:43.124128 (+174556us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:46:43.126997 (+ 2869us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0809 15:46:43.225607 (+ 98610us) log.cc:842] Serialized 52 byte log entry
0809 15:46:44.611544 (+1385937us) raft_consensus.cc:1631] Marking committed up to 4
0809 15:46:44.671234 (+ 59690us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:44.681957 (+ 10723us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 15:46:45.053882 (+371925us) raft_consensus.cc:1671] finished
0809 15:46:45.056475 (+ 2593us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:45.069651 (+ 13176us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":550,"raft.queue_time_us":6928,"raft.run_cpu_time_us":102,"raft.run_wall_time_us":6379,"thread_start_us":1354283,"threads_started":1,"wal-append.queue_time_us":1658171}
I20260809 15:46:45.276137 23282 raft_consensus.cc:2379] T 998c6cee72cf40228c6dba064c0db2da P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d in current term 7: Already voted for candidate 65c36d164b0540f099d592f38307ee6c in this term.
W20260809 15:46:45.258514 26574 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.773s user 0.005s sys 0.021s
I20260809 15:46:45.278993 23228 leader_election.cc:302] T d3be1e356b254d22b995be32f1c12f6f P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters: 61e69695258f48baa984c9d395e8ce3f
I20260809 15:46:45.287860 23228 leader_election.cc:302] T 0c29937a80724ddb9d07a27534a146f3 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:45.288264 24410 leader_election.cc:302] T 472816994d814cb198b69e5248697440 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:46:45.289827 23283 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0a51c0baaa6e432997c9d3552af4f254"
dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
mode: GRACEFUL
new_leader_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
from {username='slave'} at 127.0.0.1:33468
I20260809 15:46:45.299182 24410 leader_election.cc:302] T 0a51c0baaa6e432997c9d3552af4f254 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:46:45.277554 25885 raft_consensus.cc:482] T 953eea268e17491ca76f3a555860a512 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:45.295161 23517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00a1881267a14ba8b78c81040f17a014" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:45.281589 23521 raft_consensus.cc:2379] T 28e17dcb5efe45b8b6a8c42da03b4819 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 in current term 5: Already voted for candidate 65c36d164b0540f099d592f38307ee6c in this term.
I20260809 15:46:45.276865 23282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3be1e356b254d22b995be32f1c12f6f" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:45.282874 26469 leader_election.cc:288] T f9e7cadcd24d45fcbfa84209225e8f2d P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:44.948040 23394 rpcz_store.cc:276] 0809 15:46:43.064343 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:43.428867 (+364524us) service_pool.cc:227] Handling call
0809 15:46:43.429109 (+ 242us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:43.429266 (+ 157us) raft_consensus.cc:1520] Early marking committed up to index 2
0809 15:46:44.224279 (+795013us) log.cc:842] Serialized 14 byte log entry
0809 15:46:44.941032 (+716753us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 15:46:44.941081 (+ 49us) raft_consensus.cc:1631] Marking committed up to 2
0809 15:46:44.941119 (+ 38us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:44.941173 (+ 54us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:44.941267 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":1070144,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":35,"raft.queue_time_us":662,"raft.run_cpu_time_us":105,"raft.run_wall_time_us":3994,"thread_start_us":1510321,"threads_started":2}
I20260809 15:46:45.284871 23400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0da4d5283c8a46878bd39b85e5c71f66" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:45.397050 23400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4564dfc6d4094c1b9240b367ccecf3b1" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:45.397621 23400 raft_consensus.cc:2454] T 4564dfc6d4094c1b9240b367ccecf3b1 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 3.
I20260809 15:46:45.281054 23286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "763e58d6ba644eea830a6c1664d39c9b" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:45.263581 26420 raft_consensus.cc:2726] T 7660dfa2ef694927b8eda4100a2bcd9b P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260809 15:46:45.293450 26659 raft_consensus.cc:2726] T f7715d71dbac447b9351fc5eb36c5e80 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260809 15:46:45.277583 26488 raft_consensus.cc:482] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:45.277645 26487 raft_consensus.cc:2726] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260809 15:46:45.299660 26581 raft_consensus.cc:482] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Starting leader election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:46:45.430256 23400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472816994d814cb198b69e5248697440" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:45.431000 23400 raft_consensus.cc:2379] T 472816994d814cb198b69e5248697440 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c in current term 5: Already voted for candidate 61e69695258f48baa984c9d395e8ce3f in this term.
W20260809 15:46:45.397815 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 110016us
I20260809 15:46:45.403031 23286 raft_consensus.cc:2379] T 763e58d6ba644eea830a6c1664d39c9b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d in current term 4: Already voted for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in this term.
I20260809 15:46:45.403628 23394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05ad7d8fdca64e3da527280ca5be9280" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:45.393888 23280 raft_consensus.cc:2454] T 73f6f44c259547f1b05a3a434616fa54 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 5.
I20260809 15:46:45.417645 25746 raft_consensus.cc:482] T e71a05ba1d4d42bd95c3f85f9313399e P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
I20260809 15:46:45.418231 26420 raft_consensus.cc:3032] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 61e69695258f48baa984c9d395e8ce3f [term 2 LEADER]: Stepping down as leader of term 2
I20260809 15:46:45.295758 26490 raft_consensus.cc:2781] T d3be1e356b254d22b995be32f1c12f6f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260809 15:46:45.263403 26185 raft_consensus.cc:504] T 0a51c0baaa6e432997c9d3552af4f254 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:45.424820 23517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "538a228ddfb54bc6a36528d5ff07597b" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:45.434932 23394 raft_consensus.cc:2454] T 05ad7d8fdca64e3da527280ca5be9280 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 3.
I20260809 15:46:45.395254 26703 raft_consensus.cc:2781] T 472816994d814cb198b69e5248697440 P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Leader election won for term 5
I20260809 15:46:45.434233 23274 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55434 (request call id 180) took 1510 ms. Trace:
I20260809 15:46:45.443280 23286 raft_consensus.cc:1261] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20260809 15:46:45.444327 23286 pending_rounds.cc:76] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 67b3aeadfa7a432abe8af4d292e04ab9: Aborting all ops after (but not including) 1
I20260809 15:46:45.444558 23286 pending_rounds.cc:97] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 67b3aeadfa7a432abe8af4d292e04ab9: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20260809 15:46:45.397302 26721 raft_consensus.cc:482] T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
I20260809 15:46:45.401033 26179 raft_consensus.cc:2781] T 0a51c0baaa6e432997c9d3552af4f254 P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260809 15:46:45.443764 23274 rpcz_store.cc:276] 0809 15:46:43.924041 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:43.926534 (+ 2493us) service_pool.cc:227] Handling call
0809 15:46:43.931249 (+ 4715us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 15:46:43.931585 (+ 336us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:46:43.931616 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 15:46:43.932009 (+ 393us) log.cc:842] Serialized 26 byte log entry
0809 15:46:44.594962 (+662953us) raft_consensus.cc:1631] Marking committed up to 4
0809 15:46:44.595012 (+ 50us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:44.595075 (+ 63us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 15:46:45.434047 (+838972us) raft_consensus.cc:1671] finished
0809 15:46:45.434068 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:45.434177 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":93,"raft.run_cpu_time_us":97,"raft.run_wall_time_us":4018,"thread_start_us":661644,"threads_started":1,"wal-append.queue_time_us":990414}
I20260809 15:46:45.446803 23521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2424fd9bd7f14e508fd106cbfda786a4" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:45.395969 26574 raft_consensus.cc:2726] T 0c29937a80724ddb9d07a27534a146f3 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260809 15:46:45.444782 23286 raft_consensus.cc:2866] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 15:46:45.268519 26465 consensus_queue.cc:237] T 7baa2492df0340c4a5ac241959ce1ac3 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:45.419559 26469 raft_consensus.cc:482] T 6ae14a74a26f434db08c652e097385c8 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:45.422506 23393 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52856 (request call id 158) took 2344 ms. Trace:
I20260809 15:46:45.456681 23402 raft_consensus.cc:2396] T 28e17dcb5efe45b8b6a8c42da03b4819 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260809 15:46:45.457924 23517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73f6f44c259547f1b05a3a434616fa54" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:45.427318 26436 leader_election.cc:288] T 30c1740ef6874df0a332fbd6fd4b03ce P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:45.428231 26581 raft_consensus.cc:3037] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Advancing to term 6
I20260809 15:46:45.460726 23270 raft_consensus.cc:2379] T 0a51c0baaa6e432997c9d3552af4f254 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 61e69695258f48baa984c9d395e8ce3f in current term 4: Already voted for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in this term.
I20260809 15:46:45.461843 23286 raft_consensus.cc:1226] T cef3e7277d7d4760824a3436ed4a847d P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Rejecting Update request from peer c0f38c1908414566aa9115bb2cbdd85d for earlier term 5. Current term is 6. Ops: []
I20260809 15:46:45.484491 23283 raft_consensus.cc:595] T 0a51c0baaa6e432997c9d3552af4f254 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Received request to transfer leadership to TS c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:45.484814 23283 raft_consensus.cc:603] T 0a51c0baaa6e432997c9d3552af4f254 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260809 15:46:45.432395 26659 raft_consensus.cc:482] T 99bdb93503db4de4a1cfb0161ea0c07c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:45.485524 26659 raft_consensus.cc:504] T 99bdb93503db4de4a1cfb0161ea0c07c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:45.487550 26659 leader_election.cc:288] T 99bdb93503db4de4a1cfb0161ea0c07c P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:45.442368 26740 consensus_queue.cc:1225] T 0acc47ce21c3422698d1ef7277c3ed63 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Peer 65c36d164b0540f099d592f38307ee6c log is divergent from this leader: its last log entry 5.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260809 15:46:45.463626 23351 leader_election.cc:302] T 99bdb93503db4de4a1cfb0161ea0c07c P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:45.437755 26420 raft_consensus.cc:729] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 61e69695258f48baa984c9d395e8ce3f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 61e69695258f48baa984c9d395e8ce3f, State: Running, Role: LEADER
I20260809 15:46:45.489257 23521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5c2e425a74b4e64ae70b5def750434c" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:45.442629 26490 raft_consensus.cc:482] T d3be1e356b254d22b995be32f1c12f6f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
I20260809 15:46:45.420907 25885 raft_consensus.cc:504] T 953eea268e17491ca76f3a555860a512 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:45.491143 23402 raft_consensus.cc:3032] T ae6d2c08a7a94f58ac8dd698e3fd456c P c0f38c1908414566aa9115bb2cbdd85d [term 4 LEADER]: Stepping down as leader of term 4
I20260809 15:46:45.495314 23285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba56725e42f74a1baa4c04d9660727f9" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:45.453866 26179 raft_consensus.cc:482] T 0a51c0baaa6e432997c9d3552af4f254 P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:46:45.502367 23402 raft_consensus.cc:729] T ae6d2c08a7a94f58ac8dd698e3fd456c P c0f38c1908414566aa9115bb2cbdd85d [term 4 LEADER]: Becoming Follower/Learner. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:46:45.500806 26490 raft_consensus.cc:3037] T d3be1e356b254d22b995be32f1c12f6f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Advancing to term 5
W20260809 15:46:45.504809 26732 log.cc:924] Time spent T 4564dfc6d4094c1b9240b367ccecf3b1 P 67b3aeadfa7a432abe8af4d292e04ab9: Append to log took a long time: real 0.067s user 0.000s sys 0.000s
W20260809 15:46:45.508168 26739 log.cc:924] Time spent T 563daf42d7784db195ea99c3cb21b370 P 61e69695258f48baa984c9d395e8ce3f: Append to log took a long time: real 0.058s user 0.000s sys 0.000s
W20260809 15:46:45.508806 26714 log.cc:924] Time spent T 953eea268e17491ca76f3a555860a512 P 65c36d164b0540f099d592f38307ee6c: Append to log took a long time: real 0.101s user 0.000s sys 0.000s
W20260809 15:46:45.509577 26744 log.cc:924] Time spent T 6ced718c1a5c4da79cb8c1ebd1435b87 P 65c36d164b0540f099d592f38307ee6c: Append to log took a long time: real 0.208s user 0.000s sys 0.000s
I20260809 15:46:45.509591 26179 raft_consensus.cc:3037] T 0a51c0baaa6e432997c9d3552af4f254 P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Advancing to term 4
W20260809 15:46:45.510435 26750 log.cc:924] Time spent T c5c2e425a74b4e64ae70b5def750434c P 65c36d164b0540f099d592f38307ee6c: Append to log took a long time: real 0.109s user 0.000s sys 0.000s
I20260809 15:46:45.093915 24448 rpcz_store.cc:276] 0809 15:46:43.673246 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:43.674818 (+ 1572us) service_pool.cc:227] Handling call
0809 15:46:43.690146 (+ 15328us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 15:46:43.690504 (+ 358us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:46:43.690535 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 15:46:43.691098 (+ 563us) log.cc:842] Serialized 26 byte log entry
0809 15:46:44.471677 (+780579us) raft_consensus.cc:1631] Marking committed up to 4
0809 15:46:44.476087 (+ 4410us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:44.480822 (+ 4735us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 15:46:45.052082 (+571260us) raft_consensus.cc:1671] finished
0809 15:46:45.053918 (+ 1836us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:45.068228 (+ 14310us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":342,"raft.run_cpu_time_us":110,"raft.run_wall_time_us":111,"thread_start_us":755284,"threads_started":1,"wal-append.queue_time_us":1139061}
I20260809 15:46:45.507309 23285 raft_consensus.cc:1226] T 00a1881267a14ba8b78c81040f17a014 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Rejecting Update request from peer c0f38c1908414566aa9115bb2cbdd85d for earlier term 5. Current term is 6. Ops: []
I20260809 15:46:45.460731 26703 raft_consensus.cc:686] T 472816994d814cb198b69e5248697440 P 61e69695258f48baa984c9d395e8ce3f [term 5 LEADER]: Becoming Leader. State: Replica: 61e69695258f48baa984c9d395e8ce3f, State: Running, Role: LEADER
I20260809 15:46:45.513785 23285 raft_consensus.cc:1226] T ae6d2c08a7a94f58ac8dd698e3fd456c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Rejecting Update request from peer c0f38c1908414566aa9115bb2cbdd85d for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260809 15:46:45.400833 26485 consensus_queue.cc:237] T e14d118b2eaa42aa8a7cd4025aba7128 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:45.522375 23285 raft_consensus.cc:1261] T c5c2e425a74b4e64ae70b5def750434c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260809 15:46:45.522516 23517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472816994d814cb198b69e5248697440" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:45.437788 26488 raft_consensus.cc:504] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:45.530963 26488 leader_election.cc:288] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:45.533493 23285 raft_consensus.cc:1226] T 563daf42d7784db195ea99c3cb21b370 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 LEADER]: Rejecting Update request from peer c0f38c1908414566aa9115bb2cbdd85d for earlier term 3. Current term is 4. Ops: []
I20260809 15:46:45.534066 23517 raft_consensus.cc:2379] T 472816994d814cb198b69e5248697440 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 61e69695258f48baa984c9d395e8ce3f in current term 5: Already voted for candidate 65c36d164b0540f099d592f38307ee6c in this term.
I20260809 15:46:45.536012 23285 raft_consensus.cc:1226] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Rejecting Update request from peer c0f38c1908414566aa9115bb2cbdd85d for earlier term 3. Current term is 4. Ops: [3.5-3.5]
I20260809 15:46:45.537604 23285 raft_consensus.cc:1226] T 0da4d5283c8a46878bd39b85e5c71f66 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Rejecting Update request from peer c0f38c1908414566aa9115bb2cbdd85d for earlier term 4. Current term is 5. Ops: [4.2-4.2]
I20260809 15:46:45.505177 26574 consensus_queue.cc:1046] T e71a05ba1d4d42bd95c3f85f9313399e P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.034s
I20260809 15:46:45.519294 26719 raft_consensus.cc:482] T f7715d71dbac447b9351fc5eb36c5e80 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:45.539330 26719 raft_consensus.cc:504] T f7715d71dbac447b9351fc5eb36c5e80 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:45.502808 26436 raft_consensus.cc:482] T 2db057fd10e149a9895dff6ef8c9f038 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:45.541114 23274 raft_consensus.cc:1226] T 953eea268e17491ca76f3a555860a512 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Rejecting Update request from peer c0f38c1908414566aa9115bb2cbdd85d for earlier term 3. Current term is 4. Ops: []
W20260809 15:46:45.505023 26708 log.cc:924] Time spent T d1a134ba33ea4076bbaac0ae460c17ba P c0f38c1908414566aa9115bb2cbdd85d: Append to log took a long time: real 0.408s user 0.000s sys 0.001s
I20260809 15:46:45.442442 25746 raft_consensus.cc:504] T e71a05ba1d4d42bd95c3f85f9313399e P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:45.540292 26436 raft_consensus.cc:504] T 2db057fd10e149a9895dff6ef8c9f038 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:45.513052 23402 consensus_queue.cc:260] T ae6d2c08a7a94f58ac8dd698e3fd456c P c0f38c1908414566aa9115bb2cbdd85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:45.548712 23402 raft_consensus.cc:3037] T ae6d2c08a7a94f58ac8dd698e3fd456c P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Advancing to term 5
I20260809 15:46:45.505952 26605 leader_election.cc:288] T d1a134ba33ea4076bbaac0ae460c17ba P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:45.543807 23285 raft_consensus.cc:1226] T 0acc47ce21c3422698d1ef7277c3ed63 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Rejecting Update request from peer c0f38c1908414566aa9115bb2cbdd85d for earlier term 4. Current term is 5. Ops: [4.2-4.2]
I20260809 15:46:45.463763 26487 raft_consensus.cc:482] T 294d4bcc1e7a477cb56455581277a60f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:45.483129 26469 raft_consensus.cc:504] T 6ae14a74a26f434db08c652e097385c8 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:45.520831 26431 leader_election.cc:288] T d0721a4403a440f383318a3cea9d1442 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
W20260809 15:46:45.557976 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 94431.9us
I20260809 15:46:45.558763 23351 leader_election.cc:302] T 297c82ed906d4b48b184985eed3de601 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
W20260809 15:46:45.555848 26697 log.cc:924] Time spent T cee02fff2c9c48a19f43eb358a89e7b0 P 67b3aeadfa7a432abe8af4d292e04ab9: Append to log took a long time: real 0.603s user 0.002s sys 0.000s
I20260809 15:46:45.556020 26703 consensus_queue.cc:237] T 472816994d814cb198b69e5248697440 P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:45.459471 26721 raft_consensus.cc:504] T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:45.541913 26667 leader_election.cc:288] T 4bcac4b24e7a4448abdc934bba5a624e P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:45.229099 24455 rpcz_store.cc:276] 0809 15:46:43.837392 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:43.837591 (+ 199us) service_pool.cc:227] Handling call
0809 15:46:43.838072 (+ 481us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 15:46:43.838524 (+ 452us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:46:43.838593 (+ 69us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 15:46:43.839221 (+ 628us) log.cc:842] Serialized 26 byte log entry
0809 15:46:44.599681 (+760460us) raft_consensus.cc:1631] Marking committed up to 5
0809 15:46:44.604996 (+ 5315us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:44.612323 (+ 7327us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 15:46:45.048425 (+436102us) raft_consensus.cc:1671] finished
0809 15:46:45.051258 (+ 2833us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:45.068256 (+ 16998us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":6672,"raft.run_cpu_time_us":161,"raft.run_wall_time_us":8115,"thread_start_us":754098,"threads_started":1,"wal-append.queue_time_us":1057733}
I20260809 15:46:45.557217 26740 raft_consensus.cc:482] T d1a134ba33ea4076bbaac0ae460c17ba P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:45.556354 26487 raft_consensus.cc:504] T 294d4bcc1e7a477cb56455581277a60f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:45.549317 26778 raft_consensus.cc:482] T 05ad7d8fdca64e3da527280ca5be9280 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:45.583494 26608 raft_consensus.cc:482] T 4b30f28311964a0b9149a50495e83d8f P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:45.595227 26608 raft_consensus.cc:504] T 4b30f28311964a0b9149a50495e83d8f P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:45.597733 26608 leader_election.cc:288] T 4b30f28311964a0b9149a50495e83d8f P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
W20260809 15:46:45.584455 26718 log.cc:924] Time spent T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9: Append to log took a long time: real 0.122s user 0.000s sys 0.000s
W20260809 15:46:45.599188 26671 log.cc:924] Time spent T 2424fd9bd7f14e508fd106cbfda786a4 P c0f38c1908414566aa9115bb2cbdd85d: Append to log took a long time: real 0.919s user 0.001s sys 0.002s
I20260809 15:46:45.584867 24448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5712f6b6fffc4cc5a8ddfbe63e4097df" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
W20260809 15:46:45.585909 26694 log.cc:924] Time spent T df20671a7b874dbb98c0ef28371c1308 P 67b3aeadfa7a432abe8af4d292e04ab9: Append to log took a long time: real 0.618s user 0.008s sys 0.000s
I20260809 15:46:45.586412 26609 leader_election.cc:288] T d0721a4403a440f383318a3cea9d1442 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:45.550348 26619 leader_election.cc:288] T 4bcac4b24e7a4448abdc934bba5a624e P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:45.600036 23276 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55434 (request call id 181) took 1438 ms. Trace:
I20260809 15:46:45.603598 23277 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53554 (request call id 1354) took 1960 ms. Trace:
I20260809 15:46:45.592841 26740 raft_consensus.cc:504] T d1a134ba33ea4076bbaac0ae460c17ba P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:46:45.605343 24406 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 45410.6us
I20260809 15:46:45.576709 23285 raft_consensus.cc:1261] T 73f6f44c259547f1b05a3a434616fa54 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
W20260809 15:46:45.594813 26734 log.cc:924] Time spent T 213543cdbe814c89a17a963469922c1b P c0f38c1908414566aa9115bb2cbdd85d: Append to log took a long time: real 0.069s user 0.000s sys 0.000s
I20260809 15:46:45.594300 26778 raft_consensus.cc:504] T 05ad7d8fdca64e3da527280ca5be9280 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:45.601711 23404 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55320 (request call id 1263) took 2450 ms. Trace:
I20260809 15:46:45.612541 23404 rpcz_store.cc:276] 0809 15:46:43.150842 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:43.487109 (+336267us) service_pool.cc:227] Handling call
0809 15:46:43.487497 (+ 388us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 15:46:43.487871 (+ 374us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:46:43.487900 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 15:46:43.488258 (+ 358us) log.cc:842] Serialized 26 byte log entry
0809 15:46:44.243353 (+755095us) raft_consensus.cc:1631] Marking committed up to 1
0809 15:46:44.243393 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:44.243458 (+ 65us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 15:46:45.598625 (+1355167us) spinlock_profiling.cc:243] Waited 3.94 ms on lock 0x7b44000232cc. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e613cf11 00007f34e620aa95 00007f34e620b4b2 00007f34e8eb3511 00007f34e8eaee1a 00007f34e8ea2bee 00007f34e8ea1f36 00007f34e8ea1b58 00007f34e8eab551 00007f34e8eab509 00007f34e8eab499
0809 15:46:45.600166 (+ 1541us) raft_consensus.cc:1671] finished
0809 15:46:45.600186 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:45.601645 (+ 1459us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1120,"raft.run_cpu_time_us":156,"raft.run_wall_time_us":28849,"spinlock_wait_cycles":9052416,"thread_start_us":754598,"threads_started":1,"wal-append.queue_time_us":1106949}
I20260809 15:46:45.593775 26487 leader_election.cc:288] T 294d4bcc1e7a477cb56455581277a60f P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:45.578634 26279 raft_consensus.cc:482] T e14d118b2eaa42aa8a7cd4025aba7128 P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:45.579488 26464 leader_election.cc:288] T 3296ef50be254fc29cfc77d22a8f7a23 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
W20260809 15:46:45.622642 24410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 323517us
W20260809 15:46:45.620885 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 62205.3us
I20260809 15:46:45.600291 26638 raft_consensus.cc:3037] T 99bdb93503db4de4a1cfb0161ea0c07c P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:45.624370 23351 leader_election.cc:302] T 28e17dcb5efe45b8b6a8c42da03b4819 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:45.621044 23402 raft_consensus.cc:1261] T ae6d2c08a7a94f58ac8dd698e3fd456c P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
I20260809 15:46:45.603973 23277 rpcz_store.cc:276] 0809 15:46:43.643478 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:43.749357 (+105879us) service_pool.cc:227] Handling call
0809 15:46:43.749639 (+ 282us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 15:46:43.749956 (+ 317us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:46:43.749984 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 15:46:43.750401 (+ 417us) log.cc:842] Serialized 26 byte log entry
0809 15:46:44.735970 (+985569us) raft_consensus.cc:1631] Marking committed up to 1
0809 15:46:44.736037 (+ 67us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:44.736156 (+ 119us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 15:46:45.585363 (+849207us) spinlock_profiling.cc:243] Waited 18.5 ms on lock 0x7b44000232cc. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e613cf11 00007f34e620aa95 00007f34e620b4b2 00007f34e8eb3511 00007f34e8eaee1a 00007f34e8ea2bee 00007f34e8ea1f36 00007f34e8ea1b58 00007f34e8eab551 00007f34e8eab509 00007f34e8eab499
0809 15:46:45.603213 (+ 17850us) raft_consensus.cc:1671] finished
0809 15:46:45.603235 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:45.603532 (+ 297us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":974,"raft.run_cpu_time_us":92,"raft.run_wall_time_us":93,"spinlock_wait_cycles":42576768,"thread_start_us":985076,"threads_started":1,"wal-append.queue_time_us":1217005}
I20260809 15:46:45.625886 23351 leader_election.cc:302] T 907bacfa79d04a55a68c8dbe60056462 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 61e69695258f48baa984c9d395e8ce3f, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:45.627084 23351 leader_election.cc:302] T 0a51c0baaa6e432997c9d3552af4f254 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 61e69695258f48baa984c9d395e8ce3f, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:45.628175 23351 leader_election.cc:302] T 0c29937a80724ddb9d07a27534a146f3 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
W20260809 15:46:45.633258 23223 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.68:34689, user_credentials={real_user=slave}} blocked reactor thread for 76551.9us
I20260809 15:46:45.632879 23351 leader_election.cc:302] T 998c6cee72cf40228c6dba064c0db2da P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:45.625900 23402 pending_rounds.cc:76] T ae6d2c08a7a94f58ac8dd698e3fd456c P c0f38c1908414566aa9115bb2cbdd85d: Aborting all ops after (but not including) 2
I20260809 15:46:45.637526 23402 pending_rounds.cc:97] T ae6d2c08a7a94f58ac8dd698e3fd456c P c0f38c1908414566aa9115bb2cbdd85d: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20260809 15:46:45.637799 23402 raft_consensus.cc:2866] T ae6d2c08a7a94f58ac8dd698e3fd456c P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 15:46:45.622911 26487 raft_consensus.cc:482] T 0acc47ce21c3422698d1ef7277c3ed63 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:45.640580 26487 raft_consensus.cc:504] T 0acc47ce21c3422698d1ef7277c3ed63 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:45.641450 26740 leader_election.cc:288] T d1a134ba33ea4076bbaac0ae460c17ba P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:45.642609 26487 leader_election.cc:288] T 0acc47ce21c3422698d1ef7277c3ed63 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:45.642918 26777 raft_consensus.cc:2726] T 28e17dcb5efe45b8b6a8c42da03b4819 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260809 15:46:45.617671 26185 leader_election.cc:288] T 0a51c0baaa6e432997c9d3552af4f254 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
W20260809 15:46:45.643857 26763 log.cc:924] Time spent T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d: Append to log took a long time: real 0.084s user 0.001s sys 0.000s
I20260809 15:46:45.644830 26778 leader_election.cc:288] T 05ad7d8fdca64e3da527280ca5be9280 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:45.652801 23351 leader_election.cc:302] T a88421dba80d4d3388aa5eb62e23853b P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:45.654587 23351 leader_election.cc:302] T d3be1e356b254d22b995be32f1c12f6f P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 61e69695258f48baa984c9d395e8ce3f, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:45.656553 23351 leader_election.cc:302] T 763e58d6ba644eea830a6c1664d39c9b P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:45.631313 26719 leader_election.cc:288] T f7715d71dbac447b9351fc5eb36c5e80 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:45.637162 26743 raft_consensus.cc:2726] T 297c82ed906d4b48b184985eed3de601 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260809 15:46:45.682703 23283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e71a05ba1d4d42bd95c3f85f9313399e" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:45.653541 26740 raft_consensus.cc:2726] T 0c29937a80724ddb9d07a27534a146f3 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260809 15:46:45.679421 26488 raft_consensus.cc:482] T d1a134ba33ea4076bbaac0ae460c17ba P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:45.688366 23392 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55320 (request call id 1261) took 2548 ms. Trace:
I20260809 15:46:45.688521 24453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a51c0baaa6e432997c9d3552af4f254" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f"
I20260809 15:46:45.658123 26730 raft_consensus.cc:2726] T 0a51c0baaa6e432997c9d3552af4f254 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260809 15:46:45.632091 26279 raft_consensus.cc:504] T e14d118b2eaa42aa8a7cd4025aba7128 P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:45.686326 24448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3296ef50be254fc29cfc77d22a8f7a23" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:45.752861 26737 raft_consensus.cc:482] T 16ca412a0dae4feebe85dd01bf1a9173 P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:45.704291 26490 raft_consensus.cc:504] T d3be1e356b254d22b995be32f1c12f6f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:45.686830 26747 raft_consensus.cc:2726] T a88421dba80d4d3388aa5eb62e23853b P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260809 15:46:45.753363 26737 raft_consensus.cc:504] T 16ca412a0dae4feebe85dd01bf1a9173 P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:45.705932 23283 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "28e17dcb5efe45b8b6a8c42da03b4819"
dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
mode: GRACEFUL
new_leader_uuid: "65c36d164b0540f099d592f38307ee6c"
from {username='slave'} at 127.0.0.1:33468
I20260809 15:46:45.694741 26488 raft_consensus.cc:504] T d1a134ba33ea4076bbaac0ae460c17ba P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:45.686808 26719 raft_consensus.cc:2726] T d3be1e356b254d22b995be32f1c12f6f P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260809 15:46:45.707593 23395 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55320 (request call id 1279) took 2023 ms. Trace:
I20260809 15:46:45.752008 26469 leader_election.cc:288] T 6ae14a74a26f434db08c652e097385c8 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:45.692611 24452 raft_consensus.cc:1226] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 61e69695258f48baa984c9d395e8ce3f [term 4 LEADER]: Rejecting Update request from peer c0f38c1908414566aa9115bb2cbdd85d for earlier term 3. Current term is 4. Ops: []
I20260809 15:46:45.756870 26488 leader_election.cc:288] T d1a134ba33ea4076bbaac0ae460c17ba P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:45.463239 23393 rpcz_store.cc:276] 0809 15:46:43.059310 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:43.418328 (+359018us) service_pool.cc:227] Handling call
0809 15:46:43.451696 (+ 33368us) raft_consensus.cc:1457] Updating replica for 3 ops
0809 15:46:43.473364 (+ 21668us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:46:43.473397 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 3 ops
0809 15:46:43.480850 (+ 7453us) log.cc:842] Serialized 78 byte log entry
0809 15:46:44.287479 (+806629us) raft_consensus.cc:1631] Marking committed up to 6
0809 15:46:45.039793 (+752314us) log.cc:842] Serialized 14 byte log entry
0809 15:46:45.137488 (+ 97695us) log.cc:842] Serialized 14 byte log entry
0809 15:46:45.394448 (+256960us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:45.396702 (+ 2254us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 15:46:45.396817 (+ 115us) raft_consensus.cc:1671] finished
0809 15:46:45.396828 (+ 11us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:45.403981 (+ 7153us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":65763,"prepare.queue_time_us":197339,"prepare.run_cpu_time_us":55,"prepare.run_wall_time_us":52,"raft.queue_time_us":883870,"raft.run_cpu_time_us":305,"raft.run_wall_time_us":28480,"spinlock_wait_cycles":89561472,"thread_start_us":1774600,"threads_started":4,"wal-append.queue_time_us":1062385}
I20260809 15:46:45.717085 26648 consensus_queue.cc:1035] T 6ced718c1a5c4da79cb8c1ebd1435b87 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 15:46:45.681898 25885 leader_election.cc:288] T 953eea268e17491ca76f3a555860a512 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:45.683732 26728 raft_consensus.cc:2726] T 998c6cee72cf40228c6dba064c0db2da P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260809 15:46:45.719213 24455 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "294d4bcc1e7a477cb56455581277a60f" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:45.676625 26185 raft_consensus.cc:482] T 763e58d6ba644eea830a6c1664d39c9b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:45.759603 23274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e14d118b2eaa42aa8a7cd4025aba7128" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:45.760191 26185 raft_consensus.cc:504] T 763e58d6ba644eea830a6c1664d39c9b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:45.761801 26490 leader_election.cc:288] T d3be1e356b254d22b995be32f1c12f6f P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:45.762123 26185 leader_election.cc:288] T 763e58d6ba644eea830a6c1664d39c9b P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:45.751457 25746 leader_election.cc:288] T e71a05ba1d4d42bd95c3f85f9313399e P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:45.764083 24455 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3be1e356b254d22b995be32f1c12f6f" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f"
I20260809 15:46:45.751998 26279 leader_election.cc:288] T e14d118b2eaa42aa8a7cd4025aba7128 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:45.755012 23283 raft_consensus.cc:595] T 28e17dcb5efe45b8b6a8c42da03b4819 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Received request to transfer leadership to TS 65c36d164b0540f099d592f38307ee6c
I20260809 15:46:45.765461 23283 raft_consensus.cc:603] T 28e17dcb5efe45b8b6a8c42da03b4819 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260809 15:46:45.759227 23270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16ca412a0dae4feebe85dd01bf1a9173" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:45.693913 26436 leader_election.cc:288] T 2db057fd10e149a9895dff6ef8c9f038 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:45.605051 23276 rpcz_store.cc:276] 0809 15:46:44.161808 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:44.164338 (+ 2530us) service_pool.cc:227] Handling call
0809 15:46:44.206309 (+ 41971us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 15:46:44.215653 (+ 9344us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:46:44.218389 (+ 2736us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 15:46:44.288020 (+ 69631us) log.cc:842] Serialized 26 byte log entry
0809 15:46:45.084566 (+796546us) raft_consensus.cc:1631] Marking committed up to 5
0809 15:46:45.087911 (+ 3345us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:45.093207 (+ 5296us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 15:46:45.599798 (+506591us) raft_consensus.cc:1671] finished
0809 15:46:45.599826 (+ 28us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:45.599966 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":702545,"threads_started":1,"wal-append.queue_time_us":1092412}
I20260809 15:46:45.694164 26743 raft_consensus.cc:2726] T 763e58d6ba644eea830a6c1664d39c9b P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260809 15:46:45.766600 23270 raft_consensus.cc:2379] T 16ca412a0dae4feebe85dd01bf1a9173 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 61e69695258f48baa984c9d395e8ce3f in current term 4: Already voted for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in this term.
I20260809 15:46:45.773285 23399 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4b30f28311964a0b9149a50495e83d8f"
dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
mode: GRACEFUL
new_leader_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
from {username='slave'} at 127.0.0.1:51280
I20260809 15:46:45.773864 23399 raft_consensus.cc:595] T 4b30f28311964a0b9149a50495e83d8f P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Received request to transfer leadership to TS 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:45.774155 23399 raft_consensus.cc:603] T 4b30f28311964a0b9149a50495e83d8f P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260809 15:46:45.755657 23395 rpcz_store.cc:276] 0809 15:46:43.684197 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:44.597393 (+913196us) service_pool.cc:227] Handling call
0809 15:46:44.597684 (+ 291us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 15:46:44.597869 (+ 185us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:46:44.597893 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 15:46:44.603203 (+ 5310us) log.cc:842] Serialized 26 byte log entry
0809 15:46:45.064912 (+461709us) raft_consensus.cc:1631] Marking committed up to 2
0809 15:46:45.065509 (+ 597us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:45.066563 (+ 1054us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 15:46:45.707374 (+640811us) raft_consensus.cc:1671] finished
0809 15:46:45.707395 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:45.707534 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":393548,"threads_started":1,"wal-append.queue_time_us":1053153}
I20260809 15:46:45.785831 26570 raft_consensus.cc:482] T c5c2e425a74b4e64ae70b5def750434c P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:46:45.789779 23345 leader_election.cc:302] T e71a05ba1d4d42bd95c3f85f9313399e P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 61e69695258f48baa984c9d395e8ce3f, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:45.792591 23529 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56766 (request call id 1805) took 1446 ms. Trace:
I20260809 15:46:45.793378 23395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e4ffcf83c234ae7910ddef3ec0b4bda" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:45.794198 23397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00a1881267a14ba8b78c81040f17a014" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:45.795791 23395 raft_consensus.cc:3037] T 4564dfc6d4094c1b9240b367ccecf3b1 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:46:45.796588 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2424fd9bd7f14e508fd106cbfda786a4" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
W20260809 15:46:45.796815 24410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 146075us
I20260809 15:46:45.799984 23397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "538a228ddfb54bc6a36528d5ff07597b" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:45.802693 23404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0721a4403a440f383318a3cea9d1442" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:45.806731 23395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3296ef50be254fc29cfc77d22a8f7a23" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:45.792964 23529 rpcz_store.cc:276] 0809 15:46:44.345892 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:44.346035 (+ 143us) service_pool.cc:227] Handling call
0809 15:46:44.348658 (+ 2623us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 15:46:44.348870 (+ 212us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:46:44.348894 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 15:46:44.349299 (+ 405us) log.cc:842] Serialized 26 byte log entry
0809 15:46:44.985620 (+636321us) raft_consensus.cc:1631] Marking committed up to 7
0809 15:46:44.985864 (+ 244us) log.cc:842] Serialized 14 byte log entry
0809 15:46:45.785308 (+799444us) spinlock_profiling.cc:243] Waited 9.58 ms on lock 0x7b180034e040. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34ef1978b1 00007f34ef1a4661 00007f34e626f4ca 00007f34ef950f99 00007f34e6133990 00007f34e62f5864 00007f34e62f5155 00007f34e92183d0 00007f34e922a576 00007f34e923b0d7 00007f34e923b058
0809 15:46:45.785464 (+ 156us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:45.785542 (+ 78us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 15:46:45.785581 (+ 39us) raft_consensus.cc:1671] finished
0809 15:46:45.792412 (+ 6831us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:45.792533 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":15929,"prepare.queue_time_us":726,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":36,"raft.queue_time_us":800632,"spinlock_wait_cycles":22080640,"thread_start_us":636030,"threads_started":1,"wal-append.queue_time_us":744816}
I20260809 15:46:45.786352 26570 raft_consensus.cc:504] T c5c2e425a74b4e64ae70b5def750434c P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:46:46.004758 26797 log.cc:924] Time spent T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 61e69695258f48baa984c9d395e8ce3f: Append to log took a long time: real 0.081s user 0.000s sys 0.000s
I20260809 15:46:45.821521 23519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b30f28311964a0b9149a50495e83d8f" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
W20260809 15:46:46.006178 26693 log.cc:924] Time spent T 998c6cee72cf40228c6dba064c0db2da P 65c36d164b0540f099d592f38307ee6c: Append to log took a long time: real 1.039s user 0.007s sys 0.011s
I20260809 15:46:46.005940 23404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "953eea268e17491ca76f3a555860a512" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:46.073608 23404 raft_consensus.cc:2454] T 953eea268e17491ca76f3a555860a512 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 4.
I20260809 15:46:46.074446 23404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0acc47ce21c3422698d1ef7277c3ed63" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:45.865592 23393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73f6f44c259547f1b05a3a434616fa54" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:46.008692 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30c1740ef6874df0a332fbd6fd4b03ce" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:45.839195 26477 raft_consensus.cc:482] T 16ca412a0dae4feebe85dd01bf1a9173 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:45.838882 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bcac4b24e7a4448abdc934bba5a624e" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:45.838083 26559 raft_consensus.cc:2781] T cef3e7277d7d4760824a3436ed4a847d P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Leader election won for term 6
I20260809 15:46:46.095178 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99bdb93503db4de4a1cfb0161ea0c07c" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:46.035133 26386 raft_consensus.cc:482] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:46.090284 23472 leader_election.cc:302] T 3924b648461e463b9d7925e5671d68ae P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters: 61e69695258f48baa984c9d395e8ce3f
I20260809 15:46:45.906618 23273 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "538a228ddfb54bc6a36528d5ff07597b"
dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
mode: GRACEFUL
new_leader_uuid: "65c36d164b0540f099d592f38307ee6c"
from {username='slave'} at 127.0.0.1:33468
I20260809 15:46:45.796509 26726 raft_consensus.cc:2726] T e71a05ba1d4d42bd95c3f85f9313399e P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260809 15:46:46.051913 23395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2db057fd10e149a9895dff6ef8c9f038" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:46.052713 23521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16ca412a0dae4feebe85dd01bf1a9173" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:46.010939 23399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5c2e425a74b4e64ae70b5def750434c" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
W20260809 15:46:45.965031 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 307452us
I20260809 15:46:45.794765 26737 leader_election.cc:288] T 16ca412a0dae4feebe85dd01bf1a9173 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:46.009778 23397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9e7cadcd24d45fcbfa84209225e8f2d" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:45.791831 26743 raft_consensus.cc:482] T 213543cdbe814c89a17a963469922c1b P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:46.014977 23529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7715d71dbac447b9351fc5eb36c5e80" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:46.015002 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1a134ba33ea4076bbaac0ae460c17ba" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:46.054380 23522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e14d118b2eaa42aa8a7cd4025aba7128" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:46.134006 23522 raft_consensus.cc:2379] T e14d118b2eaa42aa8a7cd4025aba7128 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 61e69695258f48baa984c9d395e8ce3f in current term 6: Already voted for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in this term.
I20260809 15:46:46.060763 26420 consensus_queue.cc:260] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 61e69695258f48baa984c9d395e8ce3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:46.043076 26436 raft_consensus.cc:482] T 16ca412a0dae4feebe85dd01bf1a9173 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:45.904699 26399 raft_consensus.cc:482] T f7715d71dbac447b9351fc5eb36c5e80 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 15:46:46.070077 23470 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 1.08207e+06us
I20260809 15:46:45.887913 26707 raft_consensus.cc:482] T 763e58d6ba644eea830a6c1664d39c9b P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:45.934369 26777 raft_consensus.cc:2726] T 907bacfa79d04a55a68c8dbe60056462 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260809 15:46:46.091709 23517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9e7cadcd24d45fcbfa84209225e8f2d" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:45.930851 24455 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bcac4b24e7a4448abdc934bba5a624e" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:46.092880 23404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ae14a74a26f434db08c652e097385c8" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:46.093588 23519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30c1740ef6874df0a332fbd6fd4b03ce" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:46.135541 23397 raft_consensus.cc:2454] T f9e7cadcd24d45fcbfa84209225e8f2d P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 4.
I20260809 15:46:46.133580 23527 raft_consensus.cc:2454] T d1a134ba33ea4076bbaac0ae460c17ba P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 3.
I20260809 15:46:46.094791 26477 raft_consensus.cc:504] T 16ca412a0dae4feebe85dd01bf1a9173 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:45.964304 26179 raft_consensus.cc:504] T 0a51c0baaa6e432997c9d3552af4f254 P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
W20260809 15:46:46.267155 23353 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 204072us
I20260809 15:46:46.268857 23517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "294d4bcc1e7a477cb56455581277a60f" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:46.270058 23517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0721a4403a440f383318a3cea9d1442" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:46.271188 23397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1a134ba33ea4076bbaac0ae460c17ba" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:46.175132 23522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99bdb93503db4de4a1cfb0161ea0c07c" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:46.137307 23395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a51c0baaa6e432997c9d3552af4f254" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:46.272631 23522 raft_consensus.cc:2454] T 99bdb93503db4de4a1cfb0161ea0c07c P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 3.
I20260809 15:46:46.272866 23395 raft_consensus.cc:3037] T 0a51c0baaa6e432997c9d3552af4f254 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:46:46.205473 23399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3be1e356b254d22b995be32f1c12f6f" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:46.274780 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5712f6b6fffc4cc5a8ddfbe63e4097df" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:46.274878 23404 raft_consensus.cc:2454] T 6ae14a74a26f434db08c652e097385c8 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 4.
I20260809 15:46:46.271847 23397 raft_consensus.cc:2454] T d1a134ba33ea4076bbaac0ae460c17ba P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 2.
I20260809 15:46:46.309567 23395 raft_consensus.cc:2454] T 0a51c0baaa6e432997c9d3552af4f254 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 4.
W20260809 15:46:46.065191 26581 consensus_meta.cc:287] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 61e69695258f48baa984c9d395e8ce3f: Time spent flushing consensus metadata: real 0.561s user 0.009s sys 0.000s
I20260809 15:46:46.309706 23397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0721a4403a440f383318a3cea9d1442" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:46.318408 23397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05ad7d8fdca64e3da527280ca5be9280" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:46.318979 23397 raft_consensus.cc:2454] T 05ad7d8fdca64e3da527280ca5be9280 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 3.
I20260809 15:46:46.320250 23397 pending_rounds.cc:76] T 563daf42d7784db195ea99c3cb21b370 P c0f38c1908414566aa9115bb2cbdd85d: Aborting all ops after (but not including) 5
I20260809 15:46:46.320520 23397 pending_rounds.cc:97] T 563daf42d7784db195ea99c3cb21b370 P c0f38c1908414566aa9115bb2cbdd85d: Aborting uncommitted NO_OP operation due to leader change: 3.6
I20260809 15:46:46.320778 23397 raft_consensus.cc:2866] T 563daf42d7784db195ea99c3cb21b370 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 15:46:46.322387 26581 raft_consensus.cc:504] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 61e69695258f48baa984c9d395e8ce3f [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:46.325424 26581 leader_election.cc:288] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 6 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:46.328877 23395 raft_consensus.cc:1226] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Rejecting Update request from peer 61e69695258f48baa984c9d395e8ce3f for earlier term 2. Current term is 3. Ops: [2.5-2.5]
I20260809 15:46:46.310014 23404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1a134ba33ea4076bbaac0ae460c17ba" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
W20260809 15:46:46.276507 23472 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 186297us
I20260809 15:46:46.316818 24452 raft_consensus.cc:3037] T e14d118b2eaa42aa8a7cd4025aba7128 P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Advancing to term 6
I20260809 15:46:46.331728 23404 raft_consensus.cc:2454] T d1a134ba33ea4076bbaac0ae460c17ba P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 2.
I20260809 15:46:46.337037 23273 raft_consensus.cc:595] T 538a228ddfb54bc6a36528d5ff07597b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Received request to transfer leadership to TS 65c36d164b0540f099d592f38307ee6c
I20260809 15:46:46.337440 23273 raft_consensus.cc:603] T 538a228ddfb54bc6a36528d5ff07597b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260809 15:46:46.338675 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0acc47ce21c3422698d1ef7277c3ed63" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:46.345422 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05ad7d8fdca64e3da527280ca5be9280" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:46.349000 23526 raft_consensus.cc:2454] T 05ad7d8fdca64e3da527280ca5be9280 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 4.
I20260809 15:46:46.306553 26228 raft_consensus.cc:2781] T 3924b648461e463b9d7925e5671d68ae P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260809 15:46:46.351331 26228 raft_consensus.cc:482] T 3924b648461e463b9d7925e5671d68ae P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Starting leader election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
W20260809 15:46:46.352039 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 216711us
I20260809 15:46:46.352774 23351 leader_election.cc:302] T ba56725e42f74a1baa4c04d9660727f9 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:46.354259 23399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bcac4b24e7a4448abdc934bba5a624e" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:46.354420 23394 raft_consensus.cc:3032] T 0acc47ce21c3422698d1ef7277c3ed63 P c0f38c1908414566aa9115bb2cbdd85d [term 4 LEADER]: Stepping down as leader of term 4
I20260809 15:46:46.076887 26605 raft_consensus.cc:482] T 7660dfa2ef694927b8eda4100a2bcd9b P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:46:46.136875 26638 raft_consensus.cc:2726] T 99bdb93503db4de4a1cfb0161ea0c07c P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 15:46:46.215509 26420 raft_consensus.cc:3037] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 61e69695258f48baa984c9d395e8ce3f [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:46.388384 23521 raft_consensus.cc:2454] T 16ca412a0dae4feebe85dd01bf1a9173 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 4.
I20260809 15:46:46.389559 26815 consensus_queue.cc:1046] T 00a1881267a14ba8b78c81040f17a014 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260809 15:46:46.282204 26707 raft_consensus.cc:504] T 763e58d6ba644eea830a6c1664d39c9b P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:46.387629 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "953eea268e17491ca76f3a555860a512" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:46.351701 26228 raft_consensus.cc:3037] T 3924b648461e463b9d7925e5671d68ae P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Advancing to term 7
I20260809 15:46:46.204149 26436 raft_consensus.cc:504] T 16ca412a0dae4feebe85dd01bf1a9173 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:46.385776 26527 consensus_queue.cc:1046] T cef3e7277d7d4760824a3436ed4a847d P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.006s
I20260809 15:46:46.387184 26668 raft_consensus.cc:2726] T ba56725e42f74a1baa4c04d9660727f9 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260809 15:46:46.391613 26136 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 5 current term: 5 is higher.
I20260809 15:46:46.282503 26743 raft_consensus.cc:504] T 213543cdbe814c89a17a963469922c1b P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:46.393373 23517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2db057fd10e149a9895dff6ef8c9f038" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:46.214876 26559 raft_consensus.cc:686] T cef3e7277d7d4760824a3436ed4a847d P 65c36d164b0540f099d592f38307ee6c [term 6 LEADER]: Becoming Leader. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:46:45.737445 23392 rpcz_store.cc:276] 0809 15:46:43.139740 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:43.446624 (+306884us) service_pool.cc:227] Handling call
0809 15:46:43.447058 (+ 434us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:43.460656 (+ 13598us) raft_consensus.cc:1520] Early marking committed up to index 2
0809 15:46:44.484828 (+1024172us) log.cc:842] Serialized 14 byte log entry
0809 15:46:45.658476 (+1173648us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 15:46:45.676976 (+ 18500us) raft_consensus.cc:1631] Marking committed up to 2
0809 15:46:45.682437 (+ 5461us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:45.688173 (+ 5736us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:45.688299 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":231531,"prepare.queue_time_us":1028275,"prepare.run_cpu_time_us":25,"prepare.run_wall_time_us":25,"raft.queue_time_us":306089,"spinlock_wait_cycles":53592960,"thread_start_us":1899134,"threads_started":3,"wal-append.queue_time_us":974147}
I20260809 15:46:46.394238 26730 consensus_queue.cc:1035] T c5c2e425a74b4e64ae70b5def750434c P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20260809 15:46:46.334868 26399 raft_consensus.cc:504] T f7715d71dbac447b9351fc5eb36c5e80 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:46:46.396605 26804 log.cc:924] Time spent T e14d118b2eaa42aa8a7cd4025aba7128 P 67b3aeadfa7a432abe8af4d292e04ab9: Append to log took a long time: real 0.190s user 0.000s sys 0.000s
I20260809 15:46:46.397122 26490 raft_consensus.cc:482] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:46.397331 23518 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44018 (request call id 174) took 1397 ms. Trace:
W20260809 15:46:46.397578 26813 log.cc:924] Time spent T 907bacfa79d04a55a68c8dbe60056462 P c0f38c1908414566aa9115bb2cbdd85d: Append to log took a long time: real 0.051s user 0.000s sys 0.000s
I20260809 15:46:46.355688 23393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "763e58d6ba644eea830a6c1664d39c9b" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:46.403671 23519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ae14a74a26f434db08c652e097385c8" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:46.137106 26386 raft_consensus.cc:504] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:46.404878 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1a134ba33ea4076bbaac0ae460c17ba" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:46.408192 24410 leader_election.cc:302] T e14d118b2eaa42aa8a7cd4025aba7128 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:46.402808 26581 raft_consensus.cc:482] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
W20260809 15:46:46.428984 23472 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 92744.8us
W20260809 15:46:46.454119 23470 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 107912us
I20260809 15:46:46.454603 23394 raft_consensus.cc:729] T 0acc47ce21c3422698d1ef7277c3ed63 P c0f38c1908414566aa9115bb2cbdd85d [term 4 LEADER]: Becoming Follower/Learner. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:46:46.471807 26721 leader_election.cc:288] T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:46.472487 23402 raft_consensus.cc:2454] T 99bdb93503db4de4a1cfb0161ea0c07c P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 3.
I20260809 15:46:46.474452 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "763e58d6ba644eea830a6c1664d39c9b" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:46.482920 24475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3e94b7e50754a81b3d72a07476cb7b9" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
W20260809 15:46:46.473508 26802 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.699s user 0.019s sys 0.006s
I20260809 15:46:46.492157 23528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a823d35f4f348d1b52ba9e6d80b00d2" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:46.513700 23528 raft_consensus.cc:3037] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Advancing to term 6
I20260809 15:46:46.490001 24453 raft_consensus.cc:2379] T 0a51c0baaa6e432997c9d3552af4f254 P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 in current term 4: Already voted for candidate 61e69695258f48baa984c9d395e8ce3f in this term.
I20260809 15:46:46.500345 23529 raft_consensus.cc:2454] T f7715d71dbac447b9351fc5eb36c5e80 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 5.
I20260809 15:46:46.512285 23394 consensus_queue.cc:260] T 0acc47ce21c3422698d1ef7277c3ed63 P c0f38c1908414566aa9115bb2cbdd85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:46.519109 23394 raft_consensus.cc:3037] T 0acc47ce21c3422698d1ef7277c3ed63 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Advancing to term 5
I20260809 15:46:46.495280 23527 raft_consensus.cc:2454] T d1a134ba33ea4076bbaac0ae460c17ba P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 3.
I20260809 15:46:46.511535 23526 raft_consensus.cc:2454] T 763e58d6ba644eea830a6c1664d39c9b P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 4.
W20260809 15:46:46.513214 26802 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.740s user 0.019s sys 0.006s
I20260809 15:46:46.521896 24475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16ca412a0dae4feebe85dd01bf1a9173" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:46.470773 26605 raft_consensus.cc:504] T 7660dfa2ef694927b8eda4100a2bcd9b P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:46.486300 26836 raft_consensus.cc:482] T 294d4bcc1e7a477cb56455581277a60f P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:46.493927 26619 raft_consensus.cc:482] T e14d118b2eaa42aa8a7cd4025aba7128 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 15:46:46.580178 23470 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 64554.6us
I20260809 15:46:46.580369 24475 raft_consensus.cc:2454] T 16ca412a0dae4feebe85dd01bf1a9173 P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 3.
I20260809 15:46:46.509063 26570 leader_election.cc:288] T c5c2e425a74b4e64ae70b5def750434c P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:46.509552 23393 raft_consensus.cc:2454] T 763e58d6ba644eea830a6c1664d39c9b P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 4.
I20260809 15:46:46.637849 23528 raft_consensus.cc:2454] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 6.
I20260809 15:46:46.544332 26477 leader_election.cc:288] T 16ca412a0dae4feebe85dd01bf1a9173 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:46.497383 26490 raft_consensus.cc:504] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:46.514770 26581 raft_consensus.cc:504] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:46.491595 26527 consensus_queue.cc:1046] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.089s
I20260809 15:46:46.601938 26849 raft_consensus.cc:482] T c0741ab44d99439abc3aa3bdd27ee160 P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:46.655591 23396 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52856 (request call id 179) took 1024 ms. Trace:
I20260809 15:46:46.598858 25816 consensus_queue.cc:1035] T 998c6cee72cf40228c6dba064c0db2da P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.047s
I20260809 15:46:46.632864 26570 raft_consensus.cc:482] T 00a1881267a14ba8b78c81040f17a014 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:46:46.662293 23396 rpcz_store.cc:276] 0809 15:46:45.631054 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:45.631177 (+ 123us) service_pool.cc:227] Handling call
0809 15:46:45.638969 (+ 7792us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 15:46:45.639156 (+ 187us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:46:45.639178 (+ 22us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 15:46:45.639589 (+ 411us) log.cc:842] Serialized 26 byte log entry
0809 15:46:45.933540 (+293951us) raft_consensus.cc:1631] Marking committed up to 5
0809 15:46:45.933578 (+ 38us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:45.933658 (+ 80us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 15:46:46.655386 (+721728us) raft_consensus.cc:1671] finished
0809 15:46:46.655405 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:46.655530 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1408,"thread_start_us":293618,"threads_started":1,"wal-append.queue_time_us":537449}
I20260809 15:46:46.604547 23468 leader_election.cc:302] T 16ca412a0dae4feebe85dd01bf1a9173 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c; no voters:
W20260809 15:46:46.692284 23468 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.68:34689, user_credentials={real_user=slave}} blocked reactor thread for 87877.7us
W20260809 15:46:46.578066 23353 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 241765us
W20260809 15:46:46.662757 26523 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.831s user 0.013s sys 0.022s
W20260809 15:46:46.693094 26523 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.893s user 0.013s sys 0.022s
I20260809 15:46:46.691514 23394 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55320 (request call id 1293) took 1554 ms. Trace:
I20260809 15:46:46.693549 23353 leader_election.cc:302] T d1a134ba33ea4076bbaac0ae460c17ba P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:46:46.694183 26824 raft_consensus.cc:2781] T 16ca412a0dae4feebe85dd01bf1a9173 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260809 15:46:46.694591 26824 raft_consensus.cc:482] T 16ca412a0dae4feebe85dd01bf1a9173 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:46:46.694901 26824 raft_consensus.cc:3037] T 16ca412a0dae4feebe85dd01bf1a9173 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Advancing to term 5
W20260809 15:46:46.697484 26728 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.634s user 0.018s sys 0.044s
W20260809 15:46:46.660421 26574 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.754s user 0.015s sys 0.022s
W20260809 15:46:46.705860 26574 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.841s user 0.015s sys 0.022s
W20260809 15:46:46.703553 26728 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.653s user 0.018s sys 0.044s
I20260809 15:46:46.603448 26619 raft_consensus.cc:504] T e14d118b2eaa42aa8a7cd4025aba7128 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:46.603881 26836 raft_consensus.cc:504] T 294d4bcc1e7a477cb56455581277a60f P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:46.706341 26574 raft_consensus.cc:482] T 998c6cee72cf40228c6dba064c0db2da P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:46.700719 26527 consensus_queue.cc:1225] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Peer 61e69695258f48baa984c9d395e8ce3f log is divergent from this leader: its last log entry 4.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260809 15:46:46.628005 26559 consensus_queue.cc:237] T cef3e7277d7d4760824a3436ed4a847d P 65c36d164b0540f099d592f38307ee6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:46.723968 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3e94b7e50754a81b3d72a07476cb7b9" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
W20260809 15:46:46.712275 26764 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 5 current term: 5 is higher.
I20260809 15:46:46.745410 23353 leader_election.cc:302] T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c
W20260809 15:46:46.736922 26799 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.728s user 0.012s sys 0.038s
I20260809 15:46:46.725410 26668 raft_consensus.cc:2781] T d1a134ba33ea4076bbaac0ae460c17ba P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260809 15:46:46.747900 24475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16ca412a0dae4feebe85dd01bf1a9173" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:46.691584 26849 raft_consensus.cc:504] T c0741ab44d99439abc3aa3bdd27ee160 P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:46.726473 26707 leader_election.cc:288] T 763e58d6ba644eea830a6c1664d39c9b P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
W20260809 15:46:46.746208 26820 log.cc:924] Time spent T 472816994d814cb198b69e5248697440 P 61e69695258f48baa984c9d395e8ce3f: Append to log took a long time: real 0.273s user 0.000s sys 0.001s
I20260809 15:46:46.738677 26436 leader_election.cc:288] T 16ca412a0dae4feebe85dd01bf1a9173 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:46.755712 23396 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d1a134ba33ea4076bbaac0ae460c17ba"
dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
mode: GRACEFUL
new_leader_uuid: "65c36d164b0540f099d592f38307ee6c"
from {username='slave'} at 127.0.0.1:51280
I20260809 15:46:46.753366 24475 raft_consensus.cc:2454] T 16ca412a0dae4feebe85dd01bf1a9173 P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 3.
I20260809 15:46:46.736340 26815 raft_consensus.cc:482] T 30c1740ef6874df0a332fbd6fd4b03ce P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:46.757478 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "213543cdbe814c89a17a963469922c1b" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
W20260809 15:46:46.750756 26799 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.742s user 0.012s sys 0.039s
W20260809 15:46:46.751647 26611 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.533s user 0.029s sys 0.024s
I20260809 15:46:46.765023 26485 raft_consensus.cc:482] T ba56725e42f74a1baa4c04d9660727f9 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:46.765961 26771 raft_consensus.cc:3032] T 7baa2492df0340c4a5ac241959ce1ac3 P c0f38c1908414566aa9115bb2cbdd85d [term 2 LEADER]: Stepping down as leader of term 2
I20260809 15:46:46.705476 26570 raft_consensus.cc:504] T 00a1881267a14ba8b78c81040f17a014 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:46.737593 26527 raft_consensus.cc:482] T a3622f6e3e3641cea1960e3a31498bb7 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
W20260809 15:46:46.779541 26611 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.561s user 0.030s sys 0.025s
I20260809 15:46:46.772815 23228 leader_election.cc:302] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:46.720954 26574 raft_consensus.cc:504] T 998c6cee72cf40228c6dba064c0db2da P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:46.782398 26527 raft_consensus.cc:504] T a3622f6e3e3641cea1960e3a31498bb7 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:46.784180 26574 leader_election.cc:288] T 998c6cee72cf40228c6dba064c0db2da P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:46.785027 26574 raft_consensus.cc:482] T a88421dba80d4d3388aa5eb62e23853b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:46.753978 26743 leader_election.cc:288] T 213543cdbe814c89a17a963469922c1b P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:46.752978 26668 raft_consensus.cc:482] T d1a134ba33ea4076bbaac0ae460c17ba P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting leader election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:46.785490 26574 raft_consensus.cc:504] T a88421dba80d4d3388aa5eb62e23853b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:46:46.780712 24410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 372582us
I20260809 15:46:46.781322 26485 raft_consensus.cc:504] T ba56725e42f74a1baa4c04d9660727f9 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:46.748518 26399 leader_election.cc:288] T f7715d71dbac447b9351fc5eb36c5e80 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:46.778716 23223 leader_election.cc:302] T 16ca412a0dae4feebe85dd01bf1a9173 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:46:46.786921 24455 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3aa6a9a52804df0ba9b0ca3c82f38c3" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
W20260809 15:46:47.223830 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 451068us
I20260809 15:46:47.224004 26875 raft_consensus.cc:2726] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260809 15:46:47.224208 26485 leader_election.cc:288] T ba56725e42f74a1baa4c04d9660727f9 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:46.779654 26856 raft_consensus.cc:482] T a3622f6e3e3641cea1960e3a31498bb7 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
W20260809 15:46:47.226285 23223 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.68:34689, user_credentials={real_user=slave}} blocked reactor thread for 447618us
W20260809 15:46:46.868722 23286 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.515s user 0.010s sys 0.024s
W20260809 15:46:47.341089 23286 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.988s user 0.010s sys 0.034s
W20260809 15:46:46.774988 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 420505us
I20260809 15:46:47.237519 23228 leader_election.cc:302] T 00a1881267a14ba8b78c81040f17a014 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
W20260809 15:46:47.342376 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 104933us
I20260809 15:46:47.343104 23228 leader_election.cc:302] T 2424fd9bd7f14e508fd106cbfda786a4 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:47.344148 23228 leader_election.cc:302] T 538a228ddfb54bc6a36528d5ff07597b P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:46.772795 26386 leader_election.cc:288] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:46.865065 23527 raft_consensus.cc:1226] T 7baa2492df0340c4a5ac241959ce1ac3 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Rejecting Update request from peer c0f38c1908414566aa9115bb2cbdd85d for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260809 15:46:46.493741 23518 rpcz_store.cc:276] 0809 15:46:44.999947 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:45.009192 (+ 9245us) service_pool.cc:227] Handling call
0809 15:46:45.071063 (+ 61871us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 15:46:45.906481 (+835418us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:46:45.921816 (+ 15335us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 15:46:45.979994 (+ 58178us) log.cc:842] Serialized 26 byte log entry
0809 15:46:46.327972 (+347978us) raft_consensus.cc:1631] Marking committed up to 6
0809 15:46:46.328026 (+ 54us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:46.328093 (+ 67us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 15:46:46.397074 (+ 68981us) raft_consensus.cc:1671] finished
0809 15:46:46.397108 (+ 34us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:46.397267 (+ 159us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":867439,"raft.queue_time_us":784574,"raft.run_cpu_time_us":131,"raft.run_wall_time_us":45508,"thread_start_us":342595,"threads_started":1,"wal-append.queue_time_us":410371}
I20260809 15:46:47.206638 26638 raft_consensus.cc:482] T 07fcabb569414bddbd39df6cedd24e64 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
W20260809 15:46:47.284335 23517 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.800s user 0.008s sys 0.016s
W20260809 15:46:47.358325 23517 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.874s user 0.008s sys 0.016s
W20260809 15:46:47.284590 23472 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.770s user 0.003s sys 0.019s
I20260809 15:46:46.756028 26855 raft_consensus.cc:482] T c0741ab44d99439abc3aa3bdd27ee160 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
W20260809 15:46:47.285696 26843 thread.cc:636] apply [worker] (thread pool) Time spent creating pthread: real 0.516s user 0.000s sys 0.000s
I20260809 15:46:46.774334 26791 raft_consensus.cc:2726] T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260809 15:46:47.225576 26864 log.cc:924] Time spent T 563daf42d7784db195ea99c3cb21b370 P 67b3aeadfa7a432abe8af4d292e04ab9: Append to log took a long time: real 0.271s user 0.000s sys 0.000s
I20260809 15:46:47.098687 26849 leader_election.cc:288] T c0741ab44d99439abc3aa3bdd27ee160 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:46.882674 26398 raft_consensus.cc:482] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:46:47.147549 26136 raft_consensus.cc:482] T 2424fd9bd7f14e508fd106cbfda786a4 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:47.135010 26608 raft_consensus.cc:482] T 2db057fd10e149a9895dff6ef8c9f038 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:47.480229 26398 raft_consensus.cc:504] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:47.480607 26608 raft_consensus.cc:504] T 2db057fd10e149a9895dff6ef8c9f038 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:47.482343 26398 leader_election.cc:288] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:47.482452 26608 leader_election.cc:288] T 2db057fd10e149a9895dff6ef8c9f038 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:46.786827 26815 raft_consensus.cc:504] T 30c1740ef6874df0a332fbd6fd4b03ce P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:47.099825 26527 leader_election.cc:288] T a3622f6e3e3641cea1960e3a31498bb7 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:47.483024 26608 consensus_queue.cc:1046] T 0da4d5283c8a46878bd39b85e5c71f66 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260809 15:46:47.483711 26527 consensus_queue.cc:1035] T 73f6f44c259547f1b05a3a434616fa54 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 15:46:47.483933 26608 raft_consensus.cc:482] T a88421dba80d4d3388aa5eb62e23853b P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:46.958990 26862 raft_consensus.cc:482] T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
I20260809 15:46:47.332924 26570 leader_election.cc:288] T 00a1881267a14ba8b78c81040f17a014 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:46.834787 26605 leader_election.cc:288] T 7660dfa2ef694927b8eda4100a2bcd9b P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:46.778703 26436 raft_consensus.cc:482] T 07fcabb569414bddbd39df6cedd24e64 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:47.480293 26136 raft_consensus.cc:504] T 2424fd9bd7f14e508fd106cbfda786a4 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:47.485316 26436 raft_consensus.cc:504] T 07fcabb569414bddbd39df6cedd24e64 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:47.486574 26136 leader_election.cc:288] T 2424fd9bd7f14e508fd106cbfda786a4 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:47.487191 26436 leader_election.cc:288] T 07fcabb569414bddbd39df6cedd24e64 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:47.487707 26436 raft_consensus.cc:2726] T 538a228ddfb54bc6a36528d5ff07597b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260809 15:46:47.287526 26856 raft_consensus.cc:504] T a3622f6e3e3641cea1960e3a31498bb7 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:47.484364 26608 raft_consensus.cc:504] T a88421dba80d4d3388aa5eb62e23853b P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:47.335439 26574 leader_election.cc:288] T a88421dba80d4d3388aa5eb62e23853b P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:47.489846 26856 leader_election.cc:288] T a3622f6e3e3641cea1960e3a31498bb7 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:47.264462 26836 leader_election.cc:288] T 294d4bcc1e7a477cb56455581277a60f P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:47.185734 26743 raft_consensus.cc:482] T ae6d2c08a7a94f58ac8dd698e3fd456c P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 15:46:47.471402 26728 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.649s user 0.024s sys 0.031s
I20260809 15:46:47.490593 26743 raft_consensus.cc:504] T ae6d2c08a7a94f58ac8dd698e3fd456c P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:47.492421 26743 leader_election.cc:288] T ae6d2c08a7a94f58ac8dd698e3fd456c P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:46.774507 26844 raft_consensus.cc:482] T 0c48a3049897452383257aa757834fb5 P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:47.493028 26844 raft_consensus.cc:504] T 0c48a3049897452383257aa757834fb5 P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:47.489820 26608 leader_election.cc:288] T a88421dba80d4d3388aa5eb62e23853b P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:47.325896 26485 raft_consensus.cc:2781] T 16ca412a0dae4feebe85dd01bf1a9173 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260809 15:46:47.210373 24452 consensus_meta.cc:287] T e14d118b2eaa42aa8a7cd4025aba7128 P 61e69695258f48baa984c9d395e8ce3f: Time spent flushing consensus metadata: real 0.880s user 0.011s sys 0.000s
I20260809 15:46:47.162232 26581 leader_election.cc:288] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:47.186407 26619 leader_election.cc:288] T e14d118b2eaa42aa8a7cd4025aba7128 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:46.779294 26771 raft_consensus.cc:729] T 7baa2492df0340c4a5ac241959ce1ac3 P c0f38c1908414566aa9115bb2cbdd85d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:46:47.238051 26668 raft_consensus.cc:3037] T d1a134ba33ea4076bbaac0ae460c17ba P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Advancing to term 3
W20260809 15:46:47.248328 23353 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.20.202.67:40553 timed out after 1.597s (SENT) [suppressed 9 similar messages]
W20260809 15:46:47.357854 23526 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.628s user 0.025s sys 0.042s
W20260809 15:46:47.544653 23526 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.815s user 0.025s sys 0.042s
W20260809 15:46:47.358974 23472 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.845s user 0.003s sys 0.019s
I20260809 15:46:47.143404 26490 leader_election.cc:288] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:47.355329 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0741ab44d99439abc3aa3bdd27ee160" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:47.470504 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7660dfa2ef694927b8eda4100a2bcd9b" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
W20260809 15:46:47.336215 26228 consensus_meta.cc:287] T 3924b648461e463b9d7925e5671d68ae P 65c36d164b0540f099d592f38307ee6c: Time spent flushing consensus metadata: real 0.854s user 0.006s sys 0.002s
I20260809 15:46:47.354888 26875 raft_consensus.cc:2726] T 00a1881267a14ba8b78c81040f17a014 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260809 15:46:47.355225 26707 raft_consensus.cc:482] T 7baa2492df0340c4a5ac241959ce1ac3 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:47.356804 26667 consensus_queue.cc:1046] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.185s
W20260809 15:46:47.505090 26728 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.779s user 0.024s sys 0.032s
W20260809 15:46:47.554103 23522 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.685s user 0.011s sys 0.045s
W20260809 15:46:47.554643 23522 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.685s user 0.011s sys 0.045s
W20260809 15:46:47.506347 26559 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.720s user 0.012s sys 0.036s
I20260809 15:46:47.478085 26386 raft_consensus.cc:482] T d3e94b7e50754a81b3d72a07476cb7b9 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
I20260809 15:46:47.549459 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3622f6e3e3641cea1960e3a31498bb7" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
W20260809 15:46:47.548498 23472 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 1.04071e+06us
W20260809 15:46:47.534731 26799 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.682s user 0.021s sys 0.031s
W20260809 15:46:47.536760 26611 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.754s user 0.021s sys 0.032s
I20260809 15:46:47.555580 23522 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56766 (request call id 1821) took 1487 ms. Trace:
I20260809 15:46:47.473026 26638 raft_consensus.cc:504] T 07fcabb569414bddbd39df6cedd24e64 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:46:47.360132 26843 thread.cc:602] apply [worker] (thread pool) Time spent starting thread: real 0.604s user 0.001s sys 0.000s
I20260809 15:46:47.506080 26485 raft_consensus.cc:482] T 16ca412a0dae4feebe85dd01bf1a9173 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:46:47.478910 26882 raft_consensus.cc:2726] T 2424fd9bd7f14e508fd106cbfda786a4 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260809 15:46:47.545033 26824 consensus_meta.cc:287] T 16ca412a0dae4feebe85dd01bf1a9173 P 65c36d164b0540f099d592f38307ee6c: Time spent flushing consensus metadata: real 0.850s user 0.008s sys 0.000s
I20260809 15:46:47.552460 26569 raft_consensus.cc:482] T 73f6f44c259547f1b05a3a434616fa54 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:46:47.582589 26386 raft_consensus.cc:504] T d3e94b7e50754a81b3d72a07476cb7b9 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:47.656579 26485 raft_consensus.cc:3037] T 16ca412a0dae4feebe85dd01bf1a9173 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Advancing to term 5
W20260809 15:46:47.559746 26559 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.772s user 0.012s sys 0.036s
I20260809 15:46:47.556393 26707 raft_consensus.cc:504] T 7baa2492df0340c4a5ac241959ce1ac3 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:46:47.672614 23227 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.758s user 0.014s sys 0.021s
W20260809 15:46:47.672951 23227 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.088s user 0.015s sys 0.021s
I20260809 15:46:47.655383 23472 leader_election.cc:302] T 4564dfc6d4094c1b9240b367ccecf3b1 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
W20260809 15:46:47.585431 26799 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.746s user 0.021s sys 0.031s
I20260809 15:46:47.478684 26855 raft_consensus.cc:504] T c0741ab44d99439abc3aa3bdd27ee160 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:47.565699 23518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5712f6b6fffc4cc5a8ddfbe63e4097df" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
W20260809 15:46:47.642099 26611 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.859s user 0.022s sys 0.033s
I20260809 15:46:47.557149 24452 raft_consensus.cc:1261] T e14d118b2eaa42aa8a7cd4025aba7128 P 61e69695258f48baa984c9d395e8ce3f [term 6 FOLLOWER]: Refusing update from remote peer 67b3aeadfa7a432abe8af4d292e04ab9: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260809 15:46:47.694017 24452 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38722 (request call id 430) took 1377 ms. Trace:
I20260809 15:46:47.553012 26228 raft_consensus.cc:504] T 3924b648461e463b9d7925e5671d68ae P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:47.640668 26875 raft_consensus.cc:482] T f7715d71dbac447b9351fc5eb36c5e80 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:47.643925 25816 consensus_queue.cc:1035] T f9e7cadcd24d45fcbfa84209225e8f2d P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.024s
I20260809 15:46:47.484688 26862 raft_consensus.cc:504] T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:47.697022 23472 leader_election.cc:302] T 472816994d814cb198b69e5248697440 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:47.694399 24452 rpcz_store.cc:276] 0809 15:46:46.316212 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:46.316371 (+ 159us) service_pool.cc:227] Handling call
0809 15:46:46.316699 (+ 328us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:47.693741 (+1377042us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:47.693960 (+ 219us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:46:47.729188 23519 raft_consensus.cc:1226] T 3924b648461e463b9d7925e5671d68ae P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Rejecting Update request from peer 61e69695258f48baa984c9d395e8ce3f for earlier term 6. Current term is 7. Ops: [6.8-6.8]
I20260809 15:46:47.691855 26882 raft_consensus.cc:482] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
W20260809 15:46:47.359302 26865 log.cc:924] Time spent T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d: Append to log took a long time: real 0.650s user 0.001s sys 0.000s
I20260809 15:46:47.698705 26881 raft_consensus.cc:482] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 15:46:47.744189 23470 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.140s user 0.023s sys 0.085s
W20260809 15:46:47.756317 23470 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.152s user 0.023s sys 0.088s
I20260809 15:46:47.696815 26860 raft_consensus.cc:482] T d2d4f749fcfd4fb3bfb8115af3471598 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:47.742046 24410 leader_election.cc:302] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c; no voters:
W20260809 15:46:47.756868 23470 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 1.16087e+06us
I20260809 15:46:47.757928 23286 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53562 (request call id 1653) took 2683 ms. Trace:
I20260809 15:46:47.661737 26824 raft_consensus.cc:504] T 16ca412a0dae4feebe85dd01bf1a9173 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:47.662001 26569 raft_consensus.cc:504] T 73f6f44c259547f1b05a3a434616fa54 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:47.760986 23286 rpcz_store.cc:276] 0809 15:46:45.074574 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:45.489872 (+415298us) service_pool.cc:227] Handling call
0809 15:46:45.490050 (+ 178us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:45.490277 (+ 227us) raft_consensus.cc:1520] Early marking committed up to index 2
0809 15:46:46.353157 (+862880us) spinlock_profiling.cc:243] Waited 9.13 ms on lock 0x7b18003b2720. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34ef1978b1 00007f34ef1a4661 00007f34e62da345 00007f34ef111a39 00007f34e62f78bb 00007f34e62f5cbf 00007f34e62f5155 00007f34ecbf5167 00007f34e922a807 00007f34e9239f0f 00007f34e9239e88
0809 15:46:47.341578 (+988421us) log.cc:842] Serialized 14 byte log entry
0809 15:46:47.757570 (+415992us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 15:46:47.757624 (+ 54us) raft_consensus.cc:1631] Marking committed up to 2
0809 15:46:47.757683 (+ 59us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:47.757748 (+ 65us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:47.757868 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":42027,"prepare.queue_time_us":230765,"prepare.run_cpu_time_us":26,"prepare.run_wall_time_us":26,"raft.queue_time_us":3379,"raft.run_cpu_time_us":100,"raft.run_wall_time_us":8274,"spinlock_wait_cycles":21006592,"thread_start_us":1259557,"threads_started":2}
W20260809 15:46:47.771075 23353 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 522739us
I20260809 15:46:47.708599 26850 raft_consensus.cc:2726] T e14d118b2eaa42aa8a7cd4025aba7128 P 61e69695258f48baa984c9d395e8ce3f [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260809 15:46:47.770040 26874 log.cc:924] Time spent T cee02fff2c9c48a19f43eb358a89e7b0 P 65c36d164b0540f099d592f38307ee6c: Append to log took a long time: real 0.129s user 0.000s sys 0.000s
I20260809 15:46:47.755769 26882 raft_consensus.cc:504] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:47.708545 26559 raft_consensus.cc:2781] T 4564dfc6d4094c1b9240b367ccecf3b1 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260809 15:46:47.649742 26771 consensus_queue.cc:260] T 7baa2492df0340c4a5ac241959ce1ac3 P c0f38c1908414566aa9115bb2cbdd85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:46:47.788512 26741 log.cc:924] Time spent T cee02fff2c9c48a19f43eb358a89e7b0 P 61e69695258f48baa984c9d395e8ce3f: Append to log took a long time: real 2.510s user 0.007s sys 0.016s
W20260809 15:46:47.788725 26896 log.cc:924] Time spent T cef3e7277d7d4760824a3436ed4a847d P 65c36d164b0540f099d592f38307ee6c: Append to log took a long time: real 0.091s user 0.000s sys 0.000s
W20260809 15:46:47.789402 26888 log.cc:924] Time spent T d0721a4403a440f383318a3cea9d1442 P 65c36d164b0540f099d592f38307ee6c: Append to log took a long time: real 0.115s user 0.000s sys 0.000s
W20260809 15:46:47.790001 26885 log.cc:924] Time spent T d3be1e356b254d22b995be32f1c12f6f P 61e69695258f48baa984c9d395e8ce3f: Append to log took a long time: real 0.126s user 0.000s sys 0.000s
I20260809 15:46:47.795984 26386 leader_election.cc:288] T d3e94b7e50754a81b3d72a07476cb7b9 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
W20260809 15:46:47.796816 26898 log.cc:924] Time spent T 907bacfa79d04a55a68c8dbe60056462 P 61e69695258f48baa984c9d395e8ce3f: Append to log took a long time: real 0.081s user 0.000s sys 0.000s
I20260809 15:46:47.764756 26728 raft_consensus.cc:482] T d3be1e356b254d22b995be32f1c12f6f P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
I20260809 15:46:47.782436 26570 raft_consensus.cc:2726] T 472816994d814cb198b69e5248697440 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260809 15:46:47.707079 26875 raft_consensus.cc:504] T f7715d71dbac447b9351fc5eb36c5e80 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:47.780215 26906 raft_consensus.cc:2781] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 61e69695258f48baa984c9d395e8ce3f [term 6 FOLLOWER]: Leader election won for term 6
I20260809 15:46:47.763316 26881 raft_consensus.cc:504] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:47.784031 24452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f9d6565aa4c45a3b3f01210d0eb71f7" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:47.813583 24471 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38722 (request call id 421) took 2917 ms. Trace:
I20260809 15:46:47.815245 26855 leader_election.cc:288] T c0741ab44d99439abc3aa3bdd27ee160 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:47.820603 24471 rpcz_store.cc:276] 0809 15:46:44.896342 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:44.896464 (+ 122us) service_pool.cc:227] Handling call
0809 15:46:44.913887 (+ 17423us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 15:46:44.914489 (+ 602us) raft_consensus.cc:1520] Early marking committed up to index 5
0809 15:46:44.914512 (+ 23us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 15:46:44.914869 (+ 357us) log.cc:842] Serialized 26 byte log entry
0809 15:46:45.122306 (+207437us) raft_consensus.cc:1631] Marking committed up to 5
0809 15:46:45.122342 (+ 36us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:45.122405 (+ 63us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 15:46:47.782730 (+2660325us) spinlock_profiling.cc:243] Waited 6.81 ms on lock 0x7b44000232cc. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e613cf11 00007f34e620aa95 00007f34e620b4b2 00007f34e8eb3511 00007f34e8eaee1a 00007f34e8ea2bee 00007f34e8ea1f36 00007f34e8ea1b58 00007f34e8eab551 00007f34e8eab509 00007f34e8eab499
0809 15:46:47.805671 (+ 22941us) raft_consensus.cc:1671] finished
0809 15:46:47.805697 (+ 26us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:47.813502 (+ 7805us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":15653632,"thread_start_us":207162,"threads_started":1,"wal-append.queue_time_us":335458}
I20260809 15:46:47.764490 26860 raft_consensus.cc:504] T d2d4f749fcfd4fb3bfb8115af3471598 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:47.822342 26757 raft_consensus.cc:3032] T 0da4d5283c8a46878bd39b85e5c71f66 P c0f38c1908414566aa9115bb2cbdd85d [term 4 LEADER]: Stepping down as leader of term 4
I20260809 15:46:47.822719 26757 raft_consensus.cc:729] T 0da4d5283c8a46878bd39b85e5c71f66 P c0f38c1908414566aa9115bb2cbdd85d [term 4 LEADER]: Becoming Follower/Learner. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:46:47.822719 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e14d118b2eaa42aa8a7cd4025aba7128" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:47.823433 24471 raft_consensus.cc:2454] T e14d118b2eaa42aa8a7cd4025aba7128 P 61e69695258f48baa984c9d395e8ce3f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 6.
I20260809 15:46:47.824110 26659 raft_consensus.cc:3032] T e71a05ba1d4d42bd95c3f85f9313399e P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 LEADER]: Stepping down as leader of term 4
I20260809 15:46:47.824297 26862 leader_election.cc:288] T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:47.825022 26659 raft_consensus.cc:729] T e71a05ba1d4d42bd95c3f85f9313399e P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Running, Role: LEADER
I20260809 15:46:47.825860 26659 consensus_queue.cc:260] T e71a05ba1d4d42bd95c3f85f9313399e P 67b3aeadfa7a432abe8af4d292e04ab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:47.827068 26906 raft_consensus.cc:686] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 61e69695258f48baa984c9d395e8ce3f [term 6 LEADER]: Becoming Leader. State: Replica: 61e69695258f48baa984c9d395e8ce3f, State: Running, Role: LEADER
I20260809 15:46:47.828038 26906 consensus_queue.cc:237] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:47.813771 26771 raft_consensus.cc:3037] T 7baa2492df0340c4a5ac241959ce1ac3 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:47.832288 26569 leader_election.cc:288] T 73f6f44c259547f1b05a3a434616fa54 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:47.834611 26881 leader_election.cc:288] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:47.814630 26638 leader_election.cc:288] T 07fcabb569414bddbd39df6cedd24e64 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:47.835443 26824 leader_election.cc:288] T 16ca412a0dae4feebe85dd01bf1a9173 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 5 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:47.836827 26757 consensus_queue.cc:260] T 0da4d5283c8a46878bd39b85e5c71f66 P c0f38c1908414566aa9115bb2cbdd85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:47.837913 26668 raft_consensus.cc:504] T d1a134ba33ea4076bbaac0ae460c17ba P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:47.838407 26757 raft_consensus.cc:3037] T 0da4d5283c8a46878bd39b85e5c71f66 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Advancing to term 5
I20260809 15:46:47.799329 26815 leader_election.cc:288] T 30c1740ef6874df0a332fbd6fd4b03ce P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
W20260809 15:46:47.805904 26420 consensus_meta.cc:287] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 61e69695258f48baa984c9d395e8ce3f: Time spent flushing consensus metadata: real 1.334s user 0.006s sys 0.004s
I20260809 15:46:47.839743 26882 leader_election.cc:288] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:47.823658 26860 leader_election.cc:288] T d2d4f749fcfd4fb3bfb8115af3471598 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:47.803372 26559 raft_consensus.cc:482] T 4564dfc6d4094c1b9240b367ccecf3b1 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Starting leader election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
W20260809 15:46:47.741034 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 395803us
W20260809 15:46:47.919627 23351 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.565s user 0.032s sys 0.048s
W20260809 15:46:47.921973 23223 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.68:34689, user_credentials={real_user=slave}} blocked reactor thread for 43827.9us
I20260809 15:46:47.922438 23392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e71a05ba1d4d42bd95c3f85f9313399e" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:46.693894 23394 rpcz_store.cc:276] 0809 15:46:45.137690 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:45.442928 (+305238us) service_pool.cc:227] Handling call
0809 15:46:45.443218 (+ 290us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:46.354333 (+911115us) spinlock_profiling.cc:243] Waited 44.8 ms on lock 0x7b5400120d20. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e91b0730 00007f34e9222300 00007f34e9223371 00007f34e921fedf 00007f34e921f99e 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc
0809 15:46:46.691167 (+336834us) raft_consensus.cc:1520] Early marking committed up to index 1
0809 15:46:46.691196 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 15:46:46.691233 (+ 37us) raft_consensus.cc:1631] Marking committed up to 1
0809 15:46:46.691279 (+ 46us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:46.691340 (+ 61us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:46.691446 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":8452,"raft.run_cpu_time_us":125,"raft.run_wall_time_us":2592,"spinlock_wait_cycles":103075712}
I20260809 15:46:47.928426 23396 raft_consensus.cc:595] T d1a134ba33ea4076bbaac0ae460c17ba P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Received request to transfer leadership to TS 65c36d164b0540f099d592f38307ee6c
I20260809 15:46:47.930096 23400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e17dcb5efe45b8b6a8c42da03b4819" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
W20260809 15:46:47.932418 23470 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 171568us
W20260809 15:46:47.933571 23402 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55332 (request call id 1307) took 2130 ms (client timeout 2074 ms). Trace:
W20260809 15:46:47.934412 23472 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 237308us
I20260809 15:46:47.935732 23399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a51c0baaa6e432997c9d3552af4f254" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:47.935940 23517 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56770 (request call id 1301) took 1528 ms. Trace:
I20260809 15:46:47.936815 23393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a823d35f4f348d1b52ba9e6d80b00d2" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
W20260809 15:46:47.941322 24453 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38722 (request call id 426) took 2286 ms (client timeout 2229 ms). Trace:
I20260809 15:46:47.841125 26485 raft_consensus.cc:504] T 16ca412a0dae4feebe85dd01bf1a9173 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:47.814472 26228 leader_election.cc:288] T 3924b648461e463b9d7925e5671d68ae P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 7 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:47.947837 26638 raft_consensus.cc:482] T d2d4f749fcfd4fb3bfb8115af3471598 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:47.949115 26875 leader_election.cc:288] T f7715d71dbac447b9351fc5eb36c5e80 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:47.949745 26730 raft_consensus.cc:482] T df20671a7b874dbb98c0ef28371c1308 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:47.950389 26747 raft_consensus.cc:482] T 998c6cee72cf40228c6dba064c0db2da P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:47.952862 26815 raft_consensus.cc:482] T 28e17dcb5efe45b8b6a8c42da03b4819 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:47.956674 26668 leader_election.cc:288] T d1a134ba33ea4076bbaac0ae460c17ba P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:47.609818 23522 rpcz_store.cc:276] 0809 15:46:46.067748 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:46.273361 (+205613us) service_pool.cc:227] Handling call
0809 15:46:46.339858 (+ 66497us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:46.340060 (+ 202us) raft_consensus.cc:1520] Early marking committed up to index 3
0809 15:46:46.869109 (+529049us) log.cc:842] Serialized 14 byte log entry
0809 15:46:47.555190 (+686081us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 15:46:47.555238 (+ 48us) raft_consensus.cc:1631] Marking committed up to 3
0809 15:46:47.555318 (+ 80us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:47.555432 (+ 114us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:47.555535 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":151718,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":10407,"raft.queue_time_us":83,"raft.run_cpu_time_us":103,"raft.run_wall_time_us":10630,"thread_start_us":1170458,"threads_started":2}
W20260809 15:46:47.966332 23529 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56766 (request call id 1818) took 2127 ms (client timeout 1597 ms). Trace:
W20260809 15:46:48.023914 23193 auto_rebalancer.cc:254] failed to send replica move request: Illegal state: Replica c0f38c1908414566aa9115bb2cbdd85d is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 leader_uuid: "65c36d164b0540f099d592f38307ee6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } }
I20260809 15:46:48.024933 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3924b648461e463b9d7925e5671d68ae" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f"
I20260809 15:46:47.947754 26405 raft_consensus.cc:482] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:48.044711 23520 raft_consensus.cc:1226] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Rejecting Update request from peer 61e69695258f48baa984c9d395e8ce3f for earlier term 2. Current term is 3. Ops: [2.5-2.5]
I20260809 15:46:48.047051 24455 raft_consensus.cc:2396] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c for term 4 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260809 15:46:47.842566 26728 raft_consensus.cc:504] T d3be1e356b254d22b995be32f1c12f6f P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
W20260809 15:46:48.032097 26737 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.628s user 0.022s sys 0.037s
I20260809 15:46:48.638456 23396 raft_consensus.cc:603] T d1a134ba33ea4076bbaac0ae460c17ba P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260809 15:46:48.567744 23345 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.68:34689, user_credentials={real_user=slave}} blocked reactor thread for 216460us
I20260809 15:46:48.647921 23472 leader_election.cc:302] T 05ad7d8fdca64e3da527280ca5be9280 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:46:48.649369 23228 leader_election.cc:302] T 3296ef50be254fc29cfc77d22a8f7a23 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:48.653754 23470 leader_election.cc:302] T 0da4d5283c8a46878bd39b85e5c71f66 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters: c0f38c1908414566aa9115bb2cbdd85d
W20260809 15:46:48.656525 23521 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44018 (request call id 182) took 2604 ms (client timeout 1636 ms). Trace:
I20260809 15:46:48.655489 26405 raft_consensus.cc:504] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:48.657565 26559 raft_consensus.cc:3037] T 4564dfc6d4094c1b9240b367ccecf3b1 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Advancing to term 5
W20260809 15:46:48.696821 23472 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 49137us
I20260809 15:46:48.697942 23472 leader_election.cc:302] T 28e17dcb5efe45b8b6a8c42da03b4819 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:48.699412 26927 raft_consensus.cc:2781] T 05ad7d8fdca64e3da527280ca5be9280 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260809 15:46:48.699810 26927 raft_consensus.cc:482] T 05ad7d8fdca64e3da527280ca5be9280 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:46:48.700124 26399 raft_consensus.cc:2726] T 28e17dcb5efe45b8b6a8c42da03b4819 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260809 15:46:48.658890 26844 leader_election.cc:288] T 0c48a3049897452383257aa757834fb5 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:48.659072 26730 raft_consensus.cc:504] T df20671a7b874dbb98c0ef28371c1308 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:48.575534 26772 raft_consensus.cc:3032] T cef3e7277d7d4760824a3436ed4a847d P c0f38c1908414566aa9115bb2cbdd85d [term 5 LEADER]: Stepping down as leader of term 5
I20260809 15:46:48.662962 23394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e71a05ba1d4d42bd95c3f85f9313399e" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:48.665344 26707 leader_election.cc:288] T 7baa2492df0340c4a5ac241959ce1ac3 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:48.044838 23518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2424fd9bd7f14e508fd106cbfda786a4" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:48.060757 24452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3e94b7e50754a81b3d72a07476cb7b9" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:48.105278 26721 raft_consensus.cc:3032] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P c0f38c1908414566aa9115bb2cbdd85d [term 3 LEADER]: Stepping down as leader of term 3
I20260809 15:46:48.669495 23115 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:56216 (request call id 252) took 10589 ms. Trace:
I20260809 15:46:48.669792 26728 leader_election.cc:288] T d3be1e356b254d22b995be32f1c12f6f P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:48.070533 23519 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44018 (request call id 184) took 1575 ms. Trace:
I20260809 15:46:48.717337 23519 rpcz_store.cc:276] 0809 15:46:46.494936 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:46.511199 (+ 16263us) service_pool.cc:227] Handling call
0809 15:46:46.585933 (+ 74734us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 15:46:47.729593 (+1143660us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:48.070018 (+340425us) spinlock_profiling.cc:243] Waited 1.14 s on lock 0x7b54003bb0d4. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e9220f51 00007f34e921f99e 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772 00007f34e764a94c 00007f34e7649d86
0809 15:46:48.070377 (+ 359us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2628714880}
I20260809 15:46:48.718837 23519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2db057fd10e149a9895dff6ef8c9f038" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:48.720108 26485 leader_election.cc:288] T 16ca412a0dae4feebe85dd01bf1a9173 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 election: Requested vote from peers 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:48.720357 23519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "294d4bcc1e7a477cb56455581277a60f" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:48.062304 23528 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56766 (request call id 1822) took 1346 ms. Trace:
I20260809 15:46:48.721711 23528 rpcz_store.cc:276] 0809 15:46:46.715623 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:46.716090 (+ 467us) service_pool.cc:227] Handling call
0809 15:46:46.716409 (+ 319us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:47.225095 (+508686us) raft_consensus.cc:1520] Early marking committed up to index 4
0809 15:46:47.225431 (+ 336us) log.cc:842] Serialized 14 byte log entry
0809 15:46:47.740451 (+515020us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 15:46:47.740500 (+ 49us) raft_consensus.cc:1631] Marking committed up to 4
0809 15:46:47.740552 (+ 52us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:48.061852 (+321300us) spinlock_profiling.cc:243] Waited 508 ms on lock 0x7b540016ec54. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e9220f51 00007f34e921f99e 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772 00007f34e764a94c 00007f34e7649d86
0809 15:46:48.061926 (+ 74us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:48.062143 (+ 217us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":50686,"prepare.queue_time_us":12946,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":39,"raft.queue_time_us":39726,"raft.run_cpu_time_us":109,"raft.run_wall_time_us":14405,"spinlock_wait_cycles":1169393024,"thread_start_us":475027,"threads_started":1,"wal-append.queue_time_us":516951}
W20260809 15:46:48.638187 24453 rpcz_store.cc:269] 0809 15:46:45.654358 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:45.658662 (+ 4304us) service_pool.cc:227] Handling call
0809 15:46:47.940743 (+2282081us) spinlock_profiling.cc:243] Waited 614 ms on lock 0x7b5400630bd4. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e92242ae 00007f34ee161c27 00007f34e7849c04 00007f34e7849b92 00007f34e7849ac1 00007f34e7849a3c 00007f34e7848cf2 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:46:47.941058 (+ 315us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1413284096}
I20260809 15:46:48.670698 23392 raft_consensus.cc:2379] T e71a05ba1d4d42bd95c3f85f9313399e P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 61e69695258f48baa984c9d395e8ce3f in current term 5: Already voted for candidate c0f38c1908414566aa9115bb2cbdd85d in this term.
I20260809 15:46:48.723604 26815 raft_consensus.cc:504] T 28e17dcb5efe45b8b6a8c42da03b4819 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:48.724515 24453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3aa6a9a52804df0ba9b0ca3c82f38c3" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:48.726459 23472 leader_election.cc:302] T d1a134ba33ea4076bbaac0ae460c17ba P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:46:48.672549 26659 raft_consensus.cc:3037] T e71a05ba1d4d42bd95c3f85f9313399e P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Advancing to term 5
I20260809 15:46:48.636560 24463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bcac4b24e7a4448abdc934bba5a624e" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
W20260809 15:46:48.727623 24463 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38724 (request call id 472) took 2251 ms (client timeout 1925 ms). Trace:
W20260809 15:46:48.728029 24463 rpcz_store.cc:269] 0809 15:46:46.476441 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:46.477299 (+ 858us) service_pool.cc:227] Handling call
0809 15:46:48.636231 (+2158932us) spinlock_profiling.cc:243] Waited 8.26 ms on lock 0x7b5800005498. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e613cf11 00007f34e62c0851 00007f34e62c0702 00007f34ef1b0b09 00007f34ee0ec730 00007f34ee1b279b 00007f34ee1b2821 00007f34ee1621b0 00007f34ee161a94 00007f34e7849c04 00007f34e7849b92
0809 15:46:48.727549 (+ 91318us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":19003136}
I20260809 15:46:48.729067 24463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3be1e356b254d22b995be32f1c12f6f" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:48.676497 24455 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38734 (request call id 277) took 1904 ms. Trace:
I20260809 15:46:48.729902 24455 rpcz_store.cc:276] 0809 15:46:46.772081 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:46.774311 (+ 2230us) service_pool.cc:227] Handling call
0809 15:46:48.675943 (+1901632us) spinlock_profiling.cc:243] Waited 811 ms on lock 0x7b540047fc54. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e92242ae 00007f34ee161c27 00007f34e7849c04 00007f34e7849b92 00007f34e7849ac1 00007f34e7849a3c 00007f34e7848cf2 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:46:48.676310 (+ 367us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1865969024}
I20260809 15:46:48.638604 23517 rpcz_store.cc:276] 0809 15:46:46.406924 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:46.483585 (+ 76661us) service_pool.cc:227] Handling call
0809 15:46:46.483892 (+ 307us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 15:46:46.484100 (+ 208us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:46:46.484123 (+ 23us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 15:46:46.484526 (+ 403us) log.cc:842] Serialized 26 byte log entry
0809 15:46:47.358683 (+874157us) raft_consensus.cc:1631] Marking committed up to 5
0809 15:46:47.358723 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:47.358787 (+ 64us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 15:46:47.935360 (+576573us) raft_consensus.cc:1671] finished
0809 15:46:47.935394 (+ 34us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:47.935690 (+ 296us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":873888,"threads_started":1,"wal-append.queue_time_us":1066475}
W20260809 15:46:48.638947 23402 rpcz_store.cc:269] 0809 15:46:45.802601 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:46.094929 (+292328us) service_pool.cc:227] Handling call
0809 15:46:47.933015 (+1838086us) spinlock_profiling.cc:243] Waited 377 ms on lock 0x7b540025dd54. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e92242ae 00007f34ee161c27 00007f34e7849c04 00007f34e7849b92 00007f34e7849ac1 00007f34e7849a3c 00007f34e7848cf2 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:46:47.933285 (+ 270us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":866126592}
I20260809 15:46:48.731904 23517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30c1740ef6874df0a332fbd6fd4b03ce" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:48.635257 23527 raft_consensus.cc:1226] T cef3e7277d7d4760824a3436ed4a847d P 65c36d164b0540f099d592f38307ee6c [term 6 LEADER]: Rejecting Update request from peer c0f38c1908414566aa9115bb2cbdd85d for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20260809 15:46:48.733227 23517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7715d71dbac447b9351fc5eb36c5e80" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:48.733973 23527 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56766 (request call id 1827) took 1882 ms. Trace:
I20260809 15:46:48.734331 23527 rpcz_store.cc:276] 0809 15:46:46.851448 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:47.560011 (+708563us) service_pool.cc:227] Handling call
0809 15:46:48.634414 (+1074403us) spinlock_profiling.cc:243] Waited 9.32 ms on lock 0x7b5800004298. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e613cf11 00007f34e62c0851 00007f34e62c0702 00007f34ef1b0b09 00007f34ee0ec730 00007f34ee1b279b 00007f34ee1b2821 00007f34ee1762b0 00007f34ee1617f9 00007f34ee1612a2 00007f34e7847684
0809 15:46:48.634720 (+ 306us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 15:46:48.733722 (+ 99002us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:48.733922 (+ 200us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":21434240}
I20260809 15:46:48.735360 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07fcabb569414bddbd39df6cedd24e64" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:48.736495 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1a134ba33ea4076bbaac0ae460c17ba" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
W20260809 15:46:48.624806 26852 log.cc:924] Time spent T 563daf42d7784db195ea99c3cb21b370 P c0f38c1908414566aa9115bb2cbdd85d: Append to log took a long time: real 1.527s user 0.011s sys 0.004s
W20260809 15:46:48.639408 23529 rpcz_store.cc:269] 0809 15:46:45.838354 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:45.838595 (+ 241us) service_pool.cc:227] Handling call
0809 15:46:47.965856 (+2127261us) spinlock_profiling.cc:243] Waited 367 ms on lock 0x7b540016d854. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e92242ae 00007f34ee161c27 00007f34e7849c04 00007f34e7849b92 00007f34e7849ac1 00007f34e7849a3c 00007f34e7848cf2 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:46:47.966079 (+ 223us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":843857536}
I20260809 15:46:48.740609 23518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3924b648461e463b9d7925e5671d68ae" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:48.659448 26906 raft_consensus.cc:482] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:48.741654 23518 raft_consensus.cc:2379] T 3924b648461e463b9d7925e5671d68ae P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d in current term 7: Already voted for candidate 65c36d164b0540f099d592f38307ee6c in this term.
I20260809 15:46:48.741518 26906 raft_consensus.cc:504] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:48.742892 24453 raft_consensus.cc:2396] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d for term 4 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260809 15:46:48.678197 23520 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44018 (request call id 179) took 2626 ms. Trace:
I20260809 15:46:48.743402 23520 rpcz_store.cc:276] 0809 15:46:46.051353 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:47.939893 (+1888540us) service_pool.cc:227] Handling call
0809 15:46:48.036456 (+ 96563us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 15:46:48.639310 (+602854us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:48.677911 (+ 38601us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 15:46:48.744115 26757 consensus_meta.cc:287] T 0da4d5283c8a46878bd39b85e5c71f66 P c0f38c1908414566aa9115bb2cbdd85d: Time spent flushing consensus metadata: real 0.902s user 0.008s sys 0.004s
W20260809 15:46:48.649255 26771 consensus_meta.cc:287] T 7baa2492df0340c4a5ac241959ce1ac3 P c0f38c1908414566aa9115bb2cbdd85d: Time spent flushing consensus metadata: real 0.817s user 0.009s sys 0.007s
I20260809 15:46:48.731249 24455 raft_consensus.cc:1226] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 61e69695258f48baa984c9d395e8ce3f [term 4 LEADER]: Rejecting Update request from peer c0f38c1908414566aa9115bb2cbdd85d for earlier term 3. Current term is 4. Ops: []
I20260809 15:46:48.745334 23520 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44018 (request call id 186) took 1517 ms. Trace:
W20260809 15:46:48.679334 24410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 317890us
I20260809 15:46:48.745731 23520 rpcz_store.cc:276] 0809 15:46:47.228104 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:48.744143 (+1516039us) service_pool.cc:227] Handling call
0809 15:46:48.744586 (+ 443us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:48.744988 (+ 402us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:46:48.745011 (+ 23us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 15:46:48.745071 (+ 60us) raft_consensus.cc:1631] Marking committed up to 5
0809 15:46:48.745106 (+ 35us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:48.745154 (+ 48us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:48.745261 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:46:48.679616 23399 raft_consensus.cc:2379] T 0a51c0baaa6e432997c9d3552af4f254 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 61e69695258f48baa984c9d395e8ce3f in current term 4: Already voted for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in this term.
I20260809 15:46:48.674414 23396 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:51280 (request call id 13) took 1931 ms. Trace:
I20260809 15:46:48.747908 23396 rpcz_store.cc:276] 0809 15:46:46.743196 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:46.743332 (+ 136us) service_pool.cc:227] Handling call
0809 15:46:48.674162 (+1930830us) spinlock_profiling.cc:243] Waited 1.15 s on lock 0x7b5400319cd4. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e921abc3 00007f34ee16481a 00007f34e7857d04 00007f34e7857c92 00007f34e7857bc1 00007f34e7857b3c 00007f34e7856df2 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:46:48.674321 (+ 159us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2655291264}
I20260809 15:46:48.748684 23400 raft_consensus.cc:2396] T 28e17dcb5efe45b8b6a8c42da03b4819 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
W20260809 15:46:48.749768 23400 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55332 (request call id 1301) took 2954 ms (client timeout 1643 ms). Trace:
W20260809 15:46:48.750173 23400 rpcz_store.cc:269] 0809 15:46:45.795597 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:45.795707 (+ 110us) service_pool.cc:227] Handling call
0809 15:46:47.929659 (+2133952us) spinlock_profiling.cc:243] Waited 19.1 ms on lock 0x7b58001d7c98. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e613cf11 00007f34e62c0851 00007f34e62c0702 00007f34ef1b0b09 00007f34ee0ec730 00007f34ee1b279b 00007f34ee1b2821 00007f34ee1621b0 00007f34ee161a94 00007f34e7849c04 00007f34e7849b92
0809 15:46:48.749530 (+819871us) spinlock_profiling.cc:243] Waited 81.4 ms on lock 0x7b5400375ad4. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e92242ae 00007f34ee161c27 00007f34e7849c04 00007f34e7849b92 00007f34e7849ac1 00007f34e7849a3c 00007f34e7848cf2 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:46:48.749658 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":231075456}
I20260809 15:46:48.694456 24475 raft_consensus.cc:1226] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 61e69695258f48baa984c9d395e8ce3f [term 6 LEADER]: Rejecting Update request from peer c0f38c1908414566aa9115bb2cbdd85d for earlier term 5. Current term is 6. Ops: []
I20260809 15:46:48.695134 25863 raft_consensus.cc:2781] T 0da4d5283c8a46878bd39b85e5c71f66 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Leader election won for term 5
I20260809 15:46:48.751499 25863 raft_consensus.cc:686] T 0da4d5283c8a46878bd39b85e5c71f66 P 65c36d164b0540f099d592f38307ee6c [term 5 LEADER]: Becoming Leader. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:46:48.707381 26721 raft_consensus.cc:729] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P c0f38c1908414566aa9115bb2cbdd85d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:46:48.752611 25863 consensus_queue.cc:237] T 0da4d5283c8a46878bd39b85e5c71f66 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:48.753055 26721 consensus_queue.cc:260] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P c0f38c1908414566aa9115bb2cbdd85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:48.754382 26721 raft_consensus.cc:3037] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:46:48.631004 23116 catalog_manager.cc:5495] T d3be1e356b254d22b995be32f1c12f6f P 61e69695258f48baa984c9d395e8ce3f reported cstate change: term changed from 1 to 4, leader changed from c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66) to 61e69695258f48baa984c9d395e8ce3f (127.20.202.68). New cstate: current_term: 4 leader_uuid: "61e69695258f48baa984c9d395e8ce3f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 15:46:48.696048 26405 leader_election.cc:288] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:48.475639 24451 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38722 (request call id 432) took 1985 ms. Trace:
I20260809 15:46:48.324070 26279 raft_consensus.cc:482] T 563daf42d7784db195ea99c3cb21b370 P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
W20260809 15:46:48.693259 26737 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 2.357s user 0.023s sys 0.046s
I20260809 15:46:48.700147 26927 raft_consensus.cc:3037] T 05ad7d8fdca64e3da527280ca5be9280 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Advancing to term 5
I20260809 15:46:48.709679 24448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3924b648461e463b9d7925e5671d68ae" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:48.704821 26772 raft_consensus.cc:729] T cef3e7277d7d4760824a3436ed4a847d P c0f38c1908414566aa9115bb2cbdd85d [term 5 LEADER]: Becoming Follower/Learner. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:46:48.711141 26730 leader_election.cc:288] T df20671a7b874dbb98c0ef28371c1308 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:48.768010 23519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae6d2c08a7a94f58ac8dd698e3fd456c" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:48.711681 26926 raft_consensus.cc:482] T 99bdb93503db4de4a1cfb0161ea0c07c P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:48.711005 23468 leader_election.cc:302] T e14d118b2eaa42aa8a7cd4025aba7128 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c; no voters:
I20260809 15:46:48.715648 26908 consensus_queue.cc:1035] T e14d118b2eaa42aa8a7cd4025aba7128 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260809 15:46:48.716373 24452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16ca412a0dae4feebe85dd01bf1a9173" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f"
I20260809 15:46:48.723064 26399 consensus_queue.cc:1035] T ae6d2c08a7a94f58ac8dd698e3fd456c P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 15:46:48.740680 23529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a88421dba80d4d3388aa5eb62e23853b" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:48.686327 23526 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56766 (request call id 1828) took 1692 ms. Trace:
W20260809 15:46:48.690336 23521 rpcz_store.cc:269] 0809 15:46:46.052355 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:46.052479 (+ 124us) service_pool.cc:227] Handling call
0809 15:46:48.656125 (+2603646us) spinlock_profiling.cc:243] Waited 254 ms on lock 0x7b54000550d4. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e92242ae 00007f34ee161c27 00007f34e7849c04 00007f34e7849b92 00007f34e7849ac1 00007f34e7849a3c 00007f34e7848cf2 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:46:48.656314 (+ 189us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":584952064}
I20260809 15:46:48.051096 26882 raft_consensus.cc:482] T df20671a7b874dbb98c0ef28371c1308 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:48.463178 26719 raft_consensus.cc:482] T ba56725e42f74a1baa4c04d9660727f9 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:48.734037 23517 raft_consensus.cc:2454] T f7715d71dbac447b9351fc5eb36c5e80 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 5.
I20260809 15:46:48.726811 23392 raft_consensus.cc:1261] T 472816994d814cb198b69e5248697440 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Refusing update from remote peer 61e69695258f48baa984c9d395e8ce3f: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260809 15:46:48.762899 21289 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260809 15:46:48.763208 23472 leader_election.cc:302] T 4bcac4b24e7a4448abdc934bba5a624e P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:48.296617 26925 raft_consensus.cc:482] T cee02fff2c9c48a19f43eb358a89e7b0 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
W20260809 15:46:48.758700 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 109703us
I20260809 15:46:48.470431 26179 leader_election.cc:288] T 0a51c0baaa6e432997c9d3552af4f254 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 4 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
W20260809 15:46:48.787104 23468 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.68:34689, user_credentials={real_user=slave}} blocked reactor thread for 76407us
I20260809 15:46:48.726758 26815 leader_election.cc:288] T 28e17dcb5efe45b8b6a8c42da03b4819 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:48.743163 23518 raft_consensus.cc:1261] T 472816994d814cb198b69e5248697440 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Refusing update from remote peer 61e69695258f48baa984c9d395e8ce3f: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260809 15:46:48.764664 26875 raft_consensus.cc:2726] T 3296ef50be254fc29cfc77d22a8f7a23 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 15:46:48.793915 23529 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56766 (request call id 1833) took 1008 ms. Trace:
I20260809 15:46:48.795341 23529 rpcz_store.cc:276] 0809 15:46:47.785109 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:48.740370 (+955261us) service_pool.cc:227] Handling call
0809 15:46:48.793859 (+ 53489us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:46:48.788165 26719 raft_consensus.cc:504] T ba56725e42f74a1baa4c04d9660727f9 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:48.774109 23526 rpcz_store.cc:276] 0809 15:46:46.993790 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:47.570597 (+576807us) service_pool.cc:227] Handling call
0809 15:46:47.570884 (+ 287us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:47.787785 (+216901us) raft_consensus.cc:1520] Early marking committed up to index 3
0809 15:46:47.938263 (+150478us) log.cc:842] Serialized 14 byte log entry
0809 15:46:48.642852 (+704589us) log.cc:842] Serialized 14 byte log entry
0809 15:46:48.653235 (+ 10383us) spinlock_profiling.cc:243] Waited 7.06 ms on lock 0x7b1800233980. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34ef1978b1 00007f34ef1a4661 00007f34e62f8638 00007f34e62fbb11 00007f34e62fbac9 00007f34e62fba59 00007f34e62fba21 00007f34e62fad1d 00007f34ef55c204 00007f34ef55c039 00007f34e62dadc6
0809 15:46:48.683954 (+ 30719us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 15:46:48.684031 (+ 77us) raft_consensus.cc:1631] Marking committed up to 3
0809 15:46:48.684121 (+ 90us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:48.685953 (+ 1832us) spinlock_profiling.cc:243] Waited 217 ms on lock 0x7b54000f82d4. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e9220f51 00007f34e921f99e 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772 00007f34e764a94c 00007f34e7649d86
0809 15:46:48.686005 (+ 52us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:48.686151 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":62153,"prepare.queue_time_us":247774,"prepare.run_cpu_time_us":139,"prepare.run_wall_time_us":10660,"raft.queue_time_us":199248,"raft.run_cpu_time_us":332,"raft.run_wall_time_us":41434,"spinlock_wait_cycles":514502144,"thread_start_us":230650,"threads_started":1}
I20260809 15:46:48.782142 26906 leader_election.cc:288] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
W20260809 15:46:48.802999 24410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 55508.9us
I20260809 15:46:48.771733 26747 raft_consensus.cc:504] T 998c6cee72cf40228c6dba064c0db2da P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:48.774987 26638 raft_consensus.cc:504] T d2d4f749fcfd4fb3bfb8115af3471598 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:48.707578 23115 rpcz_store.cc:276] 0809 15:46:38.080583 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:38.085207 (+ 4624us) service_pool.cc:227] Handling call
0809 15:46:48.669245 (+10584038us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0809 15:46:44.975632 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0809 15:46:44.976097 (+ 465us) write_op.cc:432] Acquiring schema lock in shared mode
0809 15:46:44.976131 (+ 34us) write_op.cc:435] Acquired schema lock
0809 15:46:44.976146 (+ 15us) tablet.cc:660] Decoding operations
0809 15:46:45.513285 (+537139us) write_op.cc:620] Acquiring the partition lock for write op
0809 15:46:45.526651 (+ 13366us) write_op.cc:641] Partition lock acquired for write op
0809 15:46:45.542113 (+ 15462us) tablet.cc:683] Acquiring locks for 6 operations
0809 15:46:45.587189 (+ 45076us) tablet.cc:699] Row locks acquired
0809 15:46:45.587213 (+ 24us) write_op.cc:260] PREPARE: finished
0809 15:46:45.587374 (+ 161us) write_op.cc:270] Start()
0809 15:46:45.587501 (+ 127us) write_op.cc:276] Timestamp: P: 1786290405587323 usec, L: 0
0809 15:46:45.587522 (+ 21us) op_driver.cc:331] REPLICATION: starting
0809 15:46:45.588067 (+ 545us) log.cc:842] Serialized 2723 byte log entry
0809 15:46:46.726421 (+1138354us) op_driver.cc:444] REPLICATION: finished
0809 15:46:47.478168 (+751747us) write_op.cc:301] APPLY: starting
0809 15:46:47.480362 (+ 2194us) tablet.cc:1365] starting BulkCheckPresence
0809 15:46:47.480527 (+ 165us) tablet.cc:1368] finished BulkCheckPresence
0809 15:46:47.480648 (+ 121us) tablet.cc:1370] starting ApplyRowOperation cycle
0809 15:46:47.481478 (+ 830us) tablet.cc:1381] finished ApplyRowOperation cycle
0809 15:46:47.481686 (+ 208us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=6
0809 15:46:47.481730 (+ 44us) write_op.cc:312] APPLY: finished
0809 15:46:47.481985 (+ 255us) log.cc:842] Serialized 53 byte log entry
0809 15:46:47.485511 (+ 3526us) write_op.cc:489] Releasing partition, row and schema locks
0809 15:46:47.485630 (+ 119us) write_op.cc:454] Released schema lock
0809 15:46:47.485675 (+ 45us) write_op.cc:341] FINISH: Updating metrics
0809 15:46:47.485937 (+ 262us) write_op.cc:489] Releasing partition, row and schema locks
0809 15:46:47.485971 (+ 34us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":20,"reported_tablets":15,"child_traces":[["op",{"apply.queue_time_us":734107,"apply.run_cpu_time_us":2681,"apply.run_wall_time_us":8598,"num_ops":6,"prepare.queue_time_us":1083535,"prepare.run_cpu_time_us":23554,"prepare.run_wall_time_us":673011,"raft.queue_time_us":928183,"raft.run_cpu_time_us":2192,"raft.run_wall_time_us":953292,"replication_time_us":1130504,"thread_start_us":2484332,"threads_started":4,"wal-append.queue_time_us":182107,"wal-append.run_cpu_time_us":29093,"wal-append.run_wall_time_us":2796730}]]}
I20260809 15:46:48.814289 23286 raft_consensus.cc:1226] T 30c1740ef6874df0a332fbd6fd4b03ce P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 3. Current term is 4. Ops: []
I20260809 15:46:48.803123 24452 raft_consensus.cc:3037] T 16ca412a0dae4feebe85dd01bf1a9173 P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Advancing to term 5
I20260809 15:46:48.802973 26707 raft_consensus.cc:2726] T 4bcac4b24e7a4448abdc934bba5a624e P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 15:46:48.815104 23397 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55332 (request call id 1314) took 3001 ms. Trace:
I20260809 15:46:48.807569 23518 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44018 (request call id 185) took 1576 ms. Trace:
I20260809 15:46:48.822798 23270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4564dfc6d4094c1b9240b367ccecf3b1" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:48.822517 26772 consensus_queue.cc:260] T cef3e7277d7d4760824a3436ed4a847d P c0f38c1908414566aa9115bb2cbdd85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:48.824079 26772 raft_consensus.cc:3037] T cef3e7277d7d4760824a3436ed4a847d P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Advancing to term 6
I20260809 15:46:48.825930 21289 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 15:46:48.819092 23228 leader_election.cc:302] T 953eea268e17491ca76f3a555860a512 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:46:48.764349 24451 rpcz_store.cc:276] 0809 15:46:46.489880 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:46.490720 (+ 840us) service_pool.cc:227] Handling call
0809 15:46:48.474378 (+1983658us) spinlock_profiling.cc:243] Waited 8.06 ms on lock 0x7b5800005498. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e613cf11 00007f34e62c0851 00007f34e62c0702 00007f34ef1b0b09 00007f34ee0ec730 00007f34ee1b279b 00007f34ee1b2821 00007f34ee1762b0 00007f34ee1617f9 00007f34ee1612a2 00007f34e7847684
0809 15:46:48.474681 (+ 303us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:48.475083 (+ 402us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:46:48.475112 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 15:46:48.475176 (+ 64us) raft_consensus.cc:1631] Marking committed up to 6
0809 15:46:48.475244 (+ 68us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:48.475317 (+ 73us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:48.475455 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":18536704}
I20260809 15:46:48.776439 26882 raft_consensus.cc:504] T df20671a7b874dbb98c0ef28371c1308 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:48.812520 23116 catalog_manager.cc:5495] T d3e94b7e50754a81b3d72a07476cb7b9 P 61e69695258f48baa984c9d395e8ce3f reported cstate change: term changed from 2 to 6. New cstate: current_term: 6 leader_uuid: "61e69695258f48baa984c9d395e8ce3f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 15:46:48.836442 23273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05ad7d8fdca64e3da527280ca5be9280" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:48.818956 23397 rpcz_store.cc:276] 0809 15:46:45.813356 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:46.319727 (+506371us) service_pool.cc:227] Handling call
0809 15:46:46.320048 (+ 321us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 15:46:46.321281 (+ 1233us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:46:46.321307 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 15:46:46.322005 (+ 698us) log.cc:842] Serialized 26 byte log entry
0809 15:46:46.656205 (+334200us) raft_consensus.cc:1631] Marking committed up to 5
0809 15:46:46.656239 (+ 34us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:46.656298 (+ 59us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 15:46:48.623679 (+1967381us) spinlock_profiling.cc:243] Waited 32.6 ms on lock 0x7b44000232cc. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e613cf11 00007f34e620aa95 00007f34e620b4b2 00007f34e8eb3511 00007f34e8eaee1a 00007f34e8ea2bee 00007f34e8ea1f36 00007f34e8ea1b58 00007f34e8eab551 00007f34e8eab509 00007f34e8eab499
0809 15:46:48.803094 (+179415us) raft_consensus.cc:1671] finished
0809 15:46:48.803113 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:48.814900 (+ 11787us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":74930944,"thread_start_us":333956,"threads_started":1,"wal-append.queue_time_us":529588}
I20260809 15:46:48.803786 26279 raft_consensus.cc:504] T 563daf42d7784db195ea99c3cb21b370 P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:48.806666 26926 raft_consensus.cc:504] T 99bdb93503db4de4a1cfb0161ea0c07c P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:46:48.842588 23193 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 4b30f28311964a0b9149a50495e83d8f, TS 65c36d164b0540f099d592f38307ee6c -> TS 61e69695258f48baa984c9d395e8ce3f move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:48.845082 23522 raft_consensus.cc:1226] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Rejecting Update request from peer c0f38c1908414566aa9115bb2cbdd85d for earlier term 5. Current term is 6. Ops: []
W20260809 15:46:48.846307 23193 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 4b30f28311964a0b9149a50495e83d8f, TS 65c36d164b0540f099d592f38307ee6c -> TS 61e69695258f48baa984c9d395e8ce3f move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:48.844812 26794 raft_consensus.cc:3032] T 00a1881267a14ba8b78c81040f17a014 P c0f38c1908414566aa9115bb2cbdd85d [term 5 LEADER]: Stepping down as leader of term 5
W20260809 15:46:48.847173 26726 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.692s user 0.015s sys 0.051s
W20260809 15:46:48.852276 26726 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.750s user 0.015s sys 0.052s
I20260809 15:46:48.852520 26228 raft_consensus.cc:2781] T e14d118b2eaa42aa8a7cd4025aba7128 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260809 15:46:48.852528 23274 raft_consensus.cc:1261] T 0c29937a80724ddb9d07a27534a146f3 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 2. (index mismatch)
I20260809 15:46:48.842630 26875 raft_consensus.cc:2781] T 953eea268e17491ca76f3a555860a512 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260809 15:46:48.821943 26925 raft_consensus.cc:504] T cee02fff2c9c48a19f43eb358a89e7b0 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:48.821547 23518 rpcz_store.cc:276] 0809 15:46:47.227817 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:48.742356 (+1514539us) service_pool.cc:227] Handling call
0809 15:46:48.742748 (+ 392us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:48.794682 (+ 51934us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:48.803711 (+ 9029us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 15:46:48.850626 26740 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.690s user 0.043s sys 0.056s
W20260809 15:46:48.870734 23472 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 107681us
I20260809 15:46:48.871424 23273 raft_consensus.cc:2454] T 05ad7d8fdca64e3da527280ca5be9280 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 4.
I20260809 15:46:48.852923 26228 raft_consensus.cc:482] T e14d118b2eaa42aa8a7cd4025aba7128 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:46:48.868893 21289 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 15:46:48.859984 26875 raft_consensus.cc:482] T 953eea268e17491ca76f3a555860a512 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:48.852873 23116 catalog_manager.cc:5495] T 294d4bcc1e7a477cb56455581277a60f P 61e69695258f48baa984c9d395e8ce3f reported cstate change: term changed from 2 to 5, leader changed from 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) to 65c36d164b0540f099d592f38307ee6c (127.20.202.67). New cstate: current_term: 5 leader_uuid: "65c36d164b0540f099d592f38307ee6c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
W20260809 15:46:48.875547 26740 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.769s user 0.044s sys 0.056s
I20260809 15:46:48.866629 26485 raft_consensus.cc:482] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
I20260809 15:46:48.850837 26794 raft_consensus.cc:729] T 00a1881267a14ba8b78c81040f17a014 P c0f38c1908414566aa9115bb2cbdd85d [term 5 LEADER]: Becoming Follower/Learner. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:46:48.885691 26228 raft_consensus.cc:3037] T e14d118b2eaa42aa8a7cd4025aba7128 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Advancing to term 7
I20260809 15:46:48.850471 26559 raft_consensus.cc:504] T 4564dfc6d4094c1b9240b367ccecf3b1 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:48.891714 26875 raft_consensus.cc:3037] T 953eea268e17491ca76f3a555860a512 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Advancing to term 5
I20260809 15:46:48.898965 24410 leader_election.cc:302] T e71a05ba1d4d42bd95c3f85f9313399e P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f; no voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:48.913879 26747 leader_election.cc:288] T 998c6cee72cf40228c6dba064c0db2da P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:48.916313 26926 leader_election.cc:288] T 99bdb93503db4de4a1cfb0161ea0c07c P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:48.918959 23527 raft_consensus.cc:2454] T d1a134ba33ea4076bbaac0ae460c17ba P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 3.
I20260809 15:46:48.916565 26925 leader_election.cc:288] T cee02fff2c9c48a19f43eb358a89e7b0 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:48.922099 25639 raft_consensus.cc:2781] T d1a134ba33ea4076bbaac0ae460c17ba P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260809 15:46:48.919409 23116 catalog_manager.cc:5495] T 7660dfa2ef694927b8eda4100a2bcd9b P 61e69695258f48baa984c9d395e8ce3f reported cstate change: term changed from 2 to 5, leader changed from 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) to c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66). New cstate: current_term: 5 leader_uuid: "c0f38c1908414566aa9115bb2cbdd85d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:48.924575 23274 raft_consensus.cc:1261] T 0acc47ce21c3422698d1ef7277c3ed63 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 2. (index mismatch)
I20260809 15:46:48.928267 26638 leader_election.cc:288] T d2d4f749fcfd4fb3bfb8115af3471598 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:48.928222 26836 raft_consensus.cc:2726] T e71a05ba1d4d42bd95c3f85f9313399e P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260809 15:46:48.929733 26485 raft_consensus.cc:504] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:48.932368 26485 leader_election.cc:288] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:48.932850 26485 raft_consensus.cc:482] T ae6d2c08a7a94f58ac8dd698e3fd456c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:48.928879 23116 catalog_manager.cc:5495] T e14d118b2eaa42aa8a7cd4025aba7128 P 61e69695258f48baa984c9d395e8ce3f reported cstate change: term changed from 2 to 5, leader changed from 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) to 65c36d164b0540f099d592f38307ee6c (127.20.202.67). New cstate: current_term: 5 leader_uuid: "65c36d164b0540f099d592f38307ee6c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:48.939397 23228 leader_election.cc:302] T c5c2e425a74b4e64ae70b5def750434c P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:48.923048 26719 leader_election.cc:288] T ba56725e42f74a1baa4c04d9660727f9 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
W20260809 15:46:48.946900 24410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 55371.1us
I20260809 15:46:48.933393 26485 raft_consensus.cc:504] T ae6d2c08a7a94f58ac8dd698e3fd456c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:48.949424 26882 leader_election.cc:288] T df20671a7b874dbb98c0ef28371c1308 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:48.952891 23286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5c2e425a74b4e64ae70b5def750434c" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:48.953147 24468 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38722 (request call id 435) took 1196 ms. Trace:
I20260809 15:46:48.953585 23286 raft_consensus.cc:2454] T c5c2e425a74b4e64ae70b5def750434c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 4.
I20260809 15:46:48.955372 25639 raft_consensus.cc:482] T d1a134ba33ea4076bbaac0ae460c17ba P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:46:48.955757 25639 raft_consensus.cc:3037] T d1a134ba33ea4076bbaac0ae460c17ba P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Advancing to term 4
W20260809 15:46:48.955987 23193 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 7baa2492df0340c4a5ac241959ce1ac3, TS c0f38c1908414566aa9115bb2cbdd85d -> TS 61e69695258f48baa984c9d395e8ce3f move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:46:48.956284 23193 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 7baa2492df0340c4a5ac241959ce1ac3, TS c0f38c1908414566aa9115bb2cbdd85d -> TS 61e69695258f48baa984c9d395e8ce3f move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:48.957978 23270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0721a4403a440f383318a3cea9d1442" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:48.950917 23116 catalog_manager.cc:5495] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 61e69695258f48baa984c9d395e8ce3f reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "61e69695258f48baa984c9d395e8ce3f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 15:46:48.959085 23522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e17dcb5efe45b8b6a8c42da03b4819" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:48.960299 23273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1a134ba33ea4076bbaac0ae460c17ba" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
W20260809 15:46:48.962450 26946 log.cc:924] Time spent T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d: Append to log took a long time: real 0.077s user 0.001s sys 0.000s
I20260809 15:46:48.963284 26279 leader_election.cc:288] T 563daf42d7784db195ea99c3cb21b370 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:48.964008 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3aa6a9a52804df0ba9b0ca3c82f38c3" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:48.965317 26882 raft_consensus.cc:2726] T c5c2e425a74b4e64ae70b5def750434c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260809 15:46:48.963483 23522 raft_consensus.cc:2454] T 28e17dcb5efe45b8b6a8c42da03b4819 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 5.
I20260809 15:46:48.962368 24472 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38734 (request call id 275) took 2421 ms. Trace:
I20260809 15:46:48.966362 23270 raft_consensus.cc:2454] T d0721a4403a440f383318a3cea9d1442 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 5.
I20260809 15:46:48.969564 23527 raft_consensus.cc:2454] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 3.
W20260809 15:46:48.958702 26878 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.118s user 0.017s sys 0.015s
I20260809 15:46:48.971319 23228 leader_election.cc:302] T 73f6f44c259547f1b05a3a434616fa54 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:48.953579 24468 rpcz_store.cc:276] 0809 15:46:47.756526 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:47.764714 (+ 8188us) service_pool.cc:227] Handling call
0809 15:46:48.471512 (+706798us) spinlock_profiling.cc:243] Waited 3.6 ms on lock 0x7b5800005498. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e613cf11 00007f34e62c0851 00007f34e62c0702 00007f34ef1b0b09 00007f34ee0ec730 00007f34ee1b279b 00007f34ee1b2821 00007f34ee1762b0 00007f34ee1617f9 00007f34ee1612a2 00007f34e7847684
0809 15:46:48.471822 (+ 310us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:48.852731 (+380909us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:46:48.852765 (+ 34us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 15:46:48.852808 (+ 43us) raft_consensus.cc:1631] Marking committed up to 6
0809 15:46:48.852867 (+ 59us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:48.952774 (+ 99907us) spinlock_profiling.cc:243] Waited 379 ms on lock 0x7b5400710554. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e9220f51 00007f34e921f99e 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772 00007f34e764a94c 00007f34e7649d86
0809 15:46:48.952822 (+ 48us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:48.952994 (+ 172us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":879954560}
W20260809 15:46:48.973121 26878 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.202s user 0.017s sys 0.016s
I20260809 15:46:48.966589 24472 rpcz_store.cc:276] 0809 15:46:46.487285 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:46.487405 (+ 120us) service_pool.cc:227] Handling call
0809 15:46:48.005853 (+1518448us) spinlock_profiling.cc:243] Waited 3.47 ms on lock 0x7b5800005498. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e613cf11 00007f34e62c0851 00007f34e62c0702 00007f34ef1b0b09 00007f34ee0ec730 00007f34ee1b279b 00007f34ee1b2821 00007f34ee1762b0 00007f34ee1617f9 00007f34ee1612a2 00007f34e7847684
0809 15:46:48.006234 (+ 381us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:48.019114 (+ 12880us) raft_consensus.cc:1520] Early marking committed up to index 6
0809 15:46:48.669895 (+650781us) log.cc:842] Serialized 14 byte log entry
0809 15:46:48.906277 (+236382us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 15:46:48.906329 (+ 52us) raft_consensus.cc:1631] Marking committed up to 6
0809 15:46:48.906381 (+ 52us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:48.906454 (+ 73us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:48.908284 (+ 1830us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2883,"prepare.queue_time_us":642009,"prepare.run_cpu_time_us":29,"prepare.run_wall_time_us":31,"raft.queue_time_us":3794,"raft.run_cpu_time_us":164,"raft.run_wall_time_us":1347,"spinlock_wait_cycles":7988992,"thread_start_us":230079,"threads_started":1,"wal-append.queue_time_us":257484}
I20260809 15:46:48.978029 23273 raft_consensus.cc:2454] T d1a134ba33ea4076bbaac0ae460c17ba P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 3.
I20260809 15:46:48.980890 23116 catalog_manager.cc:5495] T 907bacfa79d04a55a68c8dbe60056462 P 61e69695258f48baa984c9d395e8ce3f reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "61e69695258f48baa984c9d395e8ce3f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 15:46:48.982466 26559 leader_election.cc:288] T 4564dfc6d4094c1b9240b367ccecf3b1 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 5 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:48.999688 23286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16ca412a0dae4feebe85dd01bf1a9173" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:49.003695 23518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df20671a7b874dbb98c0ef28371c1308" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:49.004462 26485 leader_election.cc:288] T ae6d2c08a7a94f58ac8dd698e3fd456c P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
W20260809 15:46:49.008101 23351 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.654s user 0.043s sys 0.059s
W20260809 15:46:49.006263 23345 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.68:34689, user_credentials={real_user=slave}} blocked reactor thread for 283126us
I20260809 15:46:49.005297 23116 catalog_manager.cc:5495] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 61e69695258f48baa984c9d395e8ce3f reported cstate change: term changed from 1 to 4, leader changed from c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66) to 61e69695258f48baa984c9d395e8ce3f (127.20.202.68). New cstate: current_term: 4 leader_uuid: "61e69695258f48baa984c9d395e8ce3f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 15:46:48.952497 26794 consensus_queue.cc:260] T 00a1881267a14ba8b78c81040f17a014 P c0f38c1908414566aa9115bb2cbdd85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:49.008985 26659 raft_consensus.cc:2726] T 73f6f44c259547f1b05a3a434616fa54 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260809 15:46:49.013741 23286 raft_consensus.cc:2379] T 16ca412a0dae4feebe85dd01bf1a9173 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c in current term 5: Already voted for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in this term.
I20260809 15:46:49.015405 26279 consensus_queue.cc:1035] T 472816994d814cb198b69e5248697440 P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260809 15:46:49.016707 23270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "763e58d6ba644eea830a6c1664d39c9b" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:49.017498 23270 raft_consensus.cc:2454] T 763e58d6ba644eea830a6c1664d39c9b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 4.
I20260809 15:46:49.017623 23116 catalog_manager.cc:5495] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 61e69695258f48baa984c9d395e8ce3f reported cstate change: term changed from 2 to 4, leader changed from 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) to 65c36d164b0540f099d592f38307ee6c (127.20.202.67). New cstate: current_term: 4 leader_uuid: "65c36d164b0540f099d592f38307ee6c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:49.020977 26611 raft_consensus.cc:482] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 15:46:49.019506 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 1.66235e+06us
I20260809 15:46:49.027828 26611 raft_consensus.cc:504] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:49.029973 23276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e14d118b2eaa42aa8a7cd4025aba7128" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:49.030599 23393 raft_consensus.cc:1261] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Refusing update from remote peer 61e69695258f48baa984c9d395e8ce3f: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260809 15:46:49.030154 23270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7715d71dbac447b9351fc5eb36c5e80" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:49.034418 23116 catalog_manager.cc:5495] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 61e69695258f48baa984c9d395e8ce3f reported cstate change: term changed from 1 to 5. New cstate: current_term: 5 leader_uuid: "c0f38c1908414566aa9115bb2cbdd85d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
W20260809 15:46:49.039839 23193 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet d1a134ba33ea4076bbaac0ae460c17ba, TS 67b3aeadfa7a432abe8af4d292e04ab9 -> TS 61e69695258f48baa984c9d395e8ce3f move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:46:49.040199 23193 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet d1a134ba33ea4076bbaac0ae460c17ba, TS 67b3aeadfa7a432abe8af4d292e04ab9 -> TS 61e69695258f48baa984c9d395e8ce3f move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:49.011659 26927 raft_consensus.cc:504] T 05ad7d8fdca64e3da527280ca5be9280 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:49.044685 23116 catalog_manager.cc:5495] T cee02fff2c9c48a19f43eb358a89e7b0 P 61e69695258f48baa984c9d395e8ce3f reported cstate change: term changed from 2 to 3, leader changed from 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) to 65c36d164b0540f099d592f38307ee6c (127.20.202.67). New cstate: current_term: 3 leader_uuid: "65c36d164b0540f099d592f38307ee6c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
W20260809 15:46:49.047760 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 76608.7us
I20260809 15:46:49.052193 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3aa6a9a52804df0ba9b0ca3c82f38c3" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:49.052070 23270 raft_consensus.cc:2454] T f7715d71dbac447b9351fc5eb36c5e80 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 5.
I20260809 15:46:49.054745 23527 raft_consensus.cc:2454] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 3.
I20260809 15:46:49.044442 26721 raft_consensus.cc:482] T 0acc47ce21c3422698d1ef7277c3ed63 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:49.054844 26747 raft_consensus.cc:482] T e71a05ba1d4d42bd95c3f85f9313399e P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:49.060750 23521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16ca412a0dae4feebe85dd01bf1a9173" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:49.064640 23528 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44018 (request call id 191) took 1067 ms. Trace:
I20260809 15:46:49.065757 23528 rpcz_store.cc:276] 0809 15:46:47.995696 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:48.767545 (+771849us) service_pool.cc:227] Handling call
0809 15:46:48.784003 (+ 16458us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:48.833822 (+ 49819us) raft_consensus.cc:1520] Early marking committed up to index 7
0809 15:46:48.850776 (+ 16954us) log.cc:842] Serialized 14 byte log entry
0809 15:46:49.020531 (+169755us) spinlock_profiling.cc:243] Waited 1.59 ms on lock 0x7b1800388480. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34ef1978b1 00007f34ef1a4661 00007f34e62da345 00007f34ef111a39 00007f34e62f78bb 00007f34e62f5cbf 00007f34e62f7e22 00007f34e8ea15bb 00007f34e8ea7063 00007f34e8ea757b 00007f34e922a88a
0809 15:46:49.058810 (+ 38279us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 15:46:49.058859 (+ 49us) raft_consensus.cc:1631] Marking committed up to 7
0809 15:46:49.061606 (+ 2747us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:49.061675 (+ 69us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:49.061776 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":10269,"prepare.run_cpu_time_us":74,"prepare.run_wall_time_us":12461,"raft.queue_time_us":5931,"spinlock_wait_cycles":3647744,"thread_start_us":32997,"threads_started":1}
I20260809 15:46:49.068670 23518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c48a3049897452383257aa757834fb5" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:49.075753 23400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "563daf42d7784db195ea99c3cb21b370" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:49.076550 23228 leader_election.cc:302] T f9e7cadcd24d45fcbfa84209225e8f2d P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:46:49.068084 23276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ced718c1a5c4da79cb8c1ebd1435b87" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:49.052377 26875 raft_consensus.cc:504] T 953eea268e17491ca76f3a555860a512 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:49.087859 23353 leader_election.cc:302] T f7715d71dbac447b9351fc5eb36c5e80 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:46:49.084772 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3aa6a9a52804df0ba9b0ca3c82f38c3" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
W20260809 15:46:49.090306 23193 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet ba56725e42f74a1baa4c04d9660727f9, TS 67b3aeadfa7a432abe8af4d292e04ab9 -> TS 61e69695258f48baa984c9d395e8ce3f move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:46:49.090639 23193 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet ba56725e42f74a1baa4c04d9660727f9, TS 67b3aeadfa7a432abe8af4d292e04ab9 -> TS 61e69695258f48baa984c9d395e8ce3f move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:49.068583 23116 catalog_manager.cc:5495] T 3924b648461e463b9d7925e5671d68ae P 61e69695258f48baa984c9d395e8ce3f reported cstate change: term changed from 3 to 6, leader changed from 65c36d164b0540f099d592f38307ee6c (127.20.202.67) to 61e69695258f48baa984c9d395e8ce3f (127.20.202.68). New cstate: current_term: 6 leader_uuid: "61e69695258f48baa984c9d395e8ce3f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 15:46:49.062049 26728 raft_consensus.cc:482] T 6ae14a74a26f434db08c652e097385c8 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
W20260809 15:46:49.135970 24410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 53665.2us
I20260809 15:46:49.064673 26721 raft_consensus.cc:504] T 0acc47ce21c3422698d1ef7277c3ed63 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:49.069619 26730 raft_consensus.cc:482] T 907bacfa79d04a55a68c8dbe60056462 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
I20260809 15:46:49.075827 26638 raft_consensus.cc:482] T 99bdb93503db4de4a1cfb0161ea0c07c P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 15:46:49.137678 23353 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 47330us
I20260809 15:46:49.132541 23402 raft_consensus.cc:2454] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 3.
I20260809 15:46:49.076086 26611 leader_election.cc:288] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:49.089946 26668 raft_consensus.cc:2781] T f7715d71dbac447b9351fc5eb36c5e80 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260809 15:46:49.145421 23521 raft_consensus.cc:2354] T 16ca412a0dae4feebe85dd01bf1a9173 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 61e69695258f48baa984c9d395e8ce3f for earlier term 4. Current term is 5.
I20260809 15:46:49.144098 23116 catalog_manager.cc:5495] T 3296ef50be254fc29cfc77d22a8f7a23 P 61e69695258f48baa984c9d395e8ce3f reported cstate change: term changed from 1 to 2, leader changed from 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65) to 61e69695258f48baa984c9d395e8ce3f (127.20.202.68). New cstate: current_term: 2 leader_uuid: "61e69695258f48baa984c9d395e8ce3f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 15:46:49.142030 23228 leader_election.cc:302] T d3be1e356b254d22b995be32f1c12f6f P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:49.074538 26747 raft_consensus.cc:504] T e71a05ba1d4d42bd95c3f85f9313399e P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:49.148218 23353 leader_election.cc:302] T d1a134ba33ea4076bbaac0ae460c17ba P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
W20260809 15:46:49.147934 24406 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 251855us
I20260809 15:46:49.150593 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bcac4b24e7a4448abdc934bba5a624e" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:49.166692 23353 leader_election.cc:302] T 28e17dcb5efe45b8b6a8c42da03b4819 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:46:49.148828 26668 raft_consensus.cc:482] T f7715d71dbac447b9351fc5eb36c5e80 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Starting leader election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:49.170029 23116 catalog_manager.cc:5495] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 61e69695258f48baa984c9d395e8ce3f reported cstate change: term changed from 2 to 3, leader changed from 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65) to c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66). New cstate: current_term: 3 leader_uuid: "c0f38c1908414566aa9115bb2cbdd85d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:49.176501 23280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73f6f44c259547f1b05a3a434616fa54" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:49.177268 23280 raft_consensus.cc:2454] T 73f6f44c259547f1b05a3a434616fa54 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 5.
I20260809 15:46:49.177778 23285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7baa2492df0340c4a5ac241959ce1ac3" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:49.180230 26747 leader_election.cc:288] T e71a05ba1d4d42bd95c3f85f9313399e P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:49.183889 26719 raft_consensus.cc:2781] T d1a134ba33ea4076bbaac0ae460c17ba P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader election won for term 3
I20260809 15:46:49.184429 26719 raft_consensus.cc:686] T d1a134ba33ea4076bbaac0ae460c17ba P c0f38c1908414566aa9115bb2cbdd85d [term 3 LEADER]: Becoming Leader. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:46:49.185302 26719 consensus_queue.cc:237] T d1a134ba33ea4076bbaac0ae460c17ba P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:49.173542 26485 raft_consensus.cc:2726] T d3be1e356b254d22b995be32f1c12f6f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260809 15:46:49.165508 26638 raft_consensus.cc:504] T 99bdb93503db4de4a1cfb0161ea0c07c P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:49.191751 26638 leader_election.cc:288] T 99bdb93503db4de4a1cfb0161ea0c07c P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:49.192195 26638 raft_consensus.cc:482] T 7baa2492df0340c4a5ac241959ce1ac3 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:49.192580 26638 raft_consensus.cc:504] T 7baa2492df0340c4a5ac241959ce1ac3 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:49.194356 26638 leader_election.cc:288] T 7baa2492df0340c4a5ac241959ce1ac3 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:49.162053 26611 raft_consensus.cc:482] T 3d880de657e544468c82157769622a26 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
I20260809 15:46:49.195287 26611 raft_consensus.cc:504] T 3d880de657e544468c82157769622a26 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:49.197359 26611 leader_election.cc:288] T 3d880de657e544468c82157769622a26 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:49.198472 26794 raft_consensus.cc:3037] T 00a1881267a14ba8b78c81040f17a014 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Advancing to term 6
I20260809 15:46:49.199395 26800 consensus_queue.cc:1046] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 4, Time since last communication: 0.050s
I20260809 15:46:49.205200 23392 raft_consensus.cc:3032] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d [term 5 LEADER]: Stepping down as leader of term 5
I20260809 15:46:49.205561 23392 raft_consensus.cc:729] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d [term 5 LEADER]: Becoming Follower/Learner. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:46:49.206384 23392 consensus_queue.cc:260] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:49.145709 26906 consensus_queue.cc:1046] T 3924b648461e463b9d7925e5671d68ae P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 5, Time since last communication: 0.024s
I20260809 15:46:49.210013 26619 consensus_queue.cc:1046] T 30c1740ef6874df0a332fbd6fd4b03ce P 65c36d164b0540f099d592f38307ee6c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260809 15:46:49.210608 23470 leader_election.cc:302] T c5c2e425a74b4e64ae70b5def750434c P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:46:49.211225 26619 consensus_queue.cc:1035] T 0acc47ce21c3422698d1ef7277c3ed63 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 15:46:49.212102 26956 raft_consensus.cc:3032] T 30c1740ef6874df0a332fbd6fd4b03ce P 65c36d164b0540f099d592f38307ee6c [term 3 LEADER]: Stepping down as leader of term 3
I20260809 15:46:49.212615 26956 raft_consensus.cc:729] T 30c1740ef6874df0a332fbd6fd4b03ce P 65c36d164b0540f099d592f38307ee6c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:46:49.211258 26966 consensus_queue.cc:1035] T 0c29937a80724ddb9d07a27534a146f3 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
W20260809 15:46:49.213452 23353 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 46841.9us
I20260809 15:46:49.212543 26619 raft_consensus.cc:2781] T c5c2e425a74b4e64ae70b5def750434c P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260809 15:46:49.213662 26956 consensus_queue.cc:260] T 30c1740ef6874df0a332fbd6fd4b03ce P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:49.214432 26619 raft_consensus.cc:482] T c5c2e425a74b4e64ae70b5def750434c P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Starting leader election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:46:49.214828 26619 raft_consensus.cc:3037] T c5c2e425a74b4e64ae70b5def750434c P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Advancing to term 5
I20260809 15:46:49.215034 26956 raft_consensus.cc:3037] T 30c1740ef6874df0a332fbd6fd4b03ce P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:46:49.215374 26405 raft_consensus.cc:2781] T 28e17dcb5efe45b8b6a8c42da03b4819 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260809 15:46:49.215749 26405 raft_consensus.cc:482] T 28e17dcb5efe45b8b6a8c42da03b4819 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:46:49.216089 26405 raft_consensus.cc:3037] T 28e17dcb5efe45b8b6a8c42da03b4819 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Advancing to term 6
I20260809 15:46:49.217423 23402 raft_consensus.cc:1261] T 0c29937a80724ddb9d07a27534a146f3 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 2. (term mismatch)
I20260809 15:46:49.217739 23402 pending_rounds.cc:76] T 0c29937a80724ddb9d07a27534a146f3 P c0f38c1908414566aa9115bb2cbdd85d: Aborting all ops after (but not including) 1
I20260809 15:46:49.217962 23402 pending_rounds.cc:97] T 0c29937a80724ddb9d07a27534a146f3 P c0f38c1908414566aa9115bb2cbdd85d: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20260809 15:46:49.218201 23402 raft_consensus.cc:2866] T 0c29937a80724ddb9d07a27534a146f3 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 15:46:49.212170 26721 leader_election.cc:288] T 0acc47ce21c3422698d1ef7277c3ed63 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:49.194768 23395 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52856 (request call id 184) took 1243 ms. Trace:
I20260809 15:46:49.227213 23470 leader_election.cc:302] T d0721a4403a440f383318a3cea9d1442 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters: c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:49.229358 23353 leader_election.cc:302] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
W20260809 15:46:49.227128 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 94555.5us
I20260809 15:46:49.231290 24410 leader_election.cc:302] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c; no voters:
I20260809 15:46:49.156414 23276 raft_consensus.cc:2454] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 3.
I20260809 15:46:49.153381 26728 raft_consensus.cc:504] T 6ae14a74a26f434db08c652e097385c8 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:49.175132 23283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00a1881267a14ba8b78c81040f17a014" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:49.222069 26668 raft_consensus.cc:3037] T f7715d71dbac447b9351fc5eb36c5e80 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Advancing to term 5
I20260809 15:46:49.253206 25639 raft_consensus.cc:504] T d1a134ba33ea4076bbaac0ae460c17ba P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:49.160308 26730 raft_consensus.cc:504] T 907bacfa79d04a55a68c8dbe60056462 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:49.253059 23116 catalog_manager.cc:5495] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 61e69695258f48baa984c9d395e8ce3f reported cstate change: term changed from 1 to 2, leader changed from c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66) to 61e69695258f48baa984c9d395e8ce3f (127.20.202.68). New cstate: current_term: 2 leader_uuid: "61e69695258f48baa984c9d395e8ce3f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 15:46:49.281164 23522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba56725e42f74a1baa4c04d9660727f9" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
W20260809 15:46:49.286234 23470 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 59102.3us
I20260809 15:46:49.286569 23283 raft_consensus.cc:2379] T 00a1881267a14ba8b78c81040f17a014 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c in current term 6: Already voted for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in this term.
I20260809 15:46:49.289561 23404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5c2e425a74b4e64ae70b5def750434c" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:49.251744 26906 consensus_queue.cc:1035] T 472816994d814cb198b69e5248697440 P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.039s
I20260809 15:46:49.291487 23283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05ad7d8fdca64e3da527280ca5be9280" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:49.291680 23392 raft_consensus.cc:3037] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Advancing to term 6
I20260809 15:46:49.292035 26737 raft_consensus.cc:3032] T 3924b648461e463b9d7925e5671d68ae P 61e69695258f48baa984c9d395e8ce3f [term 6 LEADER]: Stepping down as leader of term 6
I20260809 15:46:49.292414 26737 raft_consensus.cc:729] T 3924b648461e463b9d7925e5671d68ae P 61e69695258f48baa984c9d395e8ce3f [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 61e69695258f48baa984c9d395e8ce3f, State: Running, Role: LEADER
I20260809 15:46:49.293299 26737 consensus_queue.cc:260] T 3924b648461e463b9d7925e5671d68ae P 61e69695258f48baa984c9d395e8ce3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:49.294459 26737 raft_consensus.cc:3037] T 3924b648461e463b9d7925e5671d68ae P 61e69695258f48baa984c9d395e8ce3f [term 6 FOLLOWER]: Advancing to term 7
I20260809 15:46:49.292069 23283 raft_consensus.cc:3037] T 05ad7d8fdca64e3da527280ca5be9280 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Advancing to term 5
I20260809 15:46:49.295804 25639 leader_election.cc:288] T d1a134ba33ea4076bbaac0ae460c17ba P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 4 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:49.227957 23395 rpcz_store.cc:276] 0809 15:46:47.917296 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:47.927427 (+ 10131us) service_pool.cc:227] Handling call
0809 15:46:47.928074 (+ 647us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:48.668621 (+740547us) raft_consensus.cc:1520] Early marking committed up to index 5
0809 15:46:48.668988 (+ 367us) log.cc:842] Serialized 14 byte log entry
0809 15:46:48.906001 (+237013us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 15:46:48.906047 (+ 46us) raft_consensus.cc:1631] Marking committed up to 5
0809 15:46:48.910549 (+ 4502us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:48.931354 (+ 20805us) spinlock_profiling.cc:243] Waited 1 ms on lock 0x7b48002814d8. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34ee0c77f1 00007f34ee0be8e7 00007f34ee0be77a 00007f34ee1b295f 00007f34ee1b2627 00007f34ee1c6b2c 00007f34ee1c6ad8 00007f34ee1c6a58 00007f34ee1c6a13 00007f34ee1c5cbc 00007f34ed147be6
0809 15:46:49.156108 (+224754us) spinlock_profiling.cc:243] Waited 740 ms on lock 0x7b5400200654. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e9220f51 00007f34e921f99e 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772 00007f34e764a94c 00007f34e7649d86
0809 15:46:49.156157 (+ 49us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:49.161027 (+ 4870us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2610,"prepare.queue_time_us":660,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":47,"raft.queue_time_us":5936,"raft.run_cpu_time_us":1907,"raft.run_wall_time_us":24113,"spinlock_wait_cycles":1704406528,"thread_start_us":230898,"threads_started":1,"wal-append.queue_time_us":273417}
I20260809 15:46:49.298858 23518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2d4f749fcfd4fb3bfb8115af3471598" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:49.255566 26966 raft_consensus.cc:2781] T d0721a4403a440f383318a3cea9d1442 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260809 15:46:49.255184 26844 raft_consensus.cc:2781] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260809 15:46:49.303936 24410 leader_election.cc:398] T 16ca412a0dae4feebe85dd01bf1a9173 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 4 pre-election: Vote denied by peer 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553) with higher term. Message: Invalid argument: T 16ca412a0dae4feebe85dd01bf1a9173 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 61e69695258f48baa984c9d395e8ce3f for earlier term 4. Current term is 5.
I20260809 15:46:49.279181 26771 raft_consensus.cc:2781] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260809 15:46:49.308005 23400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "763e58d6ba644eea830a6c1664d39c9b" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:49.303162 23521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998c6cee72cf40228c6dba064c0db2da" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:49.308130 24410 leader_election.cc:401] T 16ca412a0dae4feebe85dd01bf1a9173 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
W20260809 15:46:49.309850 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 177125us
I20260809 15:46:49.308725 23400 raft_consensus.cc:2454] T 763e58d6ba644eea830a6c1664d39c9b P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 4.
I20260809 15:46:49.285068 26927 leader_election.cc:288] T 05ad7d8fdca64e3da527280ca5be9280 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 5 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:49.317323 23273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee02fff2c9c48a19f43eb358a89e7b0" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:49.318849 23528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99bdb93503db4de4a1cfb0161ea0c07c" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:49.319456 23528 raft_consensus.cc:2454] T 99bdb93503db4de4a1cfb0161ea0c07c P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 3.
I20260809 15:46:49.320924 26619 raft_consensus.cc:504] T c5c2e425a74b4e64ae70b5def750434c P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:49.321957 26730 leader_election.cc:288] T 907bacfa79d04a55a68c8dbe60056462 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:49.322796 23228 leader_election.cc:302] T 6ae14a74a26f434db08c652e097385c8 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:46:49.324798 23395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3aa6a9a52804df0ba9b0ca3c82f38c3" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:49.324748 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0acc47ce21c3422698d1ef7277c3ed63" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:49.325600 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7baa2492df0340c4a5ac241959ce1ac3" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:49.325691 23353 leader_election.cc:302] T 99bdb93503db4de4a1cfb0161ea0c07c P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:46:49.307955 26844 raft_consensus.cc:482] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:46:49.313418 26771 raft_consensus.cc:482] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:46:49.311110 26966 raft_consensus.cc:482] T d0721a4403a440f383318a3cea9d1442 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Starting leader election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:46:49.330794 26966 raft_consensus.cc:3037] T d0721a4403a440f383318a3cea9d1442 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Advancing to term 6
I20260809 15:46:49.314828 23276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4564dfc6d4094c1b9240b367ccecf3b1" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:49.333259 23276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99bdb93503db4de4a1cfb0161ea0c07c" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:49.334306 26771 raft_consensus.cc:3037] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Advancing to term 4
W20260809 15:46:49.337657 24452 consensus_meta.cc:287] T 16ca412a0dae4feebe85dd01bf1a9173 P 61e69695258f48baa984c9d395e8ce3f: Time spent flushing consensus metadata: real 0.500s user 0.009s sys 0.000s
I20260809 15:46:49.337986 24452 raft_consensus.cc:2454] T 16ca412a0dae4feebe85dd01bf1a9173 P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 5.
I20260809 15:46:49.338425 24452 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38734 (request call id 283) took 1294 ms. Trace:
I20260809 15:46:49.338703 24452 rpcz_store.cc:276] 0809 15:46:48.043943 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:48.715987 (+672044us) service_pool.cc:227] Handling call
0809 15:46:49.338374 (+622387us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:46:49.334079 23276 raft_consensus.cc:2396] T 99bdb93503db4de4a1cfb0161ea0c07c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260809 15:46:49.318003 26659 raft_consensus.cc:2781] T f9e7cadcd24d45fcbfa84209225e8f2d P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260809 15:46:49.349398 23470 leader_election.cc:302] T 763e58d6ba644eea830a6c1664d39c9b P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:46:49.351959 23282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d880de657e544468c82157769622a26" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:49.330906 26906 raft_consensus.cc:482] T cee02fff2c9c48a19f43eb358a89e7b0 P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
I20260809 15:46:49.362370 23282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7715d71dbac447b9351fc5eb36c5e80" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:49.363344 26771 raft_consensus.cc:504] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:49.346218 26875 leader_election.cc:288] T 953eea268e17491ca76f3a555860a512 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:49.376592 23527 raft_consensus.cc:2454] T 7baa2492df0340c4a5ac241959ce1ac3 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 3.
W20260809 15:46:49.369354 26952 log.cc:924] Time spent T 00a1881267a14ba8b78c81040f17a014 P c0f38c1908414566aa9115bb2cbdd85d: Append to log took a long time: real 0.524s user 0.004s sys 0.000s
I20260809 15:46:49.366983 23283 raft_consensus.cc:2454] T 05ad7d8fdca64e3da527280ca5be9280 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 5.
I20260809 15:46:49.375574 23282 raft_consensus.cc:2379] T f7715d71dbac447b9351fc5eb36c5e80 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d in current term 5: Already voted for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in this term.
I20260809 15:46:49.343317 26179 raft_consensus.cc:2726] T 16ca412a0dae4feebe85dd01bf1a9173 P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553) with higher term. Message: Invalid argument: T 16ca412a0dae4feebe85dd01bf1a9173 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 61e69695258f48baa984c9d395e8ce3f for earlier term 4. Current term is 5.
I20260809 15:46:49.348420 26844 raft_consensus.cc:3037] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:46:49.391412 26981 raft_consensus.cc:2781] T 99bdb93503db4de4a1cfb0161ea0c07c P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260809 15:46:49.391886 26981 raft_consensus.cc:482] T 99bdb93503db4de4a1cfb0161ea0c07c P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:46:49.392237 26981 raft_consensus.cc:3037] T 99bdb93503db4de4a1cfb0161ea0c07c P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:46:49.393057 23399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3e94b7e50754a81b3d72a07476cb7b9" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:49.394263 23399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3924b648461e463b9d7925e5671d68ae" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:49.375825 26619 leader_election.cc:288] T c5c2e425a74b4e64ae70b5def750434c P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 5 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:49.377143 23400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7715d71dbac447b9351fc5eb36c5e80" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:49.396382 23394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ced718c1a5c4da79cb8c1ebd1435b87" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:49.397413 23394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00a1881267a14ba8b78c81040f17a014" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:49.397935 23394 raft_consensus.cc:2454] T 00a1881267a14ba8b78c81040f17a014 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 6.
I20260809 15:46:49.398698 23394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7baa2492df0340c4a5ac241959ce1ac3" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:49.399500 23270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16ca412a0dae4feebe85dd01bf1a9173" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:49.399641 26906 raft_consensus.cc:504] T cee02fff2c9c48a19f43eb358a89e7b0 P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
W20260809 15:46:49.399953 24410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 117195us
I20260809 15:46:49.391609 23404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7660dfa2ef694927b8eda4100a2bcd9b" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:49.402294 23393 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52856 (request call id 185) took 1484 ms. Trace:
I20260809 15:46:49.384542 24452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee02fff2c9c48a19f43eb358a89e7b0" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:49.385015 26771 leader_election.cc:288] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 4 election: Requested vote from peers 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:49.385454 26559 raft_consensus.cc:2781] T 763e58d6ba644eea830a6c1664d39c9b P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260809 15:46:49.411965 26559 raft_consensus.cc:482] T 763e58d6ba644eea830a6c1664d39c9b P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:46:49.412362 26559 raft_consensus.cc:3037] T 763e58d6ba644eea830a6c1664d39c9b P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Advancing to term 5
I20260809 15:46:49.412837 26906 leader_election.cc:288] T cee02fff2c9c48a19f43eb358a89e7b0 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:49.386176 23286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b30f28311964a0b9149a50495e83d8f" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:49.387617 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ae14a74a26f434db08c652e097385c8" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:49.360984 26728 leader_election.cc:288] T 6ae14a74a26f434db08c652e097385c8 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:49.388202 23282 raft_consensus.cc:1261] T ae6d2c08a7a94f58ac8dd698e3fd456c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260809 15:46:49.400197 23270 raft_consensus.cc:2379] T 16ca412a0dae4feebe85dd01bf1a9173 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c in current term 5: Already voted for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in this term.
I20260809 15:46:49.404733 23353 leader_election.cc:302] T 7baa2492df0340c4a5ac241959ce1ac3 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:46:49.395159 23399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73f6f44c259547f1b05a3a434616fa54" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:49.406082 23394 raft_consensus.cc:2396] T 7baa2492df0340c4a5ac241959ce1ac3 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260809 15:46:49.390800 26228 raft_consensus.cc:504] T e14d118b2eaa42aa8a7cd4025aba7128 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:49.407495 23521 raft_consensus.cc:1261] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Refusing update from remote peer 61e69695258f48baa984c9d395e8ce3f: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260809 15:46:49.385993 23395 raft_consensus.cc:2379] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c in current term 4: Already voted for candidate c0f38c1908414566aa9115bb2cbdd85d in this term.
I20260809 15:46:49.402666 23393 rpcz_store.cc:276] 0809 15:46:47.917738 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:48.665797 (+748059us) service_pool.cc:227] Handling call
0809 15:46:48.666598 (+ 801us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:49.030955 (+364357us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:49.400928 (+369973us) spinlock_profiling.cc:243] Waited 2.53 ms on lock 0x7b5400121720. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e91b0730 00007f34e9223b42 00007f34e921ff26 00007f34e921f99e 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772
0809 15:46:49.402078 (+ 1150us) spinlock_profiling.cc:243] Waited 357 ms on lock 0x7b54003750d4. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e9220f51 00007f34e921f99e 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772 00007f34e764a94c 00007f34e7649d86
0809 15:46:49.402218 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":827060864}
I20260809 15:46:49.428988 23286 raft_consensus.cc:2454] T 4b30f28311964a0b9149a50495e83d8f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 2.
I20260809 15:46:49.379104 26726 raft_consensus.cc:482] T f9e7cadcd24d45fcbfa84209225e8f2d P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:49.435447 23270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1a134ba33ea4076bbaac0ae460c17ba" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:49.438126 23270 raft_consensus.cc:2379] T d1a134ba33ea4076bbaac0ae460c17ba P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d in current term 3: Already voted for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in this term.
I20260809 15:46:49.438177 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3aa6a9a52804df0ba9b0ca3c82f38c3" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:49.439446 23526 raft_consensus.cc:3037] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:46:49.440457 23521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee02fff2c9c48a19f43eb358a89e7b0" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:49.433663 24475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a823d35f4f348d1b52ba9e6d80b00d2" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:49.441329 23470 leader_election.cc:302] T f7715d71dbac447b9351fc5eb36c5e80 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:46:49.440917 23521 raft_consensus.cc:2454] T cee02fff2c9c48a19f43eb358a89e7b0 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 4.
I20260809 15:46:49.445098 24451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d880de657e544468c82157769622a26" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:49.445770 24451 raft_consensus.cc:2454] T 3d880de657e544468c82157769622a26 P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 1.
W20260809 15:46:49.446383 26961 log.cc:924] Time spent T df20671a7b874dbb98c0ef28371c1308 P 65c36d164b0540f099d592f38307ee6c: Append to log took a long time: real 0.438s user 0.005s sys 0.006s
I20260809 15:46:49.445657 26228 leader_election.cc:288] T e14d118b2eaa42aa8a7cd4025aba7128 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 7 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:49.452071 26399 raft_consensus.cc:2781] T f7715d71dbac447b9351fc5eb36c5e80 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260809 15:46:49.452476 23286 raft_consensus.cc:1226] T 7baa2492df0340c4a5ac241959ce1ac3 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 LEADER]: Rejecting Update request from peer c0f38c1908414566aa9115bb2cbdd85d for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260809 15:46:49.452512 26399 raft_consensus.cc:482] T f7715d71dbac447b9351fc5eb36c5e80 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:46:49.453624 26399 raft_consensus.cc:3037] T f7715d71dbac447b9351fc5eb36c5e80 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Advancing to term 6
I20260809 15:46:49.452234 23392 raft_consensus.cc:1261] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Refusing update from remote peer 61e69695258f48baa984c9d395e8ce3f: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260809 15:46:49.423851 24452 raft_consensus.cc:2454] T cee02fff2c9c48a19f43eb358a89e7b0 P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 4.
I20260809 15:46:49.435627 26719 raft_consensus.cc:2781] T 7baa2492df0340c4a5ac241959ce1ac3 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260809 15:46:49.466284 23470 leader_election.cc:302] T 73f6f44c259547f1b05a3a434616fa54 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:46:49.467485 26228 raft_consensus.cc:2781] T 73f6f44c259547f1b05a3a434616fa54 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260809 15:46:49.467862 26228 raft_consensus.cc:482] T 73f6f44c259547f1b05a3a434616fa54 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Starting leader election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:46:49.468235 26228 raft_consensus.cc:3037] T 73f6f44c259547f1b05a3a434616fa54 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Advancing to term 6
I20260809 15:46:49.468832 24410 leader_election.cc:302] T cee02fff2c9c48a19f43eb358a89e7b0 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c; no voters:
I20260809 15:46:49.468041 23470 leader_election.cc:302] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:46:49.454715 26559 raft_consensus.cc:504] T 763e58d6ba644eea830a6c1664d39c9b P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:49.475596 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "907bacfa79d04a55a68c8dbe60056462" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
W20260809 15:46:49.476060 26977 log.cc:924] Time spent T cee02fff2c9c48a19f43eb358a89e7b0 P 67b3aeadfa7a432abe8af4d292e04ab9: Append to log took a long time: real 0.128s user 0.000s sys 0.000s
I20260809 15:46:49.470954 26279 consensus_queue.cc:1035] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.009s
I20260809 15:46:49.479617 26981 raft_consensus.cc:504] T 99bdb93503db4de4a1cfb0161ea0c07c P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:49.480399 23286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "213543cdbe814c89a17a963469922c1b" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:49.481556 26981 leader_election.cc:288] T 99bdb93503db4de4a1cfb0161ea0c07c P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 4 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:49.486531 24475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0741ab44d99439abc3aa3bdd27ee160" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
W20260809 15:46:49.469933 26800 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 6 current term: 6 is higher.
I20260809 15:46:49.455039 26668 raft_consensus.cc:504] T f7715d71dbac447b9351fc5eb36c5e80 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:49.466336 26737 raft_consensus.cc:482] T 3d880de657e544468c82157769622a26 P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
I20260809 15:46:49.498405 23521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99bdb93503db4de4a1cfb0161ea0c07c" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:49.469811 26719 raft_consensus.cc:482] T 7baa2492df0340c4a5ac241959ce1ac3 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:46:49.501039 23521 raft_consensus.cc:3037] T 99bdb93503db4de4a1cfb0161ea0c07c P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:46:49.502038 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16ca412a0dae4feebe85dd01bf1a9173" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f"
I20260809 15:46:49.502231 23526 raft_consensus.cc:2454] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 4.
I20260809 15:46:49.503746 24472 raft_consensus.cc:2379] T 16ca412a0dae4feebe85dd01bf1a9173 P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 in current term 5: Already voted for candidate 65c36d164b0540f099d592f38307ee6c in this term.
I20260809 15:46:49.504681 23353 leader_election.cc:302] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:46:49.506767 26668 leader_election.cc:288] T f7715d71dbac447b9351fc5eb36c5e80 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:49.507505 23400 raft_consensus.cc:2454] T f7715d71dbac447b9351fc5eb36c5e80 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 5.
I20260809 15:46:49.509559 26228 raft_consensus.cc:504] T 73f6f44c259547f1b05a3a434616fa54 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:49.510989 23286 raft_consensus.cc:1226] T 00a1881267a14ba8b78c81040f17a014 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Rejecting Update request from peer c0f38c1908414566aa9115bb2cbdd85d for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20260809 15:46:49.511353 26826 consensus_queue.cc:1035] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.076s
I20260809 15:46:49.511978 26659 raft_consensus.cc:482] T f9e7cadcd24d45fcbfa84209225e8f2d P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:49.513473 23270 raft_consensus.cc:1226] T cef3e7277d7d4760824a3436ed4a847d P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Rejecting Update request from peer c0f38c1908414566aa9115bb2cbdd85d for earlier term 5. Current term is 6. Ops: [3.2-5.3]
I20260809 15:46:49.514108 23282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3622f6e3e3641cea1960e3a31498bb7" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
W20260809 15:46:49.560019 26982 log.cc:924] Time spent T f9e7cadcd24d45fcbfa84209225e8f2d P 67b3aeadfa7a432abe8af4d292e04ab9: Append to log took a long time: real 0.125s user 0.001s sys 0.000s
I20260809 15:46:49.560274 23282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2424fd9bd7f14e508fd106cbfda786a4" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:49.560902 23282 raft_consensus.cc:2454] T 2424fd9bd7f14e508fd106cbfda786a4 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 5.
I20260809 15:46:49.561728 23282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07fcabb569414bddbd39df6cedd24e64" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:49.562860 23282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1a134ba33ea4076bbaac0ae460c17ba" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:49.528179 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7715d71dbac447b9351fc5eb36c5e80" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:49.449493 26726 raft_consensus.cc:504] T f9e7cadcd24d45fcbfa84209225e8f2d P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:49.524891 23395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99bdb93503db4de4a1cfb0161ea0c07c" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:49.523773 26668 raft_consensus.cc:2781] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Leader election won for term 4
I20260809 15:46:49.567710 27000 raft_consensus.cc:2781] T cee02fff2c9c48a19f43eb358a89e7b0 P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260809 15:46:49.568181 27000 raft_consensus.cc:482] T cee02fff2c9c48a19f43eb358a89e7b0 P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Starting leader election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
I20260809 15:46:49.568560 27000 raft_consensus.cc:3037] T cee02fff2c9c48a19f43eb358a89e7b0 P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Advancing to term 5
I20260809 15:46:49.569516 23395 raft_consensus.cc:2379] T 99bdb93503db4de4a1cfb0161ea0c07c P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c in current term 4: Already voted for candidate c0f38c1908414566aa9115bb2cbdd85d in this term.
I20260809 15:46:49.570422 23282 raft_consensus.cc:2379] T d1a134ba33ea4076bbaac0ae460c17ba P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d in current term 3: Already voted for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in this term.
W20260809 15:46:49.561687 26989 log.cc:924] Time spent T 297c82ed906d4b48b184985eed3de601 P 67b3aeadfa7a432abe8af4d292e04ab9: Append to log took a long time: real 0.058s user 0.000s sys 0.000s
I20260809 15:46:49.572314 23470 leader_election.cc:302] T 05ad7d8fdca64e3da527280ca5be9280 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:46:49.571364 23276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2db057fd10e149a9895dff6ef8c9f038" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
W20260809 15:46:49.577513 26986 log.cc:924] Time spent T 0c48a3049897452383257aa757834fb5 P 67b3aeadfa7a432abe8af4d292e04ab9: Append to log took a long time: real 0.079s user 0.000s sys 0.000s
I20260809 15:46:49.579015 23468 leader_election.cc:302] T 16ca412a0dae4feebe85dd01bf1a9173 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c; no voters:
I20260809 15:46:49.500762 26956 raft_consensus.cc:2781] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260809 15:46:49.503343 26719 raft_consensus.cc:3037] T 7baa2492df0340c4a5ac241959ce1ac3 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:46:49.587139 23282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2d4f749fcfd4fb3bfb8115af3471598" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:49.588647 25639 raft_consensus.cc:2781] T 16ca412a0dae4feebe85dd01bf1a9173 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Leader election won for term 5
I20260809 15:46:49.563987 26659 raft_consensus.cc:3037] T f9e7cadcd24d45fcbfa84209225e8f2d P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Advancing to term 5
I20260809 15:46:49.576411 23270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df20671a7b874dbb98c0ef28371c1308" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:49.591027 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e4ffcf83c234ae7910ddef3ec0b4bda" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:49.591980 23286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7baa2492df0340c4a5ac241959ce1ac3" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:49.593909 23270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e71a05ba1d4d42bd95c3f85f9313399e" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:49.590447 23274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae6d2c08a7a94f58ac8dd698e3fd456c" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:49.595517 23270 raft_consensus.cc:2396] T e71a05ba1d4d42bd95c3f85f9313399e P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260809 15:46:49.595326 23276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30c1740ef6874df0a332fbd6fd4b03ce" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:49.524329 26737 raft_consensus.cc:504] T 3d880de657e544468c82157769622a26 P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:49.597618 25639 raft_consensus.cc:686] T 16ca412a0dae4feebe85dd01bf1a9173 P 65c36d164b0540f099d592f38307ee6c [term 5 LEADER]: Becoming Leader. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:46:49.579463 26668 raft_consensus.cc:686] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P c0f38c1908414566aa9115bb2cbdd85d [term 4 LEADER]: Becoming Leader. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:46:49.594890 23274 raft_consensus.cc:2454] T ae6d2c08a7a94f58ac8dd698e3fd456c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 5.
I20260809 15:46:49.579941 24451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a51c0baaa6e432997c9d3552af4f254" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f"
I20260809 15:46:49.600703 23276 raft_consensus.cc:2379] T 30c1740ef6874df0a332fbd6fd4b03ce P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d in current term 4: Already voted for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in this term.
I20260809 15:46:49.583618 26228 leader_election.cc:288] T 73f6f44c259547f1b05a3a434616fa54 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 6 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:49.601343 23274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0acc47ce21c3422698d1ef7277c3ed63" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:49.604295 26668 consensus_queue.cc:237] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:49.591532 26981 raft_consensus.cc:482] T 05ad7d8fdca64e3da527280ca5be9280 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:49.616480 23274 raft_consensus.cc:2454] T 0acc47ce21c3422698d1ef7277c3ed63 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 5.
I20260809 15:46:49.584292 26956 raft_consensus.cc:482] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Starting leader election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:46:49.612196 23270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3be1e356b254d22b995be32f1c12f6f" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:49.594228 26611 raft_consensus.cc:2781] T 05ad7d8fdca64e3da527280ca5be9280 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Leader election won for term 5
I20260809 15:46:49.601578 23282 raft_consensus.cc:2454] T d2d4f749fcfd4fb3bfb8115af3471598 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 4.
I20260809 15:46:49.617290 23274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7715d71dbac447b9351fc5eb36c5e80" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:49.619438 23270 raft_consensus.cc:2379] T d3be1e356b254d22b995be32f1c12f6f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d in current term 5: Already voted for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in this term.
I20260809 15:46:49.621199 24451 raft_consensus.cc:2379] T 0a51c0baaa6e432997c9d3552af4f254 P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 in current term 4: Already voted for candidate 61e69695258f48baa984c9d395e8ce3f in this term.
I20260809 15:46:49.621098 23274 raft_consensus.cc:2379] T f7715d71dbac447b9351fc5eb36c5e80 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d in current term 5: Already voted for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in this term.
I20260809 15:46:49.622098 23274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "907bacfa79d04a55a68c8dbe60056462" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:49.622659 23274 raft_consensus.cc:2454] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 4.
I20260809 15:46:49.598021 26772 raft_consensus.cc:482] T 4564dfc6d4094c1b9240b367ccecf3b1 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
I20260809 15:46:49.622946 23276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ae14a74a26f434db08c652e097385c8" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:49.623384 23274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba56725e42f74a1baa4c04d9660727f9" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:49.588361 26836 consensus_queue.cc:1035] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.029s
I20260809 15:46:49.635622 23286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e17dcb5efe45b8b6a8c42da03b4819" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:49.637272 23282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99bdb93503db4de4a1cfb0161ea0c07c" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:49.637837 23282 raft_consensus.cc:3037] T 99bdb93503db4de4a1cfb0161ea0c07c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:46:49.614996 24448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e14d118b2eaa42aa8a7cd4025aba7128" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f"
I20260809 15:46:49.659621 23270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998c6cee72cf40228c6dba064c0db2da" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:49.625659 26611 raft_consensus.cc:686] T 05ad7d8fdca64e3da527280ca5be9280 P 65c36d164b0540f099d592f38307ee6c [term 5 LEADER]: Becoming Leader. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:46:49.618527 26956 raft_consensus.cc:3037] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:46:49.636816 23286 raft_consensus.cc:2454] T 28e17dcb5efe45b8b6a8c42da03b4819 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 5.
W20260809 15:46:49.661679 23470 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 89438.7us
I20260809 15:46:49.635169 25639 consensus_queue.cc:237] T 16ca412a0dae4feebe85dd01bf1a9173 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
W20260809 15:46:49.665444 23468 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.68:34689, user_credentials={real_user=slave}} blocked reactor thread for 77111.2us
W20260809 15:46:49.666162 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 350519us
I20260809 15:46:49.666733 23276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a88421dba80d4d3388aa5eb62e23853b" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:49.636876 23274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99bdb93503db4de4a1cfb0161ea0c07c" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:49.681388 23468 leader_election.cc:302] T 3d880de657e544468c82157769622a26 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c; no voters: 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:49.666996 23351 leader_election.cc:302] T 4b30f28311964a0b9149a50495e83d8f P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c
I20260809 15:46:49.664331 23392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4564dfc6d4094c1b9240b367ccecf3b1" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:49.634687 26981 raft_consensus.cc:504] T 05ad7d8fdca64e3da527280ca5be9280 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:49.711340 23394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a823d35f4f348d1b52ba9e6d80b00d2" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:49.711927 23394 raft_consensus.cc:2454] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 6.
I20260809 15:46:49.712760 26981 leader_election.cc:288] T 05ad7d8fdca64e3da527280ca5be9280 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:49.712939 23394 pending_rounds.cc:76] T 0acc47ce21c3422698d1ef7277c3ed63 P c0f38c1908414566aa9115bb2cbdd85d: Aborting all ops after (but not including) 1
I20260809 15:46:49.713253 23394 pending_rounds.cc:97] T 0acc47ce21c3422698d1ef7277c3ed63 P c0f38c1908414566aa9115bb2cbdd85d: Aborting uncommitted NO_OP operation due to leader change: 4.2
I20260809 15:46:49.713466 23394 raft_consensus.cc:2866] T 0acc47ce21c3422698d1ef7277c3ed63 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 15:46:49.715008 23280 raft_consensus.cc:1226] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Rejecting Update request from peer c0f38c1908414566aa9115bb2cbdd85d for earlier term 3. Current term is 4. Ops: [3.5-3.5]
I20260809 15:46:49.710717 23396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05ad7d8fdca64e3da527280ca5be9280" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:49.715605 23396 raft_consensus.cc:3037] T 05ad7d8fdca64e3da527280ca5be9280 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Advancing to term 5
I20260809 15:46:49.638820 26405 raft_consensus.cc:504] T 28e17dcb5efe45b8b6a8c42da03b4819 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:46:49.765743 23223 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.68:34689, user_credentials={real_user=slave}} blocked reactor thread for 55528us
I20260809 15:46:49.662652 26772 raft_consensus.cc:504] T 4564dfc6d4094c1b9240b367ccecf3b1 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:49.679214 26875 raft_consensus.cc:2781] T 6ae14a74a26f434db08c652e097385c8 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260809 15:46:49.685961 26927 raft_consensus.cc:2781] T 3d880de657e544468c82157769622a26 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260809 15:46:49.769764 23470 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 98227.8us
W20260809 15:46:49.772960 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 108817us
I20260809 15:46:49.785853 23397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1a134ba33ea4076bbaac0ae460c17ba" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:49.692587 26559 leader_election.cc:288] T 763e58d6ba644eea830a6c1664d39c9b P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 5 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:49.784327 23228 leader_election.cc:302] T 05ad7d8fdca64e3da527280ca5be9280 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:46:49.683679 26228 consensus_queue.cc:1035] T ae6d2c08a7a94f58ac8dd698e3fd456c P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.088s
I20260809 15:46:49.854751 23528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05ad7d8fdca64e3da527280ca5be9280" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:49.870532 23521 raft_consensus.cc:2454] T 99bdb93503db4de4a1cfb0161ea0c07c P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 4.
I20260809 15:46:49.766655 26730 raft_consensus.cc:2781] T 4b30f28311964a0b9149a50495e83d8f P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260809 15:46:49.772601 26875 raft_consensus.cc:482] T 6ae14a74a26f434db08c652e097385c8 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:49.784961 26927 raft_consensus.cc:482] T 3d880de657e544468c82157769622a26 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Starting leader election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
W20260809 15:46:49.872171 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 137480us
I20260809 15:46:49.714448 26844 raft_consensus.cc:504] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:49.882061 23521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9e7cadcd24d45fcbfa84209225e8f2d" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:49.881690 23353 leader_election.cc:302] T 99bdb93503db4de4a1cfb0161ea0c07c P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
W20260809 15:46:49.891884 26988 log.cc:924] Time spent T 953eea268e17491ca76f3a555860a512 P c0f38c1908414566aa9115bb2cbdd85d: Append to log took a long time: real 0.166s user 0.001s sys 0.000s
I20260809 15:46:49.892057 26810 raft_consensus.cc:482] T 2424fd9bd7f14e508fd106cbfda786a4 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:49.875525 26730 raft_consensus.cc:482] T 4b30f28311964a0b9149a50495e83d8f P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting leader election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:49.879870 26927 raft_consensus.cc:3037] T 3d880de657e544468c82157769622a26 P 65c36d164b0540f099d592f38307ee6c [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:49.877143 26875 raft_consensus.cc:3037] T 6ae14a74a26f434db08c652e097385c8 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Advancing to term 5
I20260809 15:46:49.877818 23392 raft_consensus.cc:3037] T 4564dfc6d4094c1b9240b367ccecf3b1 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Advancing to term 5
I20260809 15:46:49.896310 23228 leader_election.cc:302] T 0a51c0baaa6e432997c9d3552af4f254 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters: 61e69695258f48baa984c9d395e8ce3f
I20260809 15:46:49.892078 26668 raft_consensus.cc:2781] T 99bdb93503db4de4a1cfb0161ea0c07c P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Leader election won for term 4
W20260809 15:46:49.902727 26992 log.cc:924] Time spent T 472816994d814cb198b69e5248697440 P 65c36d164b0540f099d592f38307ee6c: Append to log took a long time: real 0.232s user 0.000s sys 0.000s
W20260809 15:46:49.903574 27020 log.cc:924] Time spent T 2424fd9bd7f14e508fd106cbfda786a4 P c0f38c1908414566aa9115bb2cbdd85d: Append to log took a long time: real 0.193s user 0.001s sys 0.000s
W20260809 15:46:49.907853 27013 log.cc:924] Time spent T 297c82ed906d4b48b184985eed3de601 P c0f38c1908414566aa9115bb2cbdd85d: Append to log took a long time: real 0.138s user 0.000s sys 0.000s
I20260809 15:46:49.877456 26726 leader_election.cc:288] T f9e7cadcd24d45fcbfa84209225e8f2d P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:49.881114 26737 leader_election.cc:288] T 3d880de657e544468c82157769622a26 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
W20260809 15:46:49.915514 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 141878us
I20260809 15:46:49.880507 26485 raft_consensus.cc:2741] T 05ad7d8fdca64e3da527280ca5be9280 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260809 15:46:49.913234 26730 raft_consensus.cc:3037] T 4b30f28311964a0b9149a50495e83d8f P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:49.926469 23521 raft_consensus.cc:1226] T d1a134ba33ea4076bbaac0ae460c17ba P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Rejecting Update request from peer c0f38c1908414566aa9115bb2cbdd85d for earlier term 3. Current term is 4. Ops: []
W20260809 15:46:49.921649 27004 log.cc:924] Time spent T c0741ab44d99439abc3aa3bdd27ee160 P 67b3aeadfa7a432abe8af4d292e04ab9: Append to log took a long time: real 0.255s user 0.001s sys 0.000s
W20260809 15:46:49.929054 23470 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 155851us
I20260809 15:46:49.881592 26611 consensus_queue.cc:237] T 05ad7d8fdca64e3da527280ca5be9280 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:49.922392 26668 raft_consensus.cc:686] T 99bdb93503db4de4a1cfb0161ea0c07c P c0f38c1908414566aa9115bb2cbdd85d [term 4 LEADER]: Becoming Leader. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:46:49.925925 26908 raft_consensus.cc:2781] T 0a51c0baaa6e432997c9d3552af4f254 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader election won for term 4
I20260809 15:46:49.948647 23274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5c2e425a74b4e64ae70b5def750434c" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:49.950378 23276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e14d118b2eaa42aa8a7cd4025aba7128" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:49.950881 23285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1a134ba33ea4076bbaac0ae460c17ba" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:49.954114 23285 raft_consensus.cc:3037] T d1a134ba33ea4076bbaac0ae460c17ba P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:46:49.928700 26747 consensus_queue.cc:1046] T d1a134ba33ea4076bbaac0ae460c17ba P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.003s
I20260809 15:46:49.977808 23351 leader_election.cc:302] T 213543cdbe814c89a17a963469922c1b P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:49.979483 23521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d880de657e544468c82157769622a26" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:49.921911 26810 raft_consensus.cc:504] T 2424fd9bd7f14e508fd106cbfda786a4 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:49.980533 23274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "763e58d6ba644eea830a6c1664d39c9b" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:49.981173 26970 raft_consensus.cc:482] T e71a05ba1d4d42bd95c3f85f9313399e P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:49.980576 23518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e17dcb5efe45b8b6a8c42da03b4819" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:49.988363 23228 leader_election.cc:302] T d1a134ba33ea4076bbaac0ae460c17ba P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:46:49.987177 23276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73f6f44c259547f1b05a3a434616fa54" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:49.987816 23274 raft_consensus.cc:3037] T 763e58d6ba644eea830a6c1664d39c9b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Advancing to term 5
I20260809 15:46:49.931077 26399 raft_consensus.cc:504] T f7715d71dbac447b9351fc5eb36c5e80 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:49.988682 23518 raft_consensus.cc:3037] T 28e17dcb5efe45b8b6a8c42da03b4819 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Advancing to term 6
I20260809 15:46:50.150779 23282 raft_consensus.cc:2454] T 99bdb93503db4de4a1cfb0161ea0c07c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 4.
I20260809 15:46:50.006349 23395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73f6f44c259547f1b05a3a434616fa54" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
W20260809 15:46:50.125145 27038 log.cc:924] Time spent T 1f9d6565aa4c45a3b3f01210d0eb71f7 P c0f38c1908414566aa9115bb2cbdd85d: Append to log took a long time: real 0.063s user 0.000s sys 0.000s
W20260809 15:46:50.123005 26966 consensus_meta.cc:287] T d0721a4403a440f383318a3cea9d1442 P 65c36d164b0540f099d592f38307ee6c: Time spent flushing consensus metadata: real 0.792s user 0.006s sys 0.003s
W20260809 15:46:50.121802 26719 consensus_meta.cc:287] T 7baa2492df0340c4a5ac241959ce1ac3 P c0f38c1908414566aa9115bb2cbdd85d: Time spent flushing consensus metadata: real 0.529s user 0.000s sys 0.006s
I20260809 15:46:50.084695 23277 raft_consensus.cc:1261] T 0da4d5283c8a46878bd39b85e5c71f66 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
W20260809 15:46:50.083663 27027 log.cc:924] Time spent T 2db057fd10e149a9895dff6ef8c9f038 P c0f38c1908414566aa9115bb2cbdd85d: Append to log took a long time: real 0.163s user 0.001s sys 0.000s
I20260809 15:46:50.138470 23345 leader_election.cc:302] T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c
I20260809 15:46:50.095258 23400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3aa6a9a52804df0ba9b0ca3c82f38c3" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:50.002842 23396 raft_consensus.cc:2454] T 05ad7d8fdca64e3da527280ca5be9280 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 5.
I20260809 15:46:49.998940 23528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4564dfc6d4094c1b9240b367ccecf3b1" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:50.236418 23528 raft_consensus.cc:2379] T 4564dfc6d4094c1b9240b367ccecf3b1 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d in current term 5: Already voted for candidate 65c36d164b0540f099d592f38307ee6c in this term.
I20260809 15:46:50.116304 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3aa6a9a52804df0ba9b0ca3c82f38c3" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:49.998456 23270 raft_consensus.cc:1261] T cef3e7277d7d4760824a3436ed4a847d P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20260809 15:46:50.199703 23526 raft_consensus.cc:2354] T f7715d71dbac447b9351fc5eb36c5e80 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d for earlier term 5. Current term is 6.
I20260809 15:46:50.139046 23285 raft_consensus.cc:2454] T d1a134ba33ea4076bbaac0ae460c17ba P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 4.
I20260809 15:46:50.119074 23529 raft_consensus.cc:1261] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
W20260809 15:46:50.039605 27005 log.cc:924] Time spent T 07fcabb569414bddbd39df6cedd24e64 P 67b3aeadfa7a432abe8af4d292e04ab9: Append to log took a long time: real 0.177s user 0.000s sys 0.001s
I20260809 15:46:50.014696 23399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "763e58d6ba644eea830a6c1664d39c9b" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:50.259564 23399 raft_consensus.cc:3037] T 763e58d6ba644eea830a6c1664d39c9b P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Advancing to term 5
I20260809 15:46:50.129685 23468 leader_election.cc:302] T cee02fff2c9c48a19f43eb358a89e7b0 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c; no voters: 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:49.996372 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5c2e425a74b4e64ae70b5def750434c" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
W20260809 15:46:50.181844 23223 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.68:34689, user_credentials={real_user=slave}} blocked reactor thread for 44186.4us
W20260809 15:46:50.049722 27010 log.cc:924] Time spent T cef3e7277d7d4760824a3436ed4a847d P c0f38c1908414566aa9115bb2cbdd85d: Append to log took a long time: real 0.168s user 0.001s sys 0.000s
I20260809 15:46:50.085507 26810 leader_election.cc:288] T 2424fd9bd7f14e508fd106cbfda786a4 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:50.235162 23274 raft_consensus.cc:2454] T 763e58d6ba644eea830a6c1664d39c9b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 5.
W20260809 15:46:50.150898 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 219498us
W20260809 15:46:50.198833 27031 log.cc:924] Time spent T 16ca412a0dae4feebe85dd01bf1a9173 P 65c36d164b0540f099d592f38307ee6c: Append to log took a long time: real 0.296s user 0.002s sys 0.000s
I20260809 15:46:50.010632 26772 leader_election.cc:288] T 4564dfc6d4094c1b9240b367ccecf3b1 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:50.042858 26844 leader_election.cc:288] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 4 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:49.979699 26405 leader_election.cc:288] T 28e17dcb5efe45b8b6a8c42da03b4819 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 6 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:50.263669 24410 leader_election.cc:302] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:50.125608 23392 raft_consensus.cc:2454] T 4564dfc6d4094c1b9240b367ccecf3b1 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 5.
I20260809 15:46:50.026361 26638 raft_consensus.cc:2726] T 213543cdbe814c89a17a963469922c1b P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260809 15:46:50.263852 23282 raft_consensus.cc:1226] T 953eea268e17491ca76f3a555860a512 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 4. Current term is 5. Ops: []
I20260809 15:46:50.259186 26719 raft_consensus.cc:504] T 7baa2492df0340c4a5ac241959ce1ac3 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:50.278880 23351 leader_election.cc:302] T a3622f6e3e3641cea1960e3a31498bb7 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
W20260809 15:46:50.265797 23468 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.68:34689, user_credentials={real_user=slave}} blocked reactor thread for 136183us
W20260809 15:46:50.296132 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 343280us
I20260809 15:46:50.166569 26875 raft_consensus.cc:504] T 6ae14a74a26f434db08c652e097385c8 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:46:50.280447 23345 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.68:34689, user_credentials={real_user=slave}} blocked reactor thread for 142050us
W20260809 15:46:50.287318 23394 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.573s user 0.011s sys 0.011s
I20260809 15:46:50.089219 26956 raft_consensus.cc:504] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:50.002329 26927 raft_consensus.cc:504] T 3d880de657e544468c82157769622a26 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:50.005403 26659 raft_consensus.cc:504] T f9e7cadcd24d45fcbfa84209225e8f2d P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:49.998646 26970 raft_consensus.cc:504] T e71a05ba1d4d42bd95c3f85f9313399e P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
W20260809 15:46:50.297932 23394 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.583s user 0.011s sys 0.011s
I20260809 15:46:50.302716 23228 leader_election.cc:302] T 763e58d6ba644eea830a6c1664d39c9b P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:46:50.281513 26925 raft_consensus.cc:2781] T cee02fff2c9c48a19f43eb358a89e7b0 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260809 15:46:50.281843 26726 raft_consensus.cc:3037] T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Advancing to term 7
I20260809 15:46:50.299290 23397 raft_consensus.cc:1261] T 0da4d5283c8a46878bd39b85e5c71f66 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260809 15:46:50.309953 23345 leader_election.cc:302] T 4bcac4b24e7a4448abdc934bba5a624e P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c
I20260809 15:46:50.235210 26966 raft_consensus.cc:504] T d0721a4403a440f383318a3cea9d1442 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:50.313354 23518 raft_consensus.cc:2454] T 28e17dcb5efe45b8b6a8c42da03b4819 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 6.
I20260809 15:46:49.996156 26668 consensus_queue.cc:237] T 99bdb93503db4de4a1cfb0161ea0c07c P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:50.342975 23351 leader_election.cc:302] T 2424fd9bd7f14e508fd106cbfda786a4 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c
W20260809 15:46:50.328354 23223 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.68:34689, user_credentials={real_user=slave}} blocked reactor thread for 66678.8us
I20260809 15:46:50.329967 23521 raft_consensus.cc:2379] T 3d880de657e544468c82157769622a26 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 61e69695258f48baa984c9d395e8ce3f in current term 2: Already voted for candidate 65c36d164b0540f099d592f38307ee6c in this term.
I20260809 15:46:50.347450 23282 raft_consensus.cc:1226] T f9e7cadcd24d45fcbfa84209225e8f2d P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 4. Current term is 5. Ops: []
I20260809 15:46:50.373688 23283 raft_consensus.cc:1226] T 6ae14a74a26f434db08c652e097385c8 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 4. Current term is 5. Ops: []
I20260809 15:46:50.373805 23274 raft_consensus.cc:1261] T 16ca412a0dae4feebe85dd01bf1a9173 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260809 15:46:50.302124 26405 raft_consensus.cc:2726] T a3622f6e3e3641cea1960e3a31498bb7 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260809 15:46:50.376637 23522 raft_consensus.cc:1226] T d0721a4403a440f383318a3cea9d1442 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Rejecting Update request from peer c0f38c1908414566aa9115bb2cbdd85d for earlier term 5. Current term is 6. Ops: []
I20260809 15:46:50.306761 26925 raft_consensus.cc:482] T cee02fff2c9c48a19f43eb358a89e7b0 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Starting leader election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
W20260809 15:46:50.346626 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 43990.2us
I20260809 15:46:50.346463 23353 leader_election.cc:302] T 28e17dcb5efe45b8b6a8c42da03b4819 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:46:50.356112 26981 raft_consensus.cc:2726] T 4bcac4b24e7a4448abdc934bba5a624e P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 15:46:50.309316 26638 raft_consensus.cc:482] T 563daf42d7784db195ea99c3cb21b370 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
W20260809 15:46:50.510003 23345 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.68:34689, user_credentials={real_user=slave}} blocked reactor thread for 200098us
I20260809 15:46:50.512774 23283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7715d71dbac447b9351fc5eb36c5e80" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
W20260809 15:46:50.517081 24410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 197997us
I20260809 15:46:50.344594 26747 consensus_queue.cc:1035] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.181s
I20260809 15:46:50.534650 23283 raft_consensus.cc:3037] T f7715d71dbac447b9351fc5eb36c5e80 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Advancing to term 6
I20260809 15:46:50.464851 26925 raft_consensus.cc:3037] T cee02fff2c9c48a19f43eb358a89e7b0 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Advancing to term 5
I20260809 15:46:50.510239 23228 leader_election.cc:302] T 99bdb93503db4de4a1cfb0161ea0c07c P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters:
W20260809 15:46:50.507758 27000 consensus_meta.cc:287] T cee02fff2c9c48a19f43eb358a89e7b0 P 61e69695258f48baa984c9d395e8ce3f: Time spent flushing consensus metadata: real 0.939s user 0.008s sys 0.002s
I20260809 15:46:50.537127 27000 raft_consensus.cc:504] T cee02fff2c9c48a19f43eb358a89e7b0 P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:50.541296 24475 raft_consensus.cc:1226] T cee02fff2c9c48a19f43eb358a89e7b0 P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Rejecting Update request from peer 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 4. Current term is 5. Ops: []
I20260809 15:46:50.455487 26908 raft_consensus.cc:686] T 0a51c0baaa6e432997c9d3552af4f254 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 LEADER]: Becoming Leader. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Running, Role: LEADER
I20260809 15:46:50.542461 26908 consensus_queue.cc:237] T 0a51c0baaa6e432997c9d3552af4f254 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:50.539088 27000 leader_election.cc:288] T cee02fff2c9c48a19f43eb358a89e7b0 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 5 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
W20260809 15:46:50.553390 23353 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 206970us
I20260809 15:46:50.563244 23529 raft_consensus.cc:1226] T d1a134ba33ea4076bbaac0ae460c17ba P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Rejecting Update request from peer c0f38c1908414566aa9115bb2cbdd85d for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260809 15:46:50.566190 26747 consensus_queue.cc:1046] T d1a134ba33ea4076bbaac0ae460c17ba P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 15:46:50.472937 26855 raft_consensus.cc:482] T 00a1881267a14ba8b78c81040f17a014 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:50.504971 26981 raft_consensus.cc:2781] T 28e17dcb5efe45b8b6a8c42da03b4819 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Leader election won for term 6
I20260809 15:46:50.505896 26399 leader_election.cc:288] T f7715d71dbac447b9351fc5eb36c5e80 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 6 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:50.590886 23283 raft_consensus.cc:2454] T f7715d71dbac447b9351fc5eb36c5e80 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 6.
I20260809 15:46:50.599642 26527 raft_consensus.cc:2781] T 2424fd9bd7f14e508fd106cbfda786a4 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260809 15:46:50.599887 27076 raft_consensus.cc:2741] T 763e58d6ba644eea830a6c1664d39c9b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260809 15:46:50.600193 26927 leader_election.cc:288] T 3d880de657e544468c82157769622a26 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 2 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:50.377952 26730 raft_consensus.cc:504] T 4b30f28311964a0b9149a50495e83d8f P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:46:50.601014 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 258124us
I20260809 15:46:50.660928 23351 leader_election.cc:302] T 07fcabb569414bddbd39df6cedd24e64 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
W20260809 15:46:50.661715 27061 log.cc:924] Time spent T f9e7cadcd24d45fcbfa84209225e8f2d P c0f38c1908414566aa9115bb2cbdd85d: Append to log took a long time: real 0.089s user 0.000s sys 0.000s
I20260809 15:46:50.661932 26730 leader_election.cc:288] T 4b30f28311964a0b9149a50495e83d8f P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:50.663803 26970 leader_election.cc:288] T e71a05ba1d4d42bd95c3f85f9313399e P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:50.518497 26638 raft_consensus.cc:504] T 563daf42d7784db195ea99c3cb21b370 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:50.665724 26638 leader_election.cc:288] T 563daf42d7784db195ea99c3cb21b370 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:50.666637 23520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b30f28311964a0b9149a50495e83d8f" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:50.667551 24452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d880de657e544468c82157769622a26" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f"
I20260809 15:46:50.669277 26659 leader_election.cc:288] T f9e7cadcd24d45fcbfa84209225e8f2d P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:50.670620 23394 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55320 (request call id 1328) took 1173 ms. Trace:
I20260809 15:46:50.670971 23394 rpcz_store.cc:276] 0809 15:46:49.496983 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:49.712474 (+215491us) service_pool.cc:227] Handling call
0809 15:46:49.712768 (+ 294us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 15:46:49.713892 (+ 1124us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:46:49.713919 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 15:46:49.714233 (+ 314us) log.cc:842] Serialized 26 byte log entry
0809 15:46:50.302349 (+588116us) raft_consensus.cc:1631] Marking committed up to 1
0809 15:46:50.302383 (+ 34us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:50.302455 (+ 72us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 15:46:50.670438 (+367983us) raft_consensus.cc:1671] finished
0809 15:46:50.670456 (+ 18us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:50.670566 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":587533,"threads_started":1,"wal-append.queue_time_us":954535}
I20260809 15:46:50.652977 24451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "907bacfa79d04a55a68c8dbe60056462" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:50.653667 27053 consensus_queue.cc:1035] T 0c29937a80724ddb9d07a27534a146f3 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 15:46:50.656845 27055 raft_consensus.cc:2726] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20260809 15:46:50.657308 26993 log.cc:924] Time spent T 472816994d814cb198b69e5248697440 P c0f38c1908414566aa9115bb2cbdd85d: Append to log took a long time: real 0.051s user 0.000s sys 0.000s
I20260809 15:46:50.658557 24448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3aa6a9a52804df0ba9b0ca3c82f38c3" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f"
I20260809 15:46:50.674247 24448 raft_consensus.cc:2379] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d in current term 4: Already voted for candidate 61e69695258f48baa984c9d395e8ce3f in this term.
I20260809 15:46:50.659597 23529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee02fff2c9c48a19f43eb358a89e7b0" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:50.634060 26527 raft_consensus.cc:482] T 2424fd9bd7f14e508fd106cbfda786a4 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Starting leader election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:50.675205 26527 raft_consensus.cc:3037] T 2424fd9bd7f14e508fd106cbfda786a4 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Advancing to term 6
I20260809 15:46:50.676028 26956 leader_election.cc:288] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 4 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:50.528302 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e71a05ba1d4d42bd95c3f85f9313399e" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:50.680965 21289 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20260809 15:46:50.681701 24410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 75549.1us
I20260809 15:46:50.572628 26975 raft_consensus.cc:2741] T d1a134ba33ea4076bbaac0ae460c17ba P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20260809 15:46:50.695812 23286 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53562 (request call id 1700) took 1084 ms. Trace:
W20260809 15:46:50.699932 23472 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 49364.2us
I20260809 15:46:50.677485 26966 leader_election.cc:288] T d0721a4403a440f383318a3cea9d1442 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 6 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:50.697391 23286 rpcz_store.cc:276] 0809 15:46:49.611314 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:49.669116 (+ 57802us) service_pool.cc:227] Handling call
0809 15:46:49.985260 (+316144us) spinlock_profiling.cc:243] Waited 2.39 ms on lock 0x7b5800003398. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e613cf11 00007f34e62c0851 00007f34e62c0702 00007f34ef1b0b09 00007f34ee0ec730 00007f34ee1b279b 00007f34ee1b2821 00007f34ee1762b0 00007f34ee1617f9 00007f34ee1612a2 00007f34e7847684
0809 15:46:49.985430 (+ 170us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 15:46:49.985859 (+ 429us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:46:49.985884 (+ 25us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0809 15:46:50.096204 (+110320us) log.cc:842] Serialized 52 byte log entry
0809 15:46:50.292913 (+196709us) raft_consensus.cc:1631] Marking committed up to 3
0809 15:46:50.327558 (+ 34645us) log.cc:842] Serialized 14 byte log entry
0809 15:46:50.383466 (+ 55908us) log.cc:842] Serialized 14 byte log entry
0809 15:46:50.458054 (+ 74588us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:50.458125 (+ 71us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 15:46:50.695579 (+237454us) raft_consensus.cc:1671] finished
0809 15:46:50.695613 (+ 34us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:50.695757 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":265100,"prepare.queue_time_us":353890,"prepare.run_cpu_time_us":31,"prepare.run_wall_time_us":31,"raft.queue_time_us":2472,"raft.run_cpu_time_us":135,"raft.run_wall_time_us":8270,"spinlock_wait_cycles":5494528,"thread_start_us":196352,"threads_started":1,"wal-append.queue_time_us":557443}
I20260809 15:46:50.679808 26855 raft_consensus.cc:504] T 00a1881267a14ba8b78c81040f17a014 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:50.707719 26855 leader_election.cc:288] T 00a1881267a14ba8b78c81040f17a014 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
W20260809 15:46:50.715740 24406 leader_election.cc:334] T c0741ab44d99439abc3aa3bdd27ee160 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459): Timed out: RequestConsensusVote RPC to 127.20.202.65:35459 timed out after 1.504s (SENT)
I20260809 15:46:50.716176 24406 leader_election.cc:302] T c0741ab44d99439abc3aa3bdd27ee160 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:50.717017 27088 raft_consensus.cc:2726] T c0741ab44d99439abc3aa3bdd27ee160 P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260809 15:46:50.718357 27071 log.cc:924] Time spent T 0c29937a80724ddb9d07a27534a146f3 P 67b3aeadfa7a432abe8af4d292e04ab9: Append to log took a long time: real 0.362s user 0.001s sys 0.000s
I20260809 15:46:50.702975 26981 raft_consensus.cc:686] T 28e17dcb5efe45b8b6a8c42da03b4819 P c0f38c1908414566aa9115bb2cbdd85d [term 6 LEADER]: Becoming Leader. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:46:50.723730 26981 consensus_queue.cc:237] T 28e17dcb5efe45b8b6a8c42da03b4819 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:50.723022 23522 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56766 (request call id 1856) took 1290 ms. Trace:
I20260809 15:46:50.724889 23522 rpcz_store.cc:276] 0809 15:46:49.432947 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:49.433047 (+ 100us) service_pool.cc:227] Handling call
0809 15:46:49.433321 (+ 274us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:50.377035 (+943714us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:50.722797 (+345762us) spinlock_profiling.cc:243] Waited 940 ms on lock 0x7b54000910d4. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e9220f51 00007f34e921f99e 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772 00007f34e764a94c 00007f34e7649d86
0809 15:46:50.722964 (+ 167us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2162479488}
I20260809 15:46:50.707741 23351 leader_election.cc:302] T 2db057fd10e149a9895dff6ef8c9f038 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:50.697615 26405 raft_consensus.cc:2726] T 07fcabb569414bddbd39df6cedd24e64 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260809 15:46:50.736634 26800 consensus_queue.cc:1046] T d0721a4403a440f383318a3cea9d1442 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260809 15:46:50.737222 26772 raft_consensus.cc:2726] T 2db057fd10e149a9895dff6ef8c9f038 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260809 15:46:50.738374 24468 raft_consensus.cc:2396] T e71a05ba1d4d42bd95c3f85f9313399e P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260809 15:46:50.739338 27055 raft_consensus.cc:482] T e71a05ba1d4d42bd95c3f85f9313399e P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
I20260809 15:46:50.739789 27055 raft_consensus.cc:504] T e71a05ba1d4d42bd95c3f85f9313399e P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:50.745280 23522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7baa2492df0340c4a5ac241959ce1ac3" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:50.744055 23472 leader_election.cc:302] T d3e94b7e50754a81b3d72a07476cb7b9 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:50.745900 23522 raft_consensus.cc:3037] T 7baa2492df0340c4a5ac241959ce1ac3 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:46:50.747974 26726 raft_consensus.cc:2726] T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260809 15:46:50.744026 26719 leader_election.cc:288] T 7baa2492df0340c4a5ac241959ce1ac3 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 4 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:50.750051 27028 raft_consensus.cc:482] T 3924b648461e463b9d7925e5671d68ae P 61e69695258f48baa984c9d395e8ce3f [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:50.754582 27028 raft_consensus.cc:504] T 3924b648461e463b9d7925e5671d68ae P 61e69695258f48baa984c9d395e8ce3f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:50.737375 26800 raft_consensus.cc:3032] T d0721a4403a440f383318a3cea9d1442 P c0f38c1908414566aa9115bb2cbdd85d [term 5 LEADER]: Stepping down as leader of term 5
I20260809 15:46:50.749336 26875 leader_election.cc:288] T 6ae14a74a26f434db08c652e097385c8 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:50.755944 26800 raft_consensus.cc:729] T d0721a4403a440f383318a3cea9d1442 P c0f38c1908414566aa9115bb2cbdd85d [term 5 LEADER]: Becoming Follower/Learner. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:46:50.752947 26966 raft_consensus.cc:482] T 30c1740ef6874df0a332fbd6fd4b03ce P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:50.756789 26800 consensus_queue.cc:260] T d0721a4403a440f383318a3cea9d1442 P c0f38c1908414566aa9115bb2cbdd85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:50.757048 26966 raft_consensus.cc:504] T 30c1740ef6874df0a332fbd6fd4b03ce P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:50.760143 24468 raft_consensus.cc:1261] T 16ca412a0dae4feebe85dd01bf1a9173 P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
W20260809 15:46:50.761754 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 54078.6us
I20260809 15:46:50.762446 23351 leader_election.cc:302] T df20671a7b874dbb98c0ef28371c1308 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:50.762838 23472 leader_election.cc:302] T 3924b648461e463b9d7925e5671d68ae P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:50.743060 27055 leader_election.cc:288] T e71a05ba1d4d42bd95c3f85f9313399e P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:50.764125 23351 leader_election.cc:302] T ae6d2c08a7a94f58ac8dd698e3fd456c P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c
W20260809 15:46:50.790521 23470 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 476166us
I20260809 15:46:50.791206 23470 leader_election.cc:302] T e14d118b2eaa42aa8a7cd4025aba7128 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 61e69695258f48baa984c9d395e8ce3f, 67b3aeadfa7a432abe8af4d292e04ab9
W20260809 15:46:50.806630 23276 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.533s user 0.019s sys 0.023s
W20260809 15:46:50.807003 23276 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.540s user 0.019s sys 0.023s
I20260809 15:46:50.808146 23472 leader_election.cc:302] T 00a1881267a14ba8b78c81040f17a014 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters: 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:50.789723 26719 raft_consensus.cc:2726] T df20671a7b874dbb98c0ef28371c1308 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260809 15:46:50.811570 23351 leader_election.cc:302] T 30c1740ef6874df0a332fbd6fd4b03ce P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:50.795519 26726 raft_consensus.cc:2781] T ae6d2c08a7a94f58ac8dd698e3fd456c P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260809 15:46:50.805759 26800 raft_consensus.cc:3037] T d0721a4403a440f383318a3cea9d1442 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Advancing to term 6
I20260809 15:46:50.790719 26707 consensus_queue.cc:1035] T 16ca412a0dae4feebe85dd01bf1a9173 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.029s
I20260809 15:46:50.806797 27034 raft_consensus.cc:2726] T 3924b648461e463b9d7925e5671d68ae P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260809 15:46:50.823087 23522 raft_consensus.cc:2454] T 7baa2492df0340c4a5ac241959ce1ac3 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 4.
I20260809 15:46:50.823779 26926 raft_consensus.cc:2781] T 00a1881267a14ba8b78c81040f17a014 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260809 15:46:50.828810 26668 raft_consensus.cc:3037] T 30c1740ef6874df0a332fbd6fd4b03ce P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Advancing to term 4
W20260809 15:46:50.872391 23399 consensus_meta.cc:287] T 763e58d6ba644eea830a6c1664d39c9b P c0f38c1908414566aa9115bb2cbdd85d: Time spent flushing consensus metadata: real 0.593s user 0.005s sys 0.005s
I20260809 15:46:50.851074 23351 leader_election.cc:302] T 0acc47ce21c3422698d1ef7277c3ed63 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c
I20260809 15:46:50.872797 23399 raft_consensus.cc:2454] T 763e58d6ba644eea830a6c1664d39c9b P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 5.
I20260809 15:46:50.828706 27015 raft_consensus.cc:2726] T e14d118b2eaa42aa8a7cd4025aba7128 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260809 15:46:50.819670 26726 raft_consensus.cc:482] T ae6d2c08a7a94f58ac8dd698e3fd456c P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Starting leader election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:50.829077 27032 raft_consensus.cc:2726] T d3e94b7e50754a81b3d72a07476cb7b9 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260809 15:46:50.829023 23470 leader_election.cc:302] T d1a134ba33ea4076bbaac0ae460c17ba P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:46:50.870081 23353 leader_election.cc:302] T 7baa2492df0340c4a5ac241959ce1ac3 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:46:50.825228 23472 leader_election.cc:302] T 7660dfa2ef694927b8eda4100a2bcd9b P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d
W20260809 15:46:50.879935 23526 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56766 (request call id 1858) took 1352 ms (client timeout 1762 ms). Trace:
W20260809 15:46:50.880223 23526 rpcz_store.cc:269] 0809 15:46:49.527837 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:49.527964 (+ 127us) service_pool.cc:227] Handling call
0809 15:46:50.879754 (+1351790us) spinlock_profiling.cc:243] Waited 633 ms on lock 0x7b540016d854. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e92242ae 00007f34ee161c27 00007f34e7849c04 00007f34e7849b92 00007f34e7849ac1 00007f34e7849a3c 00007f34e7848cf2 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:46:50.879883 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1455295872}
I20260809 15:46:50.873519 23351 leader_election.cc:302] T d2d4f749fcfd4fb3bfb8115af3471598 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c
I20260809 15:46:50.851598 26966 leader_election.cc:288] T 30c1740ef6874df0a332fbd6fd4b03ce P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:50.870676 26926 raft_consensus.cc:482] T 00a1881267a14ba8b78c81040f17a014 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Starting leader election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:46:50.853631 27028 leader_election.cc:288] T 3924b648461e463b9d7925e5671d68ae P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
W20260809 15:46:50.883754 23470 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 54800.4us
I20260809 15:46:50.884253 23353 leader_election.cc:398] T f7715d71dbac447b9351fc5eb36c5e80 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 election: Vote denied by peer 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553) with higher term. Message: Invalid argument: T f7715d71dbac447b9351fc5eb36c5e80 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d for earlier term 5. Current term is 6.
I20260809 15:46:50.884537 23353 leader_election.cc:401] T f7715d71dbac447b9351fc5eb36c5e80 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
I20260809 15:46:50.884873 23470 leader_election.cc:302] T 763e58d6ba644eea830a6c1664d39c9b P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:46:50.885964 23394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e71a05ba1d4d42bd95c3f85f9313399e" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:50.887674 23394 raft_consensus.cc:2379] T e71a05ba1d4d42bd95c3f85f9313399e P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 61e69695258f48baa984c9d395e8ce3f in current term 5: Already voted for candidate c0f38c1908414566aa9115bb2cbdd85d in this term.
I20260809 15:46:50.888896 23404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3924b648461e463b9d7925e5671d68ae" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:50.852480 26611 consensus_queue.cc:1035] T 0da4d5283c8a46878bd39b85e5c71f66 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.026s
I20260809 15:46:50.891042 23522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3924b648461e463b9d7925e5671d68ae" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:50.876645 26726 raft_consensus.cc:3037] T ae6d2c08a7a94f58ac8dd698e3fd456c P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Advancing to term 6
I20260809 15:46:50.880810 26772 raft_consensus.cc:2781] T 0acc47ce21c3422698d1ef7277c3ed63 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260809 15:46:50.896005 25639 raft_consensus.cc:2726] T 7660dfa2ef694927b8eda4100a2bcd9b P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260809 15:46:50.883798 26730 raft_consensus.cc:2781] T 7baa2492df0340c4a5ac241959ce1ac3 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Leader election won for term 4
I20260809 15:46:50.884049 27055 raft_consensus.cc:482] T 7660dfa2ef694927b8eda4100a2bcd9b P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
W20260809 15:46:50.898409 23472 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 73254.3us
I20260809 15:46:50.893815 23522 raft_consensus.cc:2454] T 3924b648461e463b9d7925e5671d68ae P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 7.
I20260809 15:46:50.894446 23351 leader_election.cc:302] T 907bacfa79d04a55a68c8dbe60056462 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:46:50.883623 26926 raft_consensus.cc:3037] T 00a1881267a14ba8b78c81040f17a014 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Advancing to term 6
I20260809 15:46:50.901003 23351 leader_election.cc:302] T d3be1e356b254d22b995be32f1c12f6f P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 61e69695258f48baa984c9d395e8ce3f, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:50.904460 25639 raft_consensus.cc:2781] T 763e58d6ba644eea830a6c1664d39c9b P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Leader election won for term 5
W20260809 15:46:50.961640 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 60711.8us
W20260809 15:46:50.961259 23470 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 52071.8us
I20260809 15:46:50.962527 23351 leader_election.cc:302] T 6ae14a74a26f434db08c652e097385c8 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:50.962160 25639 raft_consensus.cc:686] T 763e58d6ba644eea830a6c1664d39c9b P 65c36d164b0540f099d592f38307ee6c [term 5 LEADER]: Becoming Leader. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:46:50.909161 27055 raft_consensus.cc:504] T 7660dfa2ef694927b8eda4100a2bcd9b P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:50.964977 25639 consensus_queue.cc:237] T 763e58d6ba644eea830a6c1664d39c9b P 65c36d164b0540f099d592f38307ee6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:50.895033 26772 raft_consensus.cc:482] T 0acc47ce21c3422698d1ef7277c3ed63 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Starting leader election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:50.897634 26638 raft_consensus.cc:3037] T f7715d71dbac447b9351fc5eb36c5e80 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Advancing to term 6
W20260809 15:46:50.967541 24410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 62914.5us
I20260809 15:46:50.965328 27055 leader_election.cc:288] T 7660dfa2ef694927b8eda4100a2bcd9b P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:50.968605 24410 leader_election.cc:302] T 3924b648461e463b9d7925e5671d68ae P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c; no voters: c0f38c1908414566aa9115bb2cbdd85d
W20260809 15:46:50.969465 23472 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 70045.4us
I20260809 15:46:50.888479 27015 consensus_queue.cc:1035] T cef3e7277d7d4760824a3436ed4a847d P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.016s
W20260809 15:46:50.905084 27083 log.cc:924] Time spent T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d: Append to log took a long time: real 0.053s user 0.001s sys 0.000s
I20260809 15:46:50.973855 23351 leader_election.cc:302] T ba56725e42f74a1baa4c04d9660727f9 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:50.899405 26981 raft_consensus.cc:2781] T d2d4f749fcfd4fb3bfb8115af3471598 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260809 15:46:50.973023 23522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7660dfa2ef694927b8eda4100a2bcd9b" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:50.908212 26844 raft_consensus.cc:482] T 16ca412a0dae4feebe85dd01bf1a9173 P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:50.980458 26844 raft_consensus.cc:504] T 16ca412a0dae4feebe85dd01bf1a9173 P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:50.973881 26405 raft_consensus.cc:2726] T 6ae14a74a26f434db08c652e097385c8 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260809 15:46:50.960920 26730 raft_consensus.cc:686] T 7baa2492df0340c4a5ac241959ce1ac3 P c0f38c1908414566aa9115bb2cbdd85d [term 4 LEADER]: Becoming Leader. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:46:50.983861 23522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16ca412a0dae4feebe85dd01bf1a9173" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:50.984488 26730 consensus_queue.cc:237] T 7baa2492df0340c4a5ac241959ce1ac3 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:50.975481 27055 raft_consensus.cc:2781] T 3924b648461e463b9d7925e5671d68ae P 61e69695258f48baa984c9d395e8ce3f [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260809 15:46:50.905767 26757 raft_consensus.cc:2781] T 907bacfa79d04a55a68c8dbe60056462 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260809 15:46:50.976651 23351 leader_election.cc:302] T 998c6cee72cf40228c6dba064c0db2da P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:50.978560 27084 raft_consensus.cc:3037] T d3be1e356b254d22b995be32f1c12f6f P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Advancing to term 5
W20260809 15:46:50.978729 23392 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55320 (request call id 1330) took 1392 ms (client timeout 1818 ms). Trace:
I20260809 15:46:50.979290 26740 raft_consensus.cc:3032] T d1a134ba33ea4076bbaac0ae460c17ba P c0f38c1908414566aa9115bb2cbdd85d [term 3 LEADER]: Stepping down as leader of term 3
I20260809 15:46:50.904891 27078 raft_consensus.cc:2781] T d1a134ba33ea4076bbaac0ae460c17ba P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Leader election won for term 4
I20260809 15:46:51.004999 27055 raft_consensus.cc:482] T 3924b648461e463b9d7925e5671d68ae P 61e69695258f48baa984c9d395e8ce3f [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:46:50.960201 27034 consensus_queue.cc:1046] T 953eea268e17491ca76f3a555860a512 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.027s
I20260809 15:46:50.989076 23470 leader_election.cc:302] T f7715d71dbac447b9351fc5eb36c5e80 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
W20260809 15:46:51.538067 23470 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 549021us
I20260809 15:46:51.539402 27073 raft_consensus.cc:2781] T f7715d71dbac447b9351fc5eb36c5e80 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Leader election won for term 6
I20260809 15:46:51.539919 27073 raft_consensus.cc:686] T f7715d71dbac447b9351fc5eb36c5e80 P 65c36d164b0540f099d592f38307ee6c [term 6 LEADER]: Becoming Leader. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:46:51.540781 27073 consensus_queue.cc:237] T f7715d71dbac447b9351fc5eb36c5e80 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:51.541395 26996 raft_consensus.cc:482] T e14d118b2eaa42aa8a7cd4025aba7128 P 61e69695258f48baa984c9d395e8ce3f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
I20260809 15:46:51.005631 26757 raft_consensus.cc:482] T 907bacfa79d04a55a68c8dbe60056462 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Starting leader election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
I20260809 15:46:51.541960 26996 raft_consensus.cc:504] T e14d118b2eaa42aa8a7cd4025aba7128 P 61e69695258f48baa984c9d395e8ce3f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
W20260809 15:46:51.543717 26926 consensus_meta.cc:287] T 00a1881267a14ba8b78c81040f17a014 P 65c36d164b0540f099d592f38307ee6c: Time spent flushing consensus metadata: real 0.572s user 0.002s sys 0.007s
I20260809 15:46:51.544044 26996 leader_election.cc:288] T e14d118b2eaa42aa8a7cd4025aba7128 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
W20260809 15:46:51.217684 27109 log.cc:924] Time spent T 28e17dcb5efe45b8b6a8c42da03b4819 P c0f38c1908414566aa9115bb2cbdd85d: Append to log took a long time: real 0.088s user 0.001s sys 0.000s
I20260809 15:46:50.970911 26772 raft_consensus.cc:3037] T 0acc47ce21c3422698d1ef7277c3ed63 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Advancing to term 6
W20260809 15:46:51.547582 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 570969us
I20260809 15:46:51.548730 23351 leader_election.cc:302] T a88421dba80d4d3388aa5eb62e23853b P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
W20260809 15:46:51.187207 24406 leader_election.cc:334] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459): Timed out: RequestConsensusVote RPC to 127.20.202.65:35459 timed out after 1.583s (SENT)
I20260809 15:46:51.550182 24406 leader_election.cc:302] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
W20260809 15:46:51.139417 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 666134us
W20260809 15:46:51.139618 23223 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.68:34689, user_credentials={real_user=slave}} blocked reactor thread for 576735us
I20260809 15:46:51.091643 24475 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38722 (request call id 443) took 1514 ms. Trace:
W20260809 15:46:51.019088 23472 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 48929.7us
W20260809 15:46:51.178336 23468 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.68:34689, user_credentials={real_user=slave}} blocked reactor thread for 47708.7us
I20260809 15:46:51.554414 23472 leader_election.cc:302] T 7baa2492df0340c4a5ac241959ce1ac3 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:50.987684 26747 raft_consensus.cc:2726] T ba56725e42f74a1baa4c04d9660727f9 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260809 15:46:51.551820 23395 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52856 (request call id 212) took 1431 ms. Trace:
I20260809 15:46:51.554008 24475 rpcz_store.cc:276] 0809 15:46:49.577258 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:49.586020 (+ 8762us) service_pool.cc:227] Handling call
0809 15:46:49.996497 (+410477us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:50.541687 (+545190us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:51.014016 (+472329us) spinlock_profiling.cc:243] Waited 545 ms on lock 0x7b540047ead4. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e9220f51 00007f34e921f99e 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772 00007f34e764a94c 00007f34e7649d86
0809 15:46:51.091475 (+ 77459us) spinlock_profiling.cc:243] Waited 398 ms on lock 0x7b540047ead0. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e921fa09 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:46:51.091590 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2168041600}
W20260809 15:46:51.374403 27063 log.cc:924] Time spent T f9e7cadcd24d45fcbfa84209225e8f2d P 65c36d164b0540f099d592f38307ee6c: Append to log took a long time: real 0.855s user 0.006s sys 0.007s
I20260809 15:46:51.570319 23274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30c1740ef6874df0a332fbd6fd4b03ce" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
W20260809 15:46:51.545710 26925 consensus_meta.cc:287] T cee02fff2c9c48a19f43eb358a89e7b0 P 65c36d164b0540f099d592f38307ee6c: Time spent flushing consensus metadata: real 1.009s user 0.004s sys 0.004s
I20260809 15:46:51.563597 23273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d880de657e544468c82157769622a26" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:51.566123 23280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ced718c1a5c4da79cb8c1ebd1435b87" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:51.573972 23280 raft_consensus.cc:3037] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:46:51.567114 23518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e14d118b2eaa42aa8a7cd4025aba7128" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:51.217528 23393 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55320 (request call id 1316) took 1828 ms. Trace:
I20260809 15:46:51.555663 27055 raft_consensus.cc:3037] T 3924b648461e463b9d7925e5671d68ae P 61e69695258f48baa984c9d395e8ce3f [term 7 FOLLOWER]: Advancing to term 8
I20260809 15:46:51.196794 27076 raft_consensus.cc:2741] T 99bdb93503db4de4a1cfb0161ea0c07c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20260809 15:46:51.579381 26925 raft_consensus.cc:504] T cee02fff2c9c48a19f43eb358a89e7b0 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:51.581431 26925 leader_election.cc:288] T cee02fff2c9c48a19f43eb358a89e7b0 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 5 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:51.582145 23529 raft_consensus.cc:2379] T cee02fff2c9c48a19f43eb358a89e7b0 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 61e69695258f48baa984c9d395e8ce3f in current term 5: Already voted for candidate 65c36d164b0540f099d592f38307ee6c in this term.
W20260809 15:46:51.585172 27084 consensus_meta.cc:287] T d3be1e356b254d22b995be32f1c12f6f P c0f38c1908414566aa9115bb2cbdd85d: Time spent flushing consensus metadata: real 0.513s user 0.007s sys 0.000s
I20260809 15:46:51.586714 27084 raft_consensus.cc:2726] T d3be1e356b254d22b995be32f1c12f6f P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260809 15:46:51.129750 26405 raft_consensus.cc:2726] T 998c6cee72cf40228c6dba064c0db2da P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260809 15:46:51.569320 23270 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "07fcabb569414bddbd39df6cedd24e64"
dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
mode: GRACEFUL
new_leader_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
from {username='slave'} at 127.0.0.1:33468
I20260809 15:46:51.587423 27084 raft_consensus.cc:2726] T a88421dba80d4d3388aa5eb62e23853b P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260809 15:46:51.587769 23270 raft_consensus.cc:595] T 07fcabb569414bddbd39df6cedd24e64 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Received request to transfer leadership to TS c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:51.588121 23270 raft_consensus.cc:603] T 07fcabb569414bddbd39df6cedd24e64 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260809 15:46:51.588399 23280 raft_consensus.cc:2454] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 4.
I20260809 15:46:51.568454 23228 leader_election.cc:302] T 28e17dcb5efe45b8b6a8c42da03b4819 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:51.590912 27015 raft_consensus.cc:3032] T 953eea268e17491ca76f3a555860a512 P 65c36d164b0540f099d592f38307ee6c [term 4 LEADER]: Stepping down as leader of term 4
I20260809 15:46:51.591364 27015 raft_consensus.cc:729] T 953eea268e17491ca76f3a555860a512 P 65c36d164b0540f099d592f38307ee6c [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:46:51.592193 27015 consensus_queue.cc:260] T 953eea268e17491ca76f3a555860a512 P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:51.593190 27015 raft_consensus.cc:3037] T 953eea268e17491ca76f3a555860a512 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Advancing to term 5
I20260809 15:46:51.556826 26757 raft_consensus.cc:3037] T 907bacfa79d04a55a68c8dbe60056462 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Advancing to term 5
I20260809 15:46:51.568755 23286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0721a4403a440f383318a3cea9d1442" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
W20260809 15:46:51.066709 27070 log.cc:924] Time spent T 2a823d35f4f348d1b52ba9e6d80b00d2 P 65c36d164b0540f099d592f38307ee6c: Append to log took a long time: real 0.053s user 0.000s sys 0.001s
W20260809 15:46:51.184262 27107 log.cc:924] Time spent T 297c82ed906d4b48b184985eed3de601 P 67b3aeadfa7a432abe8af4d292e04ab9: Append to log took a long time: real 0.218s user 0.000s sys 0.000s
I20260809 15:46:51.601029 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16ca412a0dae4feebe85dd01bf1a9173" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
W20260809 15:46:51.303755 27057 log.cc:924] Time spent T 00000000000000000000000000000000 P a86a2513b8024a869d80303dcbf44163: Append to log took a long time: real 1.020s user 0.000s sys 0.012s
W20260809 15:46:51.602948 23472 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 48618.2us
W20260809 15:46:51.155056 27105 log.cc:924] Time spent T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 65c36d164b0540f099d592f38307ee6c: Append to log took a long time: real 0.082s user 0.000s sys 0.000s
I20260809 15:46:51.561430 23395 rpcz_store.cc:276] 0809 15:46:50.120001 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:50.309770 (+189769us) service_pool.cc:227] Handling call
0809 15:46:50.310075 (+ 305us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 15:46:50.314989 (+ 4914us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:46:50.315017 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 15:46:50.315428 (+ 411us) log.cc:842] Serialized 26 byte log entry
0809 15:46:50.726763 (+411335us) raft_consensus.cc:1631] Marking committed up to 4
0809 15:46:50.726799 (+ 36us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:50.726858 (+ 59us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 15:46:51.129127 (+402269us) raft_consensus.cc:1671] finished
0809 15:46:51.129146 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:51.551765 (+422619us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4554,"raft.queue_time_us":5226,"raft.run_cpu_time_us":108,"raft.run_wall_time_us":18780,"thread_start_us":261185,"threads_started":1,"wal-append.queue_time_us":452694}
I20260809 15:46:51.571838 23276 raft_consensus.cc:1261] T 05ad7d8fdca64e3da527280ca5be9280 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260809 15:46:51.571686 23274 raft_consensus.cc:2454] T 30c1740ef6874df0a332fbd6fd4b03ce P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 4.
I20260809 15:46:50.975919 26981 raft_consensus.cc:482] T d2d4f749fcfd4fb3bfb8115af3471598 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Starting leader election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:51.355903 26638 raft_consensus.cc:2726] T f7715d71dbac447b9351fc5eb36c5e80 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553) with higher term. Message: Invalid argument: T f7715d71dbac447b9351fc5eb36c5e80 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d for earlier term 5. Current term is 6.
W20260809 15:46:51.574525 24406 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 387325us
I20260809 15:46:51.574596 23518 raft_consensus.cc:2379] T e14d118b2eaa42aa8a7cd4025aba7128 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 61e69695258f48baa984c9d395e8ce3f in current term 7: Already voted for candidate 65c36d164b0540f099d592f38307ee6c in this term.
I20260809 15:46:50.988891 27032 consensus_queue.cc:1046] T f9e7cadcd24d45fcbfa84209225e8f2d P 65c36d164b0540f099d592f38307ee6c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 2, Time since last communication: 0.606s
I20260809 15:46:51.004917 27053 consensus_queue.cc:1035] T 16ca412a0dae4feebe85dd01bf1a9173 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.591s
I20260809 15:46:51.555927 26975 raft_consensus.cc:482] T c5c2e425a74b4e64ae70b5def750434c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:46:51.560228 27096 raft_consensus.cc:482] T 73f6f44c259547f1b05a3a434616fa54 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:46:51.564262 27075 raft_consensus.cc:482] T 3296ef50be254fc29cfc77d22a8f7a23 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
I20260809 15:46:51.544232 26926 raft_consensus.cc:504] T 00a1881267a14ba8b78c81040f17a014 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:51.568159 24448 raft_consensus.cc:1261] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 5. (term mismatch)
W20260809 15:46:51.418776 26527 consensus_meta.cc:287] T 2424fd9bd7f14e508fd106cbfda786a4 P c0f38c1908414566aa9115bb2cbdd85d: Time spent flushing consensus metadata: real 0.743s user 0.005s sys 0.002s
W20260809 15:46:51.014443 23392 rpcz_store.cc:269] 0809 15:46:49.586541 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:49.616841 (+ 30300us) service_pool.cc:227] Handling call
0809 15:46:50.978564 (+1361723us) spinlock_profiling.cc:243] Waited 165 ms on lock 0x7b540009fbd4. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e92242ae 00007f34ee161c27 00007f34e7849c04 00007f34e7849b92 00007f34e7849ac1 00007f34e7849a3c 00007f34e7848cf2 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:46:50.978669 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":378785664}
I20260809 15:46:51.661063 23521 raft_consensus.cc:1226] T c5c2e425a74b4e64ae70b5def750434c P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Rejecting Update request from peer c0f38c1908414566aa9115bb2cbdd85d for earlier term 4. Current term is 5. Ops: []
I20260809 15:46:51.014536 26740 raft_consensus.cc:729] T d1a134ba33ea4076bbaac0ae460c17ba P c0f38c1908414566aa9115bb2cbdd85d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:46:51.595357 23286 raft_consensus.cc:3037] T d0721a4403a440f383318a3cea9d1442 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Advancing to term 6
I20260809 15:46:50.991444 26844 leader_election.cc:288] T 16ca412a0dae4feebe85dd01bf1a9173 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:51.227883 26668 raft_consensus.cc:2726] T 30c1740ef6874df0a332fbd6fd4b03ce P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260809 15:46:51.598503 26908 raft_consensus.cc:2726] T 28e17dcb5efe45b8b6a8c42da03b4819 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260809 15:46:51.601053 23522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998c6cee72cf40228c6dba064c0db2da" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 8 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:51.578148 23393 rpcz_store.cc:276] 0809 15:46:49.389189 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:49.439703 (+ 50514us) service_pool.cc:227] Handling call
0809 15:46:49.446183 (+ 6480us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 15:46:49.616901 (+170718us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:46:49.618064 (+ 1163us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0809 15:46:49.672553 (+ 54489us) log.cc:842] Serialized 52 byte log entry
0809 15:46:50.292773 (+620220us) raft_consensus.cc:1631] Marking committed up to 2
0809 15:46:50.298584 (+ 5811us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:51.058643 (+760059us) spinlock_profiling.cc:243] Waited 138 ms on lock 0x7b54000a91d4. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e9220f51 00007f34e921f99e 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772 00007f34e764a94c 00007f34e7649d86
0809 15:46:51.072065 (+ 13422us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 15:46:51.092922 (+ 20857us) raft_consensus.cc:1671] finished
0809 15:46:51.092941 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:51.197110 (+104169us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":3738,"raft.run_cpu_time_us":86,"raft.run_wall_time_us":1249,"spinlock_wait_cycles":318161280,"thread_start_us":501121,"threads_started":1,"wal-append.queue_time_us":614667}
W20260809 15:46:51.100641 27099 log.cc:924] Time spent T 2a823d35f4f348d1b52ba9e6d80b00d2 P 61e69695258f48baa984c9d395e8ce3f: Append to log took a long time: real 0.124s user 0.000s sys 0.000s
I20260809 15:46:51.093329 27078 raft_consensus.cc:686] T d1a134ba33ea4076bbaac0ae460c17ba P 65c36d164b0540f099d592f38307ee6c [term 4 LEADER]: Becoming Leader. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
W20260809 15:46:51.621634 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 72968.1us
I20260809 15:46:51.671929 27078 consensus_queue.cc:237] T d1a134ba33ea4076bbaac0ae460c17ba P 65c36d164b0540f099d592f38307ee6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:51.622807 25639 raft_consensus.cc:2726] T 7baa2492df0340c4a5ac241959ce1ac3 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260809 15:46:51.676343 26638 raft_consensus.cc:482] T 0da4d5283c8a46878bd39b85e5c71f66 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:51.676532 27118 raft_consensus.cc:482] T 297c82ed906d4b48b184985eed3de601 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:51.676820 26638 raft_consensus.cc:504] T 0da4d5283c8a46878bd39b85e5c71f66 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:51.677057 27118 raft_consensus.cc:504] T 297c82ed906d4b48b184985eed3de601 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:51.678663 26638 leader_election.cc:288] T 0da4d5283c8a46878bd39b85e5c71f66 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:51.625782 23529 raft_consensus.cc:1226] T e14d118b2eaa42aa8a7cd4025aba7128 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Rejecting Update request from peer 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 6. Current term is 7. Ops: []
I20260809 15:46:51.025116 26707 consensus_queue.cc:1046] T 6ae14a74a26f434db08c652e097385c8 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.593s
I20260809 15:46:51.571738 26826 raft_consensus.cc:2726] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260809 15:46:51.679054 27118 leader_election.cc:288] T 297c82ed906d4b48b184985eed3de601 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:51.571957 26747 raft_consensus.cc:482] T cef3e7277d7d4760824a3436ed4a847d P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:51.680161 26668 raft_consensus.cc:482] T 0c29937a80724ddb9d07a27534a146f3 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:51.679965 26747 raft_consensus.cc:504] T cef3e7277d7d4760824a3436ed4a847d P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:51.680651 26668 raft_consensus.cc:504] T 0c29937a80724ddb9d07a27534a146f3 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:51.681650 23273 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0c48a3049897452383257aa757834fb5"
dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
mode: GRACEFUL
new_leader_uuid: "65c36d164b0540f099d592f38307ee6c"
from {username='slave'} at 127.0.0.1:33468
I20260809 15:46:51.681917 26747 leader_election.cc:288] T cef3e7277d7d4760824a3436ed4a847d P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:51.682431 26747 raft_consensus.cc:482] T 0a51c0baaa6e432997c9d3552af4f254 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:51.660902 24448 pending_rounds.cc:76] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 61e69695258f48baa984c9d395e8ce3f: Aborting all ops after (but not including) 4
I20260809 15:46:51.682876 26747 raft_consensus.cc:504] T 0a51c0baaa6e432997c9d3552af4f254 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
W20260809 15:46:51.627954 24406 leader_election.cc:334] T 294d4bcc1e7a477cb56455581277a60f P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459): Timed out: RequestConsensusVote RPC to 127.20.202.65:35459 timed out after 1.637s (SENT)
I20260809 15:46:51.070456 26726 raft_consensus.cc:504] T ae6d2c08a7a94f58ac8dd698e3fd456c P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:51.625977 26981 raft_consensus.cc:3037] T d2d4f749fcfd4fb3bfb8115af3471598 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Advancing to term 5
I20260809 15:46:51.667361 26527 raft_consensus.cc:504] T 2424fd9bd7f14e508fd106cbfda786a4 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:51.685027 26726 leader_election.cc:288] T ae6d2c08a7a94f58ac8dd698e3fd456c P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 6 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:51.661249 27075 raft_consensus.cc:504] T 3296ef50be254fc29cfc77d22a8f7a23 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:51.682780 26926 leader_election.cc:288] T 00a1881267a14ba8b78c81040f17a014 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 6 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:51.686187 26707 raft_consensus.cc:3032] T 6ae14a74a26f434db08c652e097385c8 P 65c36d164b0540f099d592f38307ee6c [term 4 LEADER]: Stepping down as leader of term 4
I20260809 15:46:51.686532 26707 raft_consensus.cc:729] T 6ae14a74a26f434db08c652e097385c8 P 65c36d164b0540f099d592f38307ee6c [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:46:51.687057 27075 leader_election.cc:288] T 3296ef50be254fc29cfc77d22a8f7a23 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:51.687280 26707 consensus_queue.cc:260] T 6ae14a74a26f434db08c652e097385c8 P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:51.688261 24406 leader_election.cc:302] T 294d4bcc1e7a477cb56455581277a60f P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:51.661989 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a88421dba80d4d3388aa5eb62e23853b" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:51.654187 26975 raft_consensus.cc:504] T c5c2e425a74b4e64ae70b5def750434c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:51.658733 27076 consensus_queue.cc:1046] T cee02fff2c9c48a19f43eb358a89e7b0 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.7, Next index: 8, Last known committed idx: 5, Time since last communication: 0.006s
W20260809 15:46:51.689141 24406 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 61228.1us
I20260809 15:46:51.683382 24448 pending_rounds.cc:97] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 61e69695258f48baa984c9d395e8ce3f: Aborting uncommitted NO_OP operation due to leader change: 2.5
I20260809 15:46:51.685443 26527 leader_election.cc:288] T 2424fd9bd7f14e508fd106cbfda786a4 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 6 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:51.663028 23518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba56725e42f74a1baa4c04d9660727f9" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:51.691941 23273 raft_consensus.cc:595] T 0c48a3049897452383257aa757834fb5 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Received request to transfer leadership to TS 65c36d164b0540f099d592f38307ee6c
I20260809 15:46:51.702318 23273 raft_consensus.cc:603] T 0c48a3049897452383257aa757834fb5 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260809 15:46:51.702553 23526 raft_consensus.cc:1261] T 28e17dcb5efe45b8b6a8c42da03b4819 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260809 15:46:51.694020 23286 raft_consensus.cc:2454] T d0721a4403a440f383318a3cea9d1442 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 6.
I20260809 15:46:51.704468 23522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0da4d5283c8a46878bd39b85e5c71f66" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:51.706773 23518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07fcabb569414bddbd39df6cedd24e64" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:51.710007 26707 raft_consensus.cc:3037] T 6ae14a74a26f434db08c652e097385c8 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Advancing to term 5
I20260809 15:46:51.711844 26740 consensus_queue.cc:260] T d1a134ba33ea4076bbaac0ae460c17ba P c0f38c1908414566aa9115bb2cbdd85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:51.712890 26740 raft_consensus.cc:3037] T d1a134ba33ea4076bbaac0ae460c17ba P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:46:51.713709 24471 raft_consensus.cc:1261] T 0a51c0baaa6e432997c9d3552af4f254 P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Refusing update from remote peer 67b3aeadfa7a432abe8af4d292e04ab9: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20260809 15:46:51.628669 23472 leader_election.cc:302] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:51.699514 27097 raft_consensus.cc:2726] T 294d4bcc1e7a477cb56455581277a60f P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260809 15:46:51.720073 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "297c82ed906d4b48b184985eed3de601" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:51.714871 26747 leader_election.cc:288] T 0a51c0baaa6e432997c9d3552af4f254 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:51.685570 26726 raft_consensus.cc:482] T 953eea268e17491ca76f3a555860a512 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:51.682515 26668 leader_election.cc:288] T 0c29937a80724ddb9d07a27534a146f3 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:51.727969 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c29937a80724ddb9d07a27534a146f3" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:51.685053 26925 raft_consensus.cc:3032] T f9e7cadcd24d45fcbfa84209225e8f2d P 65c36d164b0540f099d592f38307ee6c [term 4 LEADER]: Stepping down as leader of term 4
I20260809 15:46:51.689610 27076 raft_consensus.cc:3032] T cee02fff2c9c48a19f43eb358a89e7b0 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 LEADER]: Stepping down as leader of term 4
I20260809 15:46:51.729946 23529 raft_consensus.cc:1226] T d1a134ba33ea4076bbaac0ae460c17ba P 65c36d164b0540f099d592f38307ee6c [term 4 LEADER]: Rejecting Update request from peer c0f38c1908414566aa9115bb2cbdd85d for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260809 15:46:51.714798 24451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3296ef50be254fc29cfc77d22a8f7a23" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:51.700095 24448 raft_consensus.cc:2866] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 15:46:51.705179 27118 raft_consensus.cc:482] T 538a228ddfb54bc6a36528d5ff07597b P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:51.737607 23520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2424fd9bd7f14e508fd106cbfda786a4" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:51.738785 23518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cef3e7277d7d4760824a3436ed4a847d" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:51.721354 27081 raft_consensus.cc:2726] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260809 15:46:51.739605 23472 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 111001us
I20260809 15:46:51.746515 23522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3622f6e3e3641cea1960e3a31498bb7" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:51.747057 23472 leader_election.cc:302] T 99bdb93503db4de4a1cfb0161ea0c07c P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:51.745059 23521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae6d2c08a7a94f58ac8dd698e3fd456c" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:51.723603 24452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee02fff2c9c48a19f43eb358a89e7b0" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f"
I20260809 15:46:51.660226 27096 raft_consensus.cc:504] T 73f6f44c259547f1b05a3a434616fa54 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:51.700767 27055 raft_consensus.cc:504] T 3924b648461e463b9d7925e5671d68ae P 61e69695258f48baa984c9d395e8ce3f [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:51.757676 23529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0741ab44d99439abc3aa3bdd27ee160" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
W20260809 15:46:51.740590 23228 connection.cc:232] RPC call timeout handler was delayed by -4.6955887379590422s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1622s for a response
W20260809 15:46:51.762758 23228 connection.cc:232] RPC call timeout handler was delayed by -4.49906390090473s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1569s for a response
I20260809 15:46:51.763114 23282 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c5c2e425a74b4e64ae70b5def750434c"
dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
mode: GRACEFUL
new_leader_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
from {username='slave'} at 127.0.0.1:33468
I20260809 15:46:51.763718 23282 raft_consensus.cc:595] T c5c2e425a74b4e64ae70b5def750434c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Received request to transfer leadership to TS c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:51.764062 23282 raft_consensus.cc:603] T c5c2e425a74b4e64ae70b5def750434c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260809 15:46:51.731026 26925 raft_consensus.cc:729] T f9e7cadcd24d45fcbfa84209225e8f2d P 65c36d164b0540f099d592f38307ee6c [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:46:51.762871 23285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee02fff2c9c48a19f43eb358a89e7b0" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:51.734793 27076 raft_consensus.cc:729] T cee02fff2c9c48a19f43eb358a89e7b0 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Running, Role: LEADER
W20260809 15:46:51.763088 23228 connection.cc:232] RPC call timeout handler was delayed by -4.41748873796314s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19310000000000002s for a response
I20260809 15:46:51.734709 26908 raft_consensus.cc:482] T 4b30f28311964a0b9149a50495e83d8f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:51.769670 23472 leader_election.cc:302] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters: 61e69695258f48baa984c9d395e8ce3f
I20260809 15:46:51.722946 26810 consensus_queue.cc:1035] T 0a51c0baaa6e432997c9d3552af4f254 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.029s
W20260809 15:46:51.772500 23228 connection.cc:232] RPC call timeout handler was delayed by -4.3921098781283945s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16790000000000002s for a response
W20260809 15:46:51.775194 23228 connection.cc:232] RPC call timeout handler was delayed by -4.3201174561399966s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1932s for a response
W20260809 15:46:51.775506 23228 connection.cc:232] RPC call timeout handler was delayed by -4.2998401040676981s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1708s for a response
W20260809 15:46:51.775704 23228 connection.cc:232] RPC call timeout handler was delayed by -4.2679087780416012s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1864s for a response
W20260809 15:46:51.775903 23228 connection.cc:232] RPC call timeout handler was delayed by -4.2369827050715685s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2074s for a response
W20260809 15:46:51.776183 23228 connection.cc:232] RPC call timeout handler was delayed by -4.0716166850179434s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2138s for a response
W20260809 15:46:51.776440 23228 connection.cc:232] RPC call timeout handler was delayed by -3.8821590740699321s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2163s for a response
W20260809 15:46:51.776688 23228 connection.cc:232] RPC call timeout handler was delayed by -3.6689108130522072s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.24060000000000004s for a response
W20260809 15:46:51.776937 23228 connection.cc:232] RPC call timeout handler was delayed by -3.1060087780933827s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3155s for a response
W20260809 15:46:51.777182 23228 connection.cc:232] RPC call timeout handler was delayed by -2.14909252920188s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.43630000000000008s for a response
I20260809 15:46:51.790325 25639 raft_consensus.cc:2781] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260809 15:46:51.790678 25639 raft_consensus.cc:482] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Starting leader election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
I20260809 15:46:51.790977 25639 raft_consensus.cc:3037] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Advancing to term 7
I20260809 15:46:51.755518 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e71a05ba1d4d42bd95c3f85f9313399e" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:51.759929 25863 raft_consensus.cc:2726] T 99bdb93503db4de4a1cfb0161ea0c07c P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260809 15:46:51.739965 26726 raft_consensus.cc:504] T 953eea268e17491ca76f3a555860a512 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:51.793727 23282 raft_consensus.cc:1261] T 763e58d6ba644eea830a6c1664d39c9b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260809 15:46:51.751966 26527 consensus_queue.cc:1046] T c5c2e425a74b4e64ae70b5def750434c P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.038s
I20260809 15:46:51.749961 27118 raft_consensus.cc:504] T 538a228ddfb54bc6a36528d5ff07597b P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:51.793466 24452 raft_consensus.cc:2379] T cee02fff2c9c48a19f43eb358a89e7b0 P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c in current term 5: Already voted for candidate 61e69695258f48baa984c9d395e8ce3f in this term.
I20260809 15:46:51.795825 23529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2d4f749fcfd4fb3bfb8115af3471598" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:51.808820 23282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00a1881267a14ba8b78c81040f17a014" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:51.809494 23522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ced718c1a5c4da79cb8c1ebd1435b87" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:51.809464 23282 raft_consensus.cc:2379] T 00a1881267a14ba8b78c81040f17a014 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c in current term 6: Already voted for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in this term.
I20260809 15:46:51.764907 27136 consensus_queue.cc:1035] T 28e17dcb5efe45b8b6a8c42da03b4819 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.046s
I20260809 15:46:51.813001 23522 raft_consensus.cc:2379] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 in current term 4: Already voted for candidate 65c36d164b0540f099d592f38307ee6c in this term.
I20260809 15:46:51.819481 23277 raft_consensus.cc:1226] T d1a134ba33ea4076bbaac0ae460c17ba P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Rejecting Update request from peer c0f38c1908414566aa9115bb2cbdd85d for earlier term 3. Current term is 4. Ops: []
I20260809 15:46:51.801626 26527 raft_consensus.cc:3032] T c5c2e425a74b4e64ae70b5def750434c P c0f38c1908414566aa9115bb2cbdd85d [term 4 LEADER]: Stepping down as leader of term 4
I20260809 15:46:51.821355 23283 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f7715d71dbac447b9351fc5eb36c5e80"
dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
mode: GRACEFUL
new_leader_uuid: "65c36d164b0540f099d592f38307ee6c"
from {username='slave'} at 127.0.0.1:33468
I20260809 15:46:51.821961 23283 raft_consensus.cc:595] T f7715d71dbac447b9351fc5eb36c5e80 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Received request to transfer leadership to TS 65c36d164b0540f099d592f38307ee6c
I20260809 15:46:51.822292 23283 raft_consensus.cc:603] T f7715d71dbac447b9351fc5eb36c5e80 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260809 15:46:51.789321 26908 raft_consensus.cc:504] T 4b30f28311964a0b9149a50495e83d8f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:51.825886 23285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9e7cadcd24d45fcbfa84209225e8f2d" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:51.796227 24468 raft_consensus.cc:2454] T e71a05ba1d4d42bd95c3f85f9313399e P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 4.
I20260809 15:46:51.826972 23285 raft_consensus.cc:2379] T f9e7cadcd24d45fcbfa84209225e8f2d P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d in current term 5: Already voted for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in this term.
I20260809 15:46:51.834916 23285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae6d2c08a7a94f58ac8dd698e3fd456c" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:51.835917 23285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2424fd9bd7f14e508fd106cbfda786a4" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:51.827096 23282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05ad7d8fdca64e3da527280ca5be9280" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:51.826120 23277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cef3e7277d7d4760824a3436ed4a847d" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:51.834213 23470 leader_election.cc:302] T 3d880de657e544468c82157769622a26 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 61e69695258f48baa984c9d395e8ce3f, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:51.805002 26975 leader_election.cc:288] T c5c2e425a74b4e64ae70b5def750434c P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:51.829159 23522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7715d71dbac447b9351fc5eb36c5e80" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:51.826203 23472 leader_election.cc:302] T 73f6f44c259547f1b05a3a434616fa54 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:51.836810 25639 raft_consensus.cc:504] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:51.839198 23282 raft_consensus.cc:2354] T 05ad7d8fdca64e3da527280ca5be9280 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d for earlier term 4. Current term is 5.
I20260809 15:46:51.839897 23283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4564dfc6d4094c1b9240b367ccecf3b1" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:51.841564 27134 raft_consensus.cc:482] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
I20260809 15:46:51.841883 23470 leader_election.cc:302] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:46:51.842628 27034 raft_consensus.cc:2726] T 3d880de657e544468c82157769622a26 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260809 15:46:51.842985 23470 leader_election.cc:302] T 30c1740ef6874df0a332fbd6fd4b03ce P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:46:51.843955 26966 raft_consensus.cc:2781] T 30c1740ef6874df0a332fbd6fd4b03ce P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260809 15:46:51.844303 26966 raft_consensus.cc:482] T 30c1740ef6874df0a332fbd6fd4b03ce P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:46:51.844616 26966 raft_consensus.cc:3037] T 30c1740ef6874df0a332fbd6fd4b03ce P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Advancing to term 5
I20260809 15:46:51.848801 23472 leader_election.cc:302] T c5c2e425a74b4e64ae70b5def750434c P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:51.849807 23351 leader_election.cc:302] T f9e7cadcd24d45fcbfa84209225e8f2d P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:51.850940 23351 leader_election.cc:302] T ae6d2c08a7a94f58ac8dd698e3fd456c P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:51.851986 23351 leader_election.cc:398] T 05ad7d8fdca64e3da527280ca5be9280 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 4 pre-election: Vote denied by peer 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459) with higher term. Message: Invalid argument: T 05ad7d8fdca64e3da527280ca5be9280 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d for earlier term 4. Current term is 5.
I20260809 15:46:51.852201 23351 leader_election.cc:401] T 05ad7d8fdca64e3da527280ca5be9280 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
I20260809 15:46:51.854810 23351 leader_election.cc:302] T cef3e7277d7d4760824a3436ed4a847d P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:51.842550 23277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e17dcb5efe45b8b6a8c42da03b4819" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:51.855839 23277 raft_consensus.cc:3037] T 28e17dcb5efe45b8b6a8c42da03b4819 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Advancing to term 6
I20260809 15:46:51.856920 23351 leader_election.cc:302] T 4564dfc6d4094c1b9240b367ccecf3b1 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:51.858032 23351 leader_election.cc:302] T 2424fd9bd7f14e508fd106cbfda786a4 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:51.861577 25639 leader_election.cc:288] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 7 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:51.862666 25639 raft_consensus.cc:2726] T c5c2e425a74b4e64ae70b5def750434c P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260809 15:46:51.824815 26527 raft_consensus.cc:729] T c5c2e425a74b4e64ae70b5def750434c P c0f38c1908414566aa9115bb2cbdd85d [term 4 LEADER]: Becoming Follower/Learner. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:46:51.866179 23529 raft_consensus.cc:1226] T 7baa2492df0340c4a5ac241959ce1ac3 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Rejecting Update request from peer 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 3. Current term is 4. Ops: []
I20260809 15:46:51.885140 23520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "538a228ddfb54bc6a36528d5ff07597b" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:51.865057 23285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "563daf42d7784db195ea99c3cb21b370" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:51.853664 26747 raft_consensus.cc:2726] T ae6d2c08a7a94f58ac8dd698e3fd456c P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260809 15:46:51.853790 26619 raft_consensus.cc:2726] T 73f6f44c259547f1b05a3a434616fa54 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260809 15:46:51.863502 27134 raft_consensus.cc:504] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:51.805298 27076 consensus_queue.cc:260] T cee02fff2c9c48a19f43eb358a89e7b0 P 67b3aeadfa7a432abe8af4d292e04ab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:51.866345 23282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b30f28311964a0b9149a50495e83d8f" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:51.923380 23282 raft_consensus.cc:3037] T 4b30f28311964a0b9149a50495e83d8f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:51.867173 27118 leader_election.cc:288] T 538a228ddfb54bc6a36528d5ff07597b P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:51.867357 26800 raft_consensus.cc:2726] T 4564dfc6d4094c1b9240b367ccecf3b1 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260809 15:46:51.869146 23283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0da4d5283c8a46878bd39b85e5c71f66" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:51.843204 26228 raft_consensus.cc:2781] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Leader election won for term 4
I20260809 15:46:51.871543 26405 raft_consensus.cc:2726] T 2424fd9bd7f14e508fd106cbfda786a4 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260809 15:46:51.872169 23273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7baa2492df0340c4a5ac241959ce1ac3" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:51.930956 23273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5712f6b6fffc4cc5a8ddfbe63e4097df" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:51.873438 26726 leader_election.cc:288] T 953eea268e17491ca76f3a555860a512 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:51.824956 26925 consensus_queue.cc:260] T f9e7cadcd24d45fcbfa84209225e8f2d P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:51.875939 23277 raft_consensus.cc:2454] T 28e17dcb5efe45b8b6a8c42da03b4819 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 6.
I20260809 15:46:51.997473 23518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "953eea268e17491ca76f3a555860a512" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:51.998164 23518 raft_consensus.cc:2454] T 953eea268e17491ca76f3a555860a512 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 5.
I20260809 15:46:51.847446 27116 raft_consensus.cc:482] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
I20260809 15:46:51.877729 27096 leader_election.cc:288] T 73f6f44c259547f1b05a3a434616fa54 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:51.871626 26757 raft_consensus.cc:504] T 907bacfa79d04a55a68c8dbe60056462 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:51.863996 26668 raft_consensus.cc:3037] T f9e7cadcd24d45fcbfa84209225e8f2d P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Advancing to term 5
I20260809 15:46:51.921840 23285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0741ab44d99439abc3aa3bdd27ee160" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:51.926378 23351 leader_election.cc:302] T 0da4d5283c8a46878bd39b85e5c71f66 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
W20260809 15:46:52.010461 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 84164.6us
I20260809 15:46:51.879441 26730 raft_consensus.cc:2726] T 05ad7d8fdca64e3da527280ca5be9280 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459) with higher term. Message: Invalid argument: T 05ad7d8fdca64e3da527280ca5be9280 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d for earlier term 4. Current term is 5.
I20260809 15:46:51.864957 26638 raft_consensus.cc:2726] T cef3e7277d7d4760824a3436ed4a847d P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260809 15:46:51.928188 23522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16ca412a0dae4feebe85dd01bf1a9173" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:51.939785 23273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5712f6b6fffc4cc5a8ddfbe63e4097df" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:52.006224 23277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c48a3049897452383257aa757834fb5" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:52.006996 23283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0741ab44d99439abc3aa3bdd27ee160" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:52.020717 23277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "294d4bcc1e7a477cb56455581277a60f" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:51.880378 26772 raft_consensus.cc:504] T 0acc47ce21c3422698d1ef7277c3ed63 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:52.018509 23353 leader_election.cc:302] T 953eea268e17491ca76f3a555860a512 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
W20260809 15:46:52.008596 23470 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 165670us
I20260809 15:46:52.021728 23277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "294d4bcc1e7a477cb56455581277a60f" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:52.022666 23518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df20671a7b874dbb98c0ef28371c1308" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:52.023460 23285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "563daf42d7784db195ea99c3cb21b370" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:52.024554 23273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d880de657e544468c82157769622a26" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:52.025372 23285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "953eea268e17491ca76f3a555860a512" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:52.026052 23285 raft_consensus.cc:2379] T 953eea268e17491ca76f3a555860a512 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d in current term 5: Already voted for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in this term.
I20260809 15:46:52.026227 23282 raft_consensus.cc:2454] T 4b30f28311964a0b9149a50495e83d8f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 3.
I20260809 15:46:52.027279 23351 leader_election.cc:302] T 4b30f28311964a0b9149a50495e83d8f P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c
I20260809 15:46:51.935242 27055 leader_election.cc:288] T 3924b648461e463b9d7925e5671d68ae P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 8 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:52.031939 23282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "297c82ed906d4b48b184985eed3de601" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
W20260809 15:46:52.032143 24406 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 225834us
I20260809 15:46:52.033704 23395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998c6cee72cf40228c6dba064c0db2da" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 8 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:52.034471 23395 raft_consensus.cc:2396] T 998c6cee72cf40228c6dba064c0db2da P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 for term 8 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260809 15:46:52.035466 23285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c29937a80724ddb9d07a27534a146f3" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:52.036458 23277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee02fff2c9c48a19f43eb358a89e7b0" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:52.036521 23273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "538a228ddfb54bc6a36528d5ff07597b" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:52.035818 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "563daf42d7784db195ea99c3cb21b370" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:52.035419 23351 leader_election.cc:302] T 297c82ed906d4b48b184985eed3de601 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:52.009210 26228 raft_consensus.cc:686] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 65c36d164b0540f099d592f38307ee6c [term 4 LEADER]: Becoming Leader. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
W20260809 15:46:52.038563 23472 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 182447us
I20260809 15:46:52.043164 23283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c48a3049897452383257aa757834fb5" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:52.027467 26726 raft_consensus.cc:2726] T 0da4d5283c8a46878bd39b85e5c71f66 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260809 15:46:52.046562 23270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a51c0baaa6e432997c9d3552af4f254" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:52.045882 23518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae6d2c08a7a94f58ac8dd698e3fd456c" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:52.046213 23285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee02fff2c9c48a19f43eb358a89e7b0" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:52.056905 23285 raft_consensus.cc:1261] T 99bdb93503db4de4a1cfb0161ea0c07c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260809 15:46:52.057878 23285 raft_consensus.cc:1261] T 28e17dcb5efe45b8b6a8c42da03b4819 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260809 15:46:52.059283 23404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a88421dba80d4d3388aa5eb62e23853b" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:52.059859 23404 raft_consensus.cc:2454] T a88421dba80d4d3388aa5eb62e23853b P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 3.
I20260809 15:46:52.060937 23402 raft_consensus.cc:1226] T 907bacfa79d04a55a68c8dbe60056462 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Rejecting Update request from peer 61e69695258f48baa984c9d395e8ce3f for earlier term 4. Current term is 5. Ops: []
I20260809 15:46:52.034305 26925 raft_consensus.cc:3037] T f9e7cadcd24d45fcbfa84209225e8f2d P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Advancing to term 5
I20260809 15:46:52.061401 23270 raft_consensus.cc:1226] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Rejecting Update request from peer c0f38c1908414566aa9115bb2cbdd85d for earlier term 3. Current term is 4. Ops: []
I20260809 15:46:52.045724 23223 leader_election.cc:302] T e71a05ba1d4d42bd95c3f85f9313399e P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:46:52.064553 23394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f9d6565aa4c45a3b3f01210d0eb71f7" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:52.065505 23285 raft_consensus.cc:3032] T 7baa2492df0340c4a5ac241959ce1ac3 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 LEADER]: Stepping down as leader of term 3
I20260809 15:46:52.049413 23273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a51c0baaa6e432997c9d3552af4f254" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:52.073506 23285 raft_consensus.cc:729] T 7baa2492df0340c4a5ac241959ce1ac3 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Running, Role: LEADER
W20260809 15:46:52.080870 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 45533.1us
I20260809 15:46:52.081838 23351 leader_election.cc:302] T 538a228ddfb54bc6a36528d5ff07597b P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:52.082899 23351 leader_election.cc:302] T 0c29937a80724ddb9d07a27534a146f3 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:52.019510 26966 raft_consensus.cc:504] T 30c1740ef6874df0a332fbd6fd4b03ce P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:52.049396 23400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7660dfa2ef694927b8eda4100a2bcd9b" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:52.048326 23395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "907bacfa79d04a55a68c8dbe60056462" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:52.046468 23392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7715d71dbac447b9351fc5eb36c5e80" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:52.092443 23395 raft_consensus.cc:2379] T 907bacfa79d04a55a68c8dbe60056462 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 in current term 5: Already voted for candidate c0f38c1908414566aa9115bb2cbdd85d in this term.
I20260809 15:46:52.047035 23282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a51c0baaa6e432997c9d3552af4f254" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:52.061978 27136 raft_consensus.cc:2781] T 953eea268e17491ca76f3a555860a512 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260809 15:46:52.098855 23274 raft_consensus.cc:1261] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Refusing update from remote peer 61e69695258f48baa984c9d395e8ce3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260809 15:46:52.055526 27084 raft_consensus.cc:2781] T 4b30f28311964a0b9149a50495e83d8f P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader election won for term 3
I20260809 15:46:52.051030 26747 raft_consensus.cc:482] T 00a1881267a14ba8b78c81040f17a014 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:52.088917 23522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "953eea268e17491ca76f3a555860a512" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:52.117015 23394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "953eea268e17491ca76f3a555860a512" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:52.084383 23270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e71a05ba1d4d42bd95c3f85f9313399e" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:52.075484 23285 consensus_queue.cc:260] T 7baa2492df0340c4a5ac241959ce1ac3 P 67b3aeadfa7a432abe8af4d292e04ab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:52.009727 27116 raft_consensus.cc:504] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:52.092438 23283 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f9e7cadcd24d45fcbfa84209225e8f2d"
dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
mode: GRACEFUL
new_leader_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
from {username='slave'} at 127.0.0.1:33468
I20260809 15:46:52.050246 26730 raft_consensus.cc:482] T 213543cdbe814c89a17a963469922c1b P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:52.086967 26810 raft_consensus.cc:482] T 05ad7d8fdca64e3da527280ca5be9280 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:52.092149 26228 consensus_queue.cc:237] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:52.101830 23395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e71a05ba1d4d42bd95c3f85f9313399e" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:52.106904 27076 raft_consensus.cc:3037] T cee02fff2c9c48a19f43eb358a89e7b0 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Advancing to term 5
I20260809 15:46:52.060395 27032 consensus_queue.cc:1035] T 05ad7d8fdca64e3da527280ca5be9280 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.041s
I20260809 15:46:52.086934 27073 consensus_queue.cc:1035] T 0da4d5283c8a46878bd39b85e5c71f66 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.022s
I20260809 15:46:52.109033 23404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df20671a7b874dbb98c0ef28371c1308" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:52.109936 23399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07fcabb569414bddbd39df6cedd24e64" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:52.088860 27075 raft_consensus.cc:2781] T e71a05ba1d4d42bd95c3f85f9313399e P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260809 15:46:52.111536 23529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2424fd9bd7f14e508fd106cbfda786a4" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:52.112841 23400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e4ffcf83c234ae7910ddef3ec0b4bda" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:52.113291 23397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2d4f749fcfd4fb3bfb8115af3471598" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:52.118958 23283 raft_consensus.cc:595] T f9e7cadcd24d45fcbfa84209225e8f2d P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Received request to transfer leadership to TS c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:52.114778 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3622f6e3e3641cea1960e3a31498bb7" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:52.119624 23276 raft_consensus.cc:1226] T d3be1e356b254d22b995be32f1c12f6f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Rejecting Update request from peer 61e69695258f48baa984c9d395e8ce3f for earlier term 4. Current term is 5. Ops: []
I20260809 15:46:52.121743 23395 raft_consensus.cc:2454] T e71a05ba1d4d42bd95c3f85f9313399e P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 5.
I20260809 15:46:52.122803 27174 consensus_queue.cc:1035] T 99bdb93503db4de4a1cfb0161ea0c07c P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 15:46:52.123103 26810 raft_consensus.cc:504] T 05ad7d8fdca64e3da527280ca5be9280 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:52.087462 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a51c0baaa6e432997c9d3552af4f254" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:52.102838 27136 raft_consensus.cc:482] T 953eea268e17491ca76f3a555860a512 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Starting leader election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:52.124046 23280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "563daf42d7784db195ea99c3cb21b370" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:52.098429 24452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a823d35f4f348d1b52ba9e6d80b00d2" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f"
I20260809 15:46:52.036994 26527 consensus_queue.cc:260] T c5c2e425a74b4e64ae70b5def750434c P c0f38c1908414566aa9115bb2cbdd85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:52.125434 23520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9e7cadcd24d45fcbfa84209225e8f2d" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:52.236629 23282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e14d118b2eaa42aa8a7cd4025aba7128" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:52.218039 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00a1881267a14ba8b78c81040f17a014" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:52.280409 23283 raft_consensus.cc:603] T f9e7cadcd24d45fcbfa84209225e8f2d P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260809 15:46:52.161226 23396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba56725e42f74a1baa4c04d9660727f9" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:52.244025 23274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16ca412a0dae4feebe85dd01bf1a9173" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:52.099205 26800 raft_consensus.cc:2726] T 297c82ed906d4b48b184985eed3de601 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260809 15:46:52.235654 23228 leader_election.cc:302] T 294d4bcc1e7a477cb56455581277a60f P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c
W20260809 15:46:52.284765 23470 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 195858us
I20260809 15:46:52.280259 23402 raft_consensus.cc:2454] T a3622f6e3e3641cea1960e3a31498bb7 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 3.
I20260809 15:46:52.284782 23282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d880de657e544468c82157769622a26" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:52.128594 23404 raft_consensus.cc:2454] T df20671a7b874dbb98c0ef28371c1308 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 4.
I20260809 15:46:52.236279 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ae14a74a26f434db08c652e097385c8" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:52.128707 23285 raft_consensus.cc:3037] T 7baa2492df0340c4a5ac241959ce1ac3 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:46:52.283144 23526 raft_consensus.cc:2396] T 00a1881267a14ba8b78c81040f17a014 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 for term 7 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260809 15:46:52.283092 23396 raft_consensus.cc:2454] T ba56725e42f74a1baa4c04d9660727f9 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 3.
W20260809 15:46:52.194322 24406 leader_election.cc:334] T 0c48a3049897452383257aa757834fb5 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459): Timed out: RequestConsensusVote RPC to 127.20.202.65:35459 timed out after 1.606s (SENT)
I20260809 15:46:52.289685 24406 leader_election.cc:302] T 0c48a3049897452383257aa757834fb5 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:52.289932 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ced718c1a5c4da79cb8c1ebd1435b87" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:52.298100 23404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ae14a74a26f434db08c652e097385c8" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:52.298810 23404 raft_consensus.cc:3037] T 6ae14a74a26f434db08c652e097385c8 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Advancing to term 5
W20260809 15:46:52.300204 24406 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 105918us
I20260809 15:46:52.121609 27084 raft_consensus.cc:686] T 4b30f28311964a0b9149a50495e83d8f P c0f38c1908414566aa9115bb2cbdd85d [term 3 LEADER]: Becoming Leader. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:46:52.301795 23527 raft_consensus.cc:2454] T 6ae14a74a26f434db08c652e097385c8 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 5.
I20260809 15:46:52.102499 27118 raft_consensus.cc:2726] T 538a228ddfb54bc6a36528d5ff07597b P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260809 15:46:52.125577 23393 raft_consensus.cc:1226] T 7baa2492df0340c4a5ac241959ce1ac3 P c0f38c1908414566aa9115bb2cbdd85d [term 4 LEADER]: Rejecting Update request from peer 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 3. Current term is 4. Ops: []
I20260809 15:46:52.303417 23276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee02fff2c9c48a19f43eb358a89e7b0" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
W20260809 15:46:52.304065 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 68502.8us
I20260809 15:46:52.305939 27079 raft_consensus.cc:2726] T 0c48a3049897452383257aa757834fb5 P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260809 15:46:52.124579 26981 raft_consensus.cc:504] T d2d4f749fcfd4fb3bfb8115af3471598 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:52.306185 23400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7715d71dbac447b9351fc5eb36c5e80" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:52.121471 26747 raft_consensus.cc:504] T 00a1881267a14ba8b78c81040f17a014 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:52.317553 23285 raft_consensus.cc:1261] T 7baa2492df0340c4a5ac241959ce1ac3 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20260809 15:46:52.306862 26740 consensus_meta.cc:287] T d1a134ba33ea4076bbaac0ae460c17ba P c0f38c1908414566aa9115bb2cbdd85d: Time spent flushing consensus metadata: real 0.594s user 0.004s sys 0.005s
I20260809 15:46:52.299505 23399 raft_consensus.cc:2454] T 07fcabb569414bddbd39df6cedd24e64 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 5.
I20260809 15:46:52.320358 23399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae6d2c08a7a94f58ac8dd698e3fd456c" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:52.321072 23399 raft_consensus.cc:2379] T ae6d2c08a7a94f58ac8dd698e3fd456c P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 in current term 6: Already voted for candidate c0f38c1908414566aa9115bb2cbdd85d in this term.
I20260809 15:46:52.320269 26996 consensus_queue.cc:1046] T 907bacfa79d04a55a68c8dbe60056462 P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.002s
I20260809 15:46:52.322070 23399 raft_consensus.cc:1226] T 4564dfc6d4094c1b9240b367ccecf3b1 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Rejecting Update request from peer 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260809 15:46:52.322702 26996 raft_consensus.cc:3032] T 907bacfa79d04a55a68c8dbe60056462 P 61e69695258f48baa984c9d395e8ce3f [term 4 LEADER]: Stepping down as leader of term 4
I20260809 15:46:52.323065 26996 raft_consensus.cc:729] T 907bacfa79d04a55a68c8dbe60056462 P 61e69695258f48baa984c9d395e8ce3f [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 61e69695258f48baa984c9d395e8ce3f, State: Running, Role: LEADER
I20260809 15:46:52.315037 23345 leader_election.cc:302] T e71a05ba1d4d42bd95c3f85f9313399e P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 61e69695258f48baa984c9d395e8ce3f, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:52.304608 23522 raft_consensus.cc:2454] T 953eea268e17491ca76f3a555860a512 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 5.
I20260809 15:46:52.308377 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3924b648461e463b9d7925e5671d68ae" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:52.235654 26966 leader_election.cc:288] T 30c1740ef6874df0a332fbd6fd4b03ce P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 5 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:52.323232 23399 raft_consensus.cc:1226] T e71a05ba1d4d42bd95c3f85f9313399e P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Rejecting Update request from peer 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260809 15:46:52.323822 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3924b648461e463b9d7925e5671d68ae" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:52.329136 23526 raft_consensus.cc:3037] T 3924b648461e463b9d7925e5671d68ae P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Advancing to term 8
I20260809 15:46:52.324518 23470 leader_election.cc:302] T d0721a4403a440f383318a3cea9d1442 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:46:52.327389 23402 raft_consensus.cc:3037] T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Advancing to term 8
I20260809 15:46:52.121589 26405 raft_consensus.cc:2726] T 0c29937a80724ddb9d07a27534a146f3 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260809 15:46:52.209926 26757 leader_election.cc:288] T 907bacfa79d04a55a68c8dbe60056462 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:52.287734 27136 raft_consensus.cc:3037] T 953eea268e17491ca76f3a555860a512 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Advancing to term 5
W20260809 15:46:52.331657 24406 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.20.202.65:35459 timed out after 1.780s (SENT) [suppressed 12 similar messages]
I20260809 15:46:52.216746 26772 leader_election.cc:288] T 0acc47ce21c3422698d1ef7277c3ed63 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 6 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
W20260809 15:46:52.339471 23472 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 263288us
I20260809 15:46:52.340723 23400 raft_consensus.cc:1226] T d3be1e356b254d22b995be32f1c12f6f P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Rejecting Update request from peer 61e69695258f48baa984c9d395e8ce3f for earlier term 4. Current term is 5. Ops: []
I20260809 15:46:52.342200 21289 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260809 15:46:52.215674 27134 leader_election.cc:288] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:52.344112 26399 raft_consensus.cc:2781] T d0721a4403a440f383318a3cea9d1442 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Leader election won for term 6
I20260809 15:46:52.344452 23470 leader_election.cc:302] T cee02fff2c9c48a19f43eb358a89e7b0 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 61e69695258f48baa984c9d395e8ce3f, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:52.344681 26399 raft_consensus.cc:686] T d0721a4403a440f383318a3cea9d1442 P 65c36d164b0540f099d592f38307ee6c [term 6 LEADER]: Becoming Leader. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:46:52.345621 26399 consensus_queue.cc:237] T d0721a4403a440f383318a3cea9d1442 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:52.307243 27084 consensus_queue.cc:237] T 4b30f28311964a0b9149a50495e83d8f P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:52.313383 23393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00a1881267a14ba8b78c81040f17a014" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:52.314833 26810 leader_election.cc:288] T 05ad7d8fdca64e3da527280ca5be9280 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:52.315227 24410 leader_election.cc:302] T 7660dfa2ef694927b8eda4100a2bcd9b P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:52.299860 27072 raft_consensus.cc:2726] T 294d4bcc1e7a477cb56455581277a60f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260809 15:46:52.331779 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 247913us
I20260809 15:46:52.332983 23345 leader_election.cc:302] T 563daf42d7784db195ea99c3cb21b370 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 61e69695258f48baa984c9d395e8ce3f, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:52.351438 21289 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 15:46:52.352100 21289 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 15:46:52.124117 26730 raft_consensus.cc:504] T 213543cdbe814c89a17a963469922c1b P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:52.337684 26996 consensus_queue.cc:260] T 907bacfa79d04a55a68c8dbe60056462 P 61e69695258f48baa984c9d395e8ce3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:52.328639 27116 leader_election.cc:288] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:52.216006 26908 leader_election.cc:288] T 4b30f28311964a0b9149a50495e83d8f P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:52.312242 23396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2424fd9bd7f14e508fd106cbfda786a4" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:52.365753 23397 raft_consensus.cc:2379] T d2d4f749fcfd4fb3bfb8115af3471598 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 in current term 5: Already voted for candidate c0f38c1908414566aa9115bb2cbdd85d in this term.
I20260809 15:46:52.365911 23396 raft_consensus.cc:2379] T 2424fd9bd7f14e508fd106cbfda786a4 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 in current term 6: Already voted for candidate c0f38c1908414566aa9115bb2cbdd85d in this term.
I20260809 15:46:52.357915 23529 raft_consensus.cc:1226] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Rejecting Update request from peer 61e69695258f48baa984c9d395e8ce3f for earlier term 6. Current term is 7. Ops: []
I20260809 15:46:52.359786 23351 leader_election.cc:302] T 0a51c0baaa6e432997c9d3552af4f254 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 61e69695258f48baa984c9d395e8ce3f, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:52.363735 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a823d35f4f348d1b52ba9e6d80b00d2" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:52.370512 23522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a823d35f4f348d1b52ba9e6d80b00d2" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:52.363699 23400 raft_consensus.cc:1261] T 0a51c0baaa6e432997c9d3552af4f254 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Refusing update from remote peer 67b3aeadfa7a432abe8af4d292e04ab9: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20260809 15:46:52.302073 27075 raft_consensus.cc:482] T e71a05ba1d4d42bd95c3f85f9313399e P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:46:52.377497 23395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9e7cadcd24d45fcbfa84209225e8f2d" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:52.381497 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0acc47ce21c3422698d1ef7277c3ed63" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
W20260809 15:46:52.379791 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 50706.6us
I20260809 15:46:52.382429 23228 leader_election.cc:302] T 30c1740ef6874df0a332fbd6fd4b03ce P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:52.380689 23472 leader_election.cc:302] T 4564dfc6d4094c1b9240b367ccecf3b1 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters: 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:52.327730 23392 raft_consensus.cc:2454] T f7715d71dbac447b9351fc5eb36c5e80 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 6.
I20260809 15:46:52.384325 23521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad8a3b25d7b49e5a2142d5709edd0c7" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:52.384469 26826 raft_consensus.cc:482] T c0741ab44d99439abc3aa3bdd27ee160 P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:52.385207 24452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "907bacfa79d04a55a68c8dbe60056462" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f"
I20260809 15:46:52.360630 27097 raft_consensus.cc:2726] T 7660dfa2ef694927b8eda4100a2bcd9b P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260809 15:46:52.336436 27175 raft_consensus.cc:2726] T e71a05ba1d4d42bd95c3f85f9313399e P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260809 15:46:52.388552 23393 raft_consensus.cc:2396] T 00a1881267a14ba8b78c81040f17a014 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 for term 7 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260809 15:46:52.390650 23517 raft_consensus.cc:1226] T 73f6f44c259547f1b05a3a434616fa54 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Rejecting Update request from peer c0f38c1908414566aa9115bb2cbdd85d for earlier term 5. Current term is 6. Ops: []
I20260809 15:46:52.362704 26757 raft_consensus.cc:2726] T 563daf42d7784db195ea99c3cb21b370 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260809 15:46:52.403019 23273 raft_consensus.cc:1261] T f7715d71dbac447b9351fc5eb36c5e80 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260809 15:46:52.385545 27053 consensus_queue.cc:1035] T 763e58d6ba644eea830a6c1664d39c9b P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 15:46:52.399255 23353 leader_election.cc:302] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c
I20260809 15:46:52.371724 26668 raft_consensus.cc:2726] T f9e7cadcd24d45fcbfa84209225e8f2d P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260809 15:46:52.379596 27116 raft_consensus.cc:2726] T 0a51c0baaa6e432997c9d3552af4f254 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260809 15:46:52.376056 26611 raft_consensus.cc:2726] T cee02fff2c9c48a19f43eb358a89e7b0 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260809 15:46:52.425750 23526 raft_consensus.cc:2454] T 3924b648461e463b9d7925e5671d68ae P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 8.
W20260809 15:46:52.457005 23472 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 76393.6us
W20260809 15:46:52.460065 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 77698.7us
I20260809 15:46:52.462178 23286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30c1740ef6874df0a332fbd6fd4b03ce" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:52.358589 26740 consensus_queue.cc:1035] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.064s
W20260809 15:46:52.466303 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 97648.2us
I20260809 15:46:52.414321 27075 raft_consensus.cc:3037] T e71a05ba1d4d42bd95c3f85f9313399e P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Advancing to term 6
I20260809 15:46:52.467671 23518 raft_consensus.cc:1226] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 65c36d164b0540f099d592f38307ee6c [term 4 LEADER]: Rejecting Update request from peer c0f38c1908414566aa9115bb2cbdd85d for earlier term 3. Current term is 4. Ops: []
I20260809 15:46:52.463680 27079 raft_consensus.cc:482] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 15:46:52.476456 24410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 77775.5us
I20260809 15:46:52.458865 26908 raft_consensus.cc:2726] T 30c1740ef6874df0a332fbd6fd4b03ce P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20260809 15:46:52.468406 23353 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 69251.7us
I20260809 15:46:52.421043 26996 raft_consensus.cc:3037] T 907bacfa79d04a55a68c8dbe60056462 P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Advancing to term 5
I20260809 15:46:52.421151 26826 raft_consensus.cc:504] T c0741ab44d99439abc3aa3bdd27ee160 P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:52.483862 26747 leader_election.cc:288] T 00a1881267a14ba8b78c81040f17a014 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:52.484102 23399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ced718c1a5c4da79cb8c1ebd1435b87" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:52.496891 23517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2d4f749fcfd4fb3bfb8115af3471598" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:52.497308 23273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2d4f749fcfd4fb3bfb8115af3471598" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:52.503662 23273 raft_consensus.cc:3037] T d2d4f749fcfd4fb3bfb8115af3471598 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Advancing to term 5
I20260809 15:46:52.500928 23399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30c1740ef6874df0a332fbd6fd4b03ce" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:52.504724 23399 raft_consensus.cc:2454] T 30c1740ef6874df0a332fbd6fd4b03ce P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 4.
I20260809 15:46:52.496628 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00a1881267a14ba8b78c81040f17a014" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:52.505532 23526 raft_consensus.cc:2454] T 00a1881267a14ba8b78c81040f17a014 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 6.
I20260809 15:46:52.506541 23353 leader_election.cc:302] T 00a1881267a14ba8b78c81040f17a014 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:46:52.507557 23286 raft_consensus.cc:3037] T 30c1740ef6874df0a332fbd6fd4b03ce P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Advancing to term 5
I20260809 15:46:52.503332 23397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30c1740ef6874df0a332fbd6fd4b03ce" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:52.508491 23397 raft_consensus.cc:3037] T 30c1740ef6874df0a332fbd6fd4b03ce P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Advancing to term 5
I20260809 15:46:52.509608 26719 raft_consensus.cc:2781] T 00a1881267a14ba8b78c81040f17a014 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260809 15:46:52.510038 26719 raft_consensus.cc:482] T 00a1881267a14ba8b78c81040f17a014 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:46:52.510444 26719 raft_consensus.cc:3037] T 00a1881267a14ba8b78c81040f17a014 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Advancing to term 7
I20260809 15:46:52.484658 26772 raft_consensus.cc:2726] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260809 15:46:52.489452 27116 consensus_queue.cc:1046] T 6ced718c1a5c4da79cb8c1ebd1435b87 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 15:46:52.490132 27084 consensus_queue.cc:1035] T 7baa2492df0340c4a5ac241959ce1ac3 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 15:46:52.489368 26527 raft_consensus.cc:3037] T c5c2e425a74b4e64ae70b5def750434c P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Advancing to term 5
I20260809 15:46:52.490720 23276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0acc47ce21c3422698d1ef7277c3ed63" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:52.491405 26981 leader_election.cc:288] T d2d4f749fcfd4fb3bfb8115af3471598 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:52.491583 23285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "907bacfa79d04a55a68c8dbe60056462" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:52.499876 23393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0721a4403a440f383318a3cea9d1442" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:52.491415 23282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00a1881267a14ba8b78c81040f17a014" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:52.466547 27011 raft_consensus.cc:2781] T 4564dfc6d4094c1b9240b367ccecf3b1 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Leader election won for term 5
I20260809 15:46:52.493076 23228 leader_election.cc:302] T d0721a4403a440f383318a3cea9d1442 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:52.496202 23395 raft_consensus.cc:2454] T f9e7cadcd24d45fcbfa84209225e8f2d P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 5.
I20260809 15:46:52.469522 24448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad8a3b25d7b49e5a2142d5709edd0c7" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:52.492209 27194 consensus_queue.cc:1046] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.022s
I20260809 15:46:52.517007 23526 raft_consensus.cc:1261] T 7baa2492df0340c4a5ac241959ce1ac3 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260809 15:46:52.484503 23396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00a1881267a14ba8b78c81040f17a014" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:52.522068 23402 raft_consensus.cc:2454] T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 8.
I20260809 15:46:52.524402 23399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a823d35f4f348d1b52ba9e6d80b00d2" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:52.498783 23392 raft_consensus.cc:1261] T 05ad7d8fdca64e3da527280ca5be9280 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260809 15:46:52.479236 27178 consensus_queue.cc:1035] T 28e17dcb5efe45b8b6a8c42da03b4819 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.070s
I20260809 15:46:52.496986 27118 consensus_queue.cc:1046] T 73f6f44c259547f1b05a3a434616fa54 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.033s
I20260809 15:46:52.498718 27136 raft_consensus.cc:504] T 953eea268e17491ca76f3a555860a512 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:52.494747 27079 raft_consensus.cc:504] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:52.530699 23282 raft_consensus.cc:2454] T 00a1881267a14ba8b78c81040f17a014 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 6.
I20260809 15:46:52.718370 23404 raft_consensus.cc:2454] T 6ae14a74a26f434db08c652e097385c8 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 5.
I20260809 15:46:52.862586 23273 raft_consensus.cc:2454] T d2d4f749fcfd4fb3bfb8115af3471598 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 5.
I20260809 15:46:52.955585 23397 raft_consensus.cc:2454] T 30c1740ef6874df0a332fbd6fd4b03ce P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 5.
W20260809 15:46:53.077117 24406 leader_election.cc:334] T 563daf42d7784db195ea99c3cb21b370 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459): Timed out: RequestConsensusVote RPC to 127.20.202.65:35459 timed out after 1.669s (SENT)
I20260809 15:46:53.083221 24406 leader_election.cc:302] T 563daf42d7784db195ea99c3cb21b370 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f; no voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:53.084746 23404 raft_consensus.cc:1226] T ae6d2c08a7a94f58ac8dd698e3fd456c P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 5. Current term is 6. Ops: []
I20260809 15:46:53.086011 23404 raft_consensus.cc:1226] T 4b30f28311964a0b9149a50495e83d8f P c0f38c1908414566aa9115bb2cbdd85d [term 3 LEADER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 2. Current term is 3. Ops: []
I20260809 15:46:52.578302 23520 raft_consensus.cc:2454] T f9e7cadcd24d45fcbfa84209225e8f2d P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 5.
W20260809 15:46:52.969116 24410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 451891us
I20260809 15:46:52.529091 27196 consensus_queue.cc:1046] T d3be1e356b254d22b995be32f1c12f6f P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: INVALID_TERM, Last received: 4.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.575s
W20260809 15:46:53.076099 23286 consensus_meta.cc:287] T 30c1740ef6874df0a332fbd6fd4b03ce P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent flushing consensus metadata: real 0.568s user 0.021s sys 0.013s
I20260809 15:46:53.090909 23286 raft_consensus.cc:2454] T 30c1740ef6874df0a332fbd6fd4b03ce P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 5.
I20260809 15:46:52.541857 23351 leader_election.cc:302] T 0acc47ce21c3422698d1ef7277c3ed63 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:53.092324 23345 leader_election.cc:302] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c
I20260809 15:46:52.581727 23277 raft_consensus.cc:2454] T cee02fff2c9c48a19f43eb358a89e7b0 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 5.
I20260809 15:46:52.538503 26810 raft_consensus.cc:2726] T d0721a4403a440f383318a3cea9d1442 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260809 15:46:53.093814 23517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0741ab44d99439abc3aa3bdd27ee160" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:53.078397 23402 pending_rounds.cc:76] T cef3e7277d7d4760824a3436ed4a847d P c0f38c1908414566aa9115bb2cbdd85d: Aborting all ops after (but not including) 2
I20260809 15:46:52.537868 27084 consensus_queue.cc:1225] T 7baa2492df0340c4a5ac241959ce1ac3 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Peer 67b3aeadfa7a432abe8af4d292e04ab9 log is divergent from this leader: its last log entry 3.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260809 15:46:53.100346 27055 raft_consensus.cc:2726] T 563daf42d7784db195ea99c3cb21b370 P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260809 15:46:52.763829 27198 log.cc:924] Time spent T 4b30f28311964a0b9149a50495e83d8f P c0f38c1908414566aa9115bb2cbdd85d: Append to log took a long time: real 0.145s user 0.001s sys 0.000s
I20260809 15:46:52.688362 27076 raft_consensus.cc:482] T d1a134ba33ea4076bbaac0ae460c17ba P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:53.097450 26996 raft_consensus.cc:3032] T d3be1e356b254d22b995be32f1c12f6f P 61e69695258f48baa984c9d395e8ce3f [term 4 LEADER]: Stepping down as leader of term 4
I20260809 15:46:53.098026 24410 leader_election.cc:302] T 3924b648461e463b9d7925e5671d68ae P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c; no voters:
I20260809 15:46:53.099779 23402 pending_rounds.cc:97] T cef3e7277d7d4760824a3436ed4a847d P c0f38c1908414566aa9115bb2cbdd85d: Aborting uncommitted NO_OP operation due to leader change: 5.3
I20260809 15:46:53.105095 23404 raft_consensus.cc:1226] T d2d4f749fcfd4fb3bfb8115af3471598 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 4. Current term is 5. Ops: []
W20260809 15:46:53.102648 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 627908us
I20260809 15:46:53.105329 23402 raft_consensus.cc:2866] T cef3e7277d7d4760824a3436ed4a847d P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 15:46:53.105985 23394 raft_consensus.cc:2379] T 953eea268e17491ca76f3a555860a512 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 in current term 5: Already voted for candidate c0f38c1908414566aa9115bb2cbdd85d in this term.
I20260809 15:46:53.107564 27136 leader_election.cc:288] T 953eea268e17491ca76f3a555860a512 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:53.108119 27136 raft_consensus.cc:2726] T 0acc47ce21c3422698d1ef7277c3ed63 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260809 15:46:52.538523 27134 raft_consensus.cc:3032] T 6ced718c1a5c4da79cb8c1ebd1435b87 P c0f38c1908414566aa9115bb2cbdd85d [term 3 LEADER]: Stepping down as leader of term 3
I20260809 15:46:53.110375 23392 raft_consensus.cc:1226] T 2424fd9bd7f14e508fd106cbfda786a4 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 5. Current term is 6. Ops: []
I20260809 15:46:53.110327 27134 raft_consensus.cc:729] T 6ced718c1a5c4da79cb8c1ebd1435b87 P c0f38c1908414566aa9115bb2cbdd85d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:46:53.103080 27084 raft_consensus.cc:2726] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260809 15:46:53.111296 27134 consensus_queue.cc:260] T 6ced718c1a5c4da79cb8c1ebd1435b87 P c0f38c1908414566aa9115bb2cbdd85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:53.112414 27134 raft_consensus.cc:3037] T 6ced718c1a5c4da79cb8c1ebd1435b87 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:46:53.104941 26996 raft_consensus.cc:729] T d3be1e356b254d22b995be32f1c12f6f P 61e69695258f48baa984c9d395e8ce3f [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 61e69695258f48baa984c9d395e8ce3f, State: Running, Role: LEADER
I20260809 15:46:52.572166 27011 raft_consensus.cc:686] T 4564dfc6d4094c1b9240b367ccecf3b1 P 65c36d164b0540f099d592f38307ee6c [term 5 LEADER]: Becoming Leader. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:46:53.115048 26619 consensus_queue.cc:1035] T f7715d71dbac447b9351fc5eb36c5e80 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20260809 15:46:53.115499 26996 consensus_queue.cc:260] T d3be1e356b254d22b995be32f1c12f6f P 61e69695258f48baa984c9d395e8ce3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:52.940402 26527 raft_consensus.cc:482] T 763e58d6ba644eea830a6c1664d39c9b P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:53.124096 23404 raft_consensus.cc:1226] T 0acc47ce21c3422698d1ef7277c3ed63 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 5. Current term is 6. Ops: []
I20260809 15:46:52.580237 26826 leader_election.cc:288] T c0741ab44d99439abc3aa3bdd27ee160 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:53.124022 23228 leader_election.cc:302] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c
I20260809 15:46:52.913587 24452 raft_consensus.cc:2454] T 907bacfa79d04a55a68c8dbe60056462 P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 5.
I20260809 15:46:52.923128 23393 raft_consensus.cc:2454] T d0721a4403a440f383318a3cea9d1442 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 6.
I20260809 15:46:53.116102 27011 consensus_queue.cc:237] T 4564dfc6d4094c1b9240b367ccecf3b1 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:52.903149 27075 raft_consensus.cc:504] T e71a05ba1d4d42bd95c3f85f9313399e P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:53.133153 26996 raft_consensus.cc:3037] T d3be1e356b254d22b995be32f1c12f6f P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Advancing to term 5
I20260809 15:46:52.866802 26719 raft_consensus.cc:504] T 00a1881267a14ba8b78c81040f17a014 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:53.128268 27166 raft_consensus.cc:482] T e14d118b2eaa42aa8a7cd4025aba7128 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:53.128664 27197 raft_consensus.cc:2781] T 3924b648461e463b9d7925e5671d68ae P 61e69695258f48baa984c9d395e8ce3f [term 8 FOLLOWER]: Leader election won for term 8
W20260809 15:46:53.142371 26399 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.631s user 0.010s sys 0.022s
I20260809 15:46:53.117486 27076 raft_consensus.cc:504] T d1a134ba33ea4076bbaac0ae460c17ba P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:53.158774 23270 raft_consensus.cc:2354] T e71a05ba1d4d42bd95c3f85f9313399e P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 61e69695258f48baa984c9d395e8ce3f for earlier term 5. Current term is 6.
I20260809 15:46:53.152122 27166 raft_consensus.cc:504] T e14d118b2eaa42aa8a7cd4025aba7128 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:53.160179 27096 raft_consensus.cc:2726] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260809 15:46:53.161326 27079 leader_election.cc:288] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
W20260809 15:46:53.157886 26399 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.647s user 0.010s sys 0.022s
I20260809 15:46:53.162613 23396 raft_consensus.cc:2354] T 00a1881267a14ba8b78c81040f17a014 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c for earlier term 6. Current term is 7.
I20260809 15:46:53.130410 26527 raft_consensus.cc:504] T 763e58d6ba644eea830a6c1664d39c9b P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:53.171069 23517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3aa6a9a52804df0ba9b0ca3c82f38c3" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:53.151602 27197 raft_consensus.cc:686] T 3924b648461e463b9d7925e5671d68ae P 61e69695258f48baa984c9d395e8ce3f [term 8 LEADER]: Becoming Leader. State: Replica: 61e69695258f48baa984c9d395e8ce3f, State: Running, Role: LEADER
I20260809 15:46:53.174285 23517 raft_consensus.cc:2396] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 61e69695258f48baa984c9d395e8ce3f for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260809 15:46:53.189522 23518 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56766 (request call id 1886) took 1110 ms. Trace:
I20260809 15:46:53.189791 23518 rpcz_store.cc:276] 0809 15:46:52.079333 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:52.079483 (+ 150us) service_pool.cc:227] Handling call
0809 15:46:52.092562 (+ 13079us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:52.479037 (+386475us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:53.189365 (+710328us) spinlock_profiling.cc:243] Waited 375 ms on lock 0x7b5400107cd4. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e9220f51 00007f34e921f99e 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772 00007f34e764a94c 00007f34e7649d86
0809 15:46:53.189480 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":862324992}
I20260809 15:46:53.192906 23404 raft_consensus.cc:1261] T 763e58d6ba644eea830a6c1664d39c9b P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260809 15:46:53.203975 27197 consensus_queue.cc:237] T 3924b648461e463b9d7925e5671d68ae P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
W20260809 15:46:53.209292 23345 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.68:34689, user_credentials={real_user=slave}} blocked reactor thread for 117054us
W20260809 15:46:53.210692 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 668867us
I20260809 15:46:53.211143 23517 raft_consensus.cc:1226] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Rejecting Update request from peer 61e69695258f48baa984c9d395e8ce3f for earlier term 6. Current term is 7. Ops: []
I20260809 15:46:53.212862 27136 raft_consensus.cc:482] T d3be1e356b254d22b995be32f1c12f6f P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:53.212718 27206 consensus_queue.cc:1046] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.006s
I20260809 15:46:53.213230 27174 raft_consensus.cc:482] T d1a134ba33ea4076bbaac0ae460c17ba P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:53.209230 27196 raft_consensus.cc:482] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
W20260809 15:46:53.261279 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 48254.8us
I20260809 15:46:53.262434 23351 leader_election.cc:302] T d2d4f749fcfd4fb3bfb8115af3471598 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c
I20260809 15:46:53.202755 23393 raft_consensus.cc:1261] T f7715d71dbac447b9351fc5eb36c5e80 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
W20260809 15:46:53.264251 23223 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.68:34689, user_credentials={real_user=slave}} blocked reactor thread for 133648us
I20260809 15:46:53.224992 26826 raft_consensus.cc:482] T 294d4bcc1e7a477cb56455581277a60f P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:53.258543 26719 leader_election.cc:288] T 00a1881267a14ba8b78c81040f17a014 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 7 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
W20260809 15:46:53.275522 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 151598us
I20260809 15:46:53.278481 23273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "213543cdbe814c89a17a963469922c1b" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:53.282485 23282 raft_consensus.cc:1261] T 4b30f28311964a0b9149a50495e83d8f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260809 15:46:53.283136 23228 leader_election.cc:302] T 0acc47ce21c3422698d1ef7277c3ed63 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:53.283432 24475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e71a05ba1d4d42bd95c3f85f9313399e" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f"
I20260809 15:46:53.283977 24475 raft_consensus.cc:3037] T e71a05ba1d4d42bd95c3f85f9313399e P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Advancing to term 6
I20260809 15:46:53.270895 27075 leader_election.cc:288] T e71a05ba1d4d42bd95c3f85f9313399e P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 6 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:53.247902 27136 raft_consensus.cc:504] T d3be1e356b254d22b995be32f1c12f6f P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:53.286823 23282 pending_rounds.cc:76] T 7baa2492df0340c4a5ac241959ce1ac3 P 67b3aeadfa7a432abe8af4d292e04ab9: Aborting all ops after (but not including) 1
I20260809 15:46:53.293294 23282 pending_rounds.cc:97] T 7baa2492df0340c4a5ac241959ce1ac3 P 67b3aeadfa7a432abe8af4d292e04ab9: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20260809 15:46:53.293555 23282 raft_consensus.cc:2866] T 7baa2492df0340c4a5ac241959ce1ac3 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 15:46:53.293673 23472 leader_election.cc:302] T 30c1740ef6874df0a332fbd6fd4b03ce P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:46:53.287626 23285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "953eea268e17491ca76f3a555860a512" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:53.294544 23472 leader_election.cc:302] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:53.288874 23274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "763e58d6ba644eea830a6c1664d39c9b" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:53.295186 23285 raft_consensus.cc:2379] T 953eea268e17491ca76f3a555860a512 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d in current term 5: Already voted for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in this term.
I20260809 15:46:53.295562 23274 raft_consensus.cc:2454] T 763e58d6ba644eea830a6c1664d39c9b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 5.
I20260809 15:46:53.263720 27174 raft_consensus.cc:504] T d1a134ba33ea4076bbaac0ae460c17ba P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:53.311151 23276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00a1881267a14ba8b78c81040f17a014" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:53.311744 23276 raft_consensus.cc:3037] T 00a1881267a14ba8b78c81040f17a014 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Advancing to term 7
I20260809 15:46:53.273793 27213 raft_consensus.cc:2781] T d2d4f749fcfd4fb3bfb8115af3471598 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Leader election won for term 5
I20260809 15:46:53.263446 27196 raft_consensus.cc:504] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:53.273435 26826 raft_consensus.cc:504] T 294d4bcc1e7a477cb56455581277a60f P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:53.299180 23393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3aa6a9a52804df0ba9b0ca3c82f38c3" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:53.284793 27076 leader_election.cc:288] T d1a134ba33ea4076bbaac0ae460c17ba P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:53.292114 27229 raft_consensus.cc:2726] T 0acc47ce21c3422698d1ef7277c3ed63 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260809 15:46:53.299700 26966 raft_consensus.cc:2781] T 30c1740ef6874df0a332fbd6fd4b03ce P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Leader election won for term 5
I20260809 15:46:53.299527 26527 leader_election.cc:288] T 763e58d6ba644eea830a6c1664d39c9b P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
W20260809 15:46:53.325601 27207 log.cc:924] Time spent T 0a51c0baaa6e432997c9d3552af4f254 P 67b3aeadfa7a432abe8af4d292e04ab9: Append to log took a long time: real 0.138s user 0.001s sys 0.000s
I20260809 15:46:53.348001 23518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "213543cdbe814c89a17a963469922c1b" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:53.328900 23392 raft_consensus.cc:1261] T 6ced718c1a5c4da79cb8c1ebd1435b87 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260809 15:46:53.329219 27213 raft_consensus.cc:686] T d2d4f749fcfd4fb3bfb8115af3471598 P c0f38c1908414566aa9115bb2cbdd85d [term 5 LEADER]: Becoming Leader. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
W20260809 15:46:53.346293 27209 log.cc:924] Time spent T d0721a4403a440f383318a3cea9d1442 P 65c36d164b0540f099d592f38307ee6c: Append to log took a long time: real 0.144s user 0.001s sys 0.000s
I20260809 15:46:53.316630 27166 leader_election.cc:288] T e14d118b2eaa42aa8a7cd4025aba7128 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:53.367318 23526 raft_consensus.cc:1261] T 99bdb93503db4de4a1cfb0161ea0c07c P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260809 15:46:53.315132 27219 raft_consensus.cc:2726] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260809 15:46:53.380357 24410 leader_election.cc:302] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:53.381532 23351 leader_election.cc:302] T 763e58d6ba644eea830a6c1664d39c9b P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters:
W20260809 15:46:53.377764 24406 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 90623.2us
I20260809 15:46:53.380451 23518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "953eea268e17491ca76f3a555860a512" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
W20260809 15:46:53.383388 24406 leader_election.cc:334] T 0a51c0baaa6e432997c9d3552af4f254 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459): Timed out: RequestConsensusVote RPC to 127.20.202.65:35459 timed out after 3.015s (SENT)
I20260809 15:46:53.384063 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00a1881267a14ba8b78c81040f17a014" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:53.384481 24406 leader_election.cc:302] T 0a51c0baaa6e432997c9d3552af4f254 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f; no voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:53.384657 23526 raft_consensus.cc:3037] T 00a1881267a14ba8b78c81040f17a014 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Advancing to term 7
I20260809 15:46:53.350616 26966 raft_consensus.cc:686] T 30c1740ef6874df0a332fbd6fd4b03ce P 65c36d164b0540f099d592f38307ee6c [term 5 LEADER]: Becoming Leader. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:46:53.388978 23518 raft_consensus.cc:2379] T 953eea268e17491ca76f3a555860a512 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d in current term 5: Already voted for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in this term.
I20260809 15:46:53.349637 27221 consensus_queue.cc:1046] T ae6d2c08a7a94f58ac8dd698e3fd456c P 65c36d164b0540f099d592f38307ee6c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.045s
I20260809 15:46:53.337944 27222 consensus_queue.cc:1035] T 7baa2492df0340c4a5ac241959ce1ac3 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.058s
I20260809 15:46:53.348424 27034 consensus_queue.cc:1046] T 4b30f28311964a0b9149a50495e83d8f P 65c36d164b0540f099d592f38307ee6c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.033s
I20260809 15:46:53.407435 23353 leader_election.cc:302] T 213543cdbe814c89a17a963469922c1b P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:53.410224 23526 raft_consensus.cc:2454] T 00a1881267a14ba8b78c81040f17a014 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 7.
I20260809 15:46:53.411516 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3be1e356b254d22b995be32f1c12f6f" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:53.411676 23228 leader_election.cc:302] T 2db057fd10e149a9895dff6ef8c9f038 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:53.412318 23521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "763e58d6ba644eea830a6c1664d39c9b" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:53.413391 27034 consensus_queue.cc:1225] T 4b30f28311964a0b9149a50495e83d8f P 65c36d164b0540f099d592f38307ee6c [LEADER]: Peer c0f38c1908414566aa9115bb2cbdd85d log is divergent from this leader: its last log entry 3.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260809 15:46:53.389804 27206 raft_consensus.cc:2726] T 0a51c0baaa6e432997c9d3552af4f254 P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260809 15:46:53.415460 27076 raft_consensus.cc:2726] T 2db057fd10e149a9895dff6ef8c9f038 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260809 15:46:53.415828 27229 consensus_queue.cc:1046] T e14d118b2eaa42aa8a7cd4025aba7128 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260809 15:46:53.415495 23520 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56770 (request call id 1319) took 1290 ms. Trace:
I20260809 15:46:53.416492 23520 rpcz_store.cc:276] 0809 15:46:52.125075 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:52.125190 (+ 115us) service_pool.cc:227] Handling call
0809 15:46:53.415352 (+1290162us) spinlock_profiling.cc:243] Waited 136 ms on lock 0x7b54000915d4. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e92242ae 00007f34ee161c27 00007f34e7849c04 00007f34e7849b92 00007f34e7849ac1 00007f34e7849a3c 00007f34e7848cf2 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:46:53.415443 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":311893376}
I20260809 15:46:53.416808 27229 raft_consensus.cc:3032] T e14d118b2eaa42aa8a7cd4025aba7128 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 LEADER]: Stepping down as leader of term 6
I20260809 15:46:53.417143 27229 raft_consensus.cc:729] T e14d118b2eaa42aa8a7cd4025aba7128 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Running, Role: LEADER
I20260809 15:46:53.417253 23270 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55434 (request call id 208) took 1338 ms. Trace:
I20260809 15:46:53.417385 27196 leader_election.cc:288] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:53.393250 26826 leader_election.cc:288] T 294d4bcc1e7a477cb56455581277a60f P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:53.418491 23282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e14d118b2eaa42aa8a7cd4025aba7128" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:53.418265 27229 consensus_queue.cc:260] T e14d118b2eaa42aa8a7cd4025aba7128 P 67b3aeadfa7a432abe8af4d292e04ab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:53.419024 23520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5712f6b6fffc4cc5a8ddfbe63e4097df" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:53.419518 24468 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38724 (request call id 488) took 1066 ms. Trace:
I20260809 15:46:53.419888 24468 rpcz_store.cc:276] 0809 15:46:52.352726 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:52.365263 (+ 12537us) service_pool.cc:227] Handling call
0809 15:46:52.398855 (+ 33592us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:52.464939 (+ 66084us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:46:52.468837 (+ 3898us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 15:46:52.474793 (+ 5956us) raft_consensus.cc:1631] Marking committed up to 7
0809 15:46:52.477570 (+ 2777us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:53.419305 (+941735us) spinlock_profiling.cc:243] Waited 56.8 ms on lock 0x7b54007116d4. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e9220f51 00007f34e921f99e 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772 00007f34e764a94c 00007f34e7649d86
0809 15:46:53.419346 (+ 41us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:53.419463 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":130718976}
I20260809 15:46:53.421525 23394 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55332 (request call id 1335) took 1379 ms. Trace:
I20260809 15:46:53.421893 23394 rpcz_store.cc:276] 0809 15:46:52.042161 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:52.083587 (+ 41426us) service_pool.cc:227] Handling call
0809 15:46:52.116880 (+ 33293us) spinlock_profiling.cc:243] Waited 2.49 ms on lock 0x7b58001d7c98. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e613cf11 00007f34e62c0851 00007f34e62c0702 00007f34ef1b0b09 00007f34ee0ec730 00007f34ee1b279b 00007f34ee1b2821 00007f34ee1621b0 00007f34ee161a94 00007f34e7849c04 00007f34e7849b92
0809 15:46:53.421351 (+1304471us) spinlock_profiling.cc:243] Waited 988 ms on lock 0x7b54002ff154. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e92242ae 00007f34ee161c27 00007f34e7849c04 00007f34e7849b92 00007f34e7849ac1 00007f34e7849a3c 00007f34e7848cf2 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:46:53.421471 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2278393984}
I20260809 15:46:53.429323 27229 raft_consensus.cc:3037] T e14d118b2eaa42aa8a7cd4025aba7128 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Advancing to term 7
W20260809 15:46:53.429575 23277 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55434 (request call id 203) took 1550 ms (client timeout 1966 ms). Trace:
W20260809 15:46:53.429947 23277 rpcz_store.cc:269] 0809 15:46:51.878983 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:52.024601 (+145618us) service_pool.cc:227] Handling call
0809 15:46:53.429437 (+1404836us) spinlock_profiling.cc:243] Waited 544 ms on lock 0x7b54002f58d4. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e92242ae 00007f34ee161c27 00007f34e7849c04 00007f34e7849b92 00007f34e7849ac1 00007f34e7849a3c 00007f34e7848cf2 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:46:53.429524 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1252341120}
I20260809 15:46:53.430962 23277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3be1e356b254d22b995be32f1c12f6f" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:53.431506 23277 raft_consensus.cc:2454] T d3be1e356b254d22b995be32f1c12f6f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 5.
I20260809 15:46:53.431905 23396 raft_consensus.cc:1261] T d1a134ba33ea4076bbaac0ae460c17ba P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 3. (term mismatch)
I20260809 15:46:53.432233 23396 pending_rounds.cc:76] T d1a134ba33ea4076bbaac0ae460c17ba P c0f38c1908414566aa9115bb2cbdd85d: Aborting all ops after (but not including) 2
I20260809 15:46:53.432453 23396 pending_rounds.cc:97] T d1a134ba33ea4076bbaac0ae460c17ba P c0f38c1908414566aa9115bb2cbdd85d: Aborting uncommitted NO_OP operation due to leader change: 3.3
I20260809 15:46:53.432680 23396 raft_consensus.cc:2866] T d1a134ba33ea4076bbaac0ae460c17ba P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 15:46:53.385501 26527 consensus_queue.cc:1035] T 4b30f28311964a0b9149a50495e83d8f P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.051s
I20260809 15:46:53.417637 23270 rpcz_store.cc:276] 0809 15:46:52.078319 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:52.084194 (+ 5875us) service_pool.cc:227] Handling call
0809 15:46:53.417060 (+1332866us) spinlock_profiling.cc:243] Waited 1.04 s on lock 0x7b540008a554. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e92242ae 00007f34ee161c27 00007f34e7849c04 00007f34e7849b92 00007f34e7849ac1 00007f34e7849a3c 00007f34e7848cf2 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:46:53.417172 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2390995200}
I20260809 15:46:53.438447 23286 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53554 (request call id 1477) took 1036 ms. Trace:
I20260809 15:46:53.438782 23286 rpcz_store.cc:276] 0809 15:46:52.401660 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:52.402312 (+ 652us) service_pool.cc:227] Handling call
0809 15:46:53.438292 (+1035980us) spinlock_profiling.cc:243] Waited 36.8 ms on lock 0x7b540030fa54. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e92242ae 00007f34ee161c27 00007f34e7849c04 00007f34e7849b92 00007f34e7849ac1 00007f34e7849a3c 00007f34e7848cf2 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:46:53.438385 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":84589440}
I20260809 15:46:53.439877 23270 raft_consensus.cc:1261] T d1a134ba33ea4076bbaac0ae460c17ba P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20260809 15:46:53.395946 27224 log.cc:924] Time spent T 0da4d5283c8a46878bd39b85e5c71f66 P c0f38c1908414566aa9115bb2cbdd85d: Append to log took a long time: real 0.053s user 0.000s sys 0.000s
I20260809 15:46:53.398960 27116 raft_consensus.cc:3032] T 73f6f44c259547f1b05a3a434616fa54 P c0f38c1908414566aa9115bb2cbdd85d [term 5 LEADER]: Stepping down as leader of term 5
I20260809 15:46:53.451920 23351 leader_election.cc:302] T d3be1e356b254d22b995be32f1c12f6f P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:46:53.390014 26719 raft_consensus.cc:2781] T 763e58d6ba644eea830a6c1664d39c9b P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260809 15:46:53.448866 23518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "294d4bcc1e7a477cb56455581277a60f" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:53.454348 23228 leader_election.cc:302] T c0741ab44d99439abc3aa3bdd27ee160 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c
I20260809 15:46:53.400204 27136 leader_election.cc:288] T d3be1e356b254d22b995be32f1c12f6f P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:53.458705 23282 raft_consensus.cc:2396] T e14d118b2eaa42aa8a7cd4025aba7128 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c for term 8 because replica has last-logged OpId of term: 6 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20260809 15:46:53.463902 23286 raft_consensus.cc:1261] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260809 15:46:53.452893 27178 raft_consensus.cc:2726] T 213543cdbe814c89a17a963469922c1b P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260809 15:46:53.480129 23274 raft_consensus.cc:1226] T 4b30f28311964a0b9149a50495e83d8f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 2. Current term is 3. Ops: []
W20260809 15:46:53.481618 23472 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 167775us
I20260809 15:46:53.453943 27116 raft_consensus.cc:729] T 73f6f44c259547f1b05a3a434616fa54 P c0f38c1908414566aa9115bb2cbdd85d [term 5 LEADER]: Becoming Follower/Learner. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:46:53.456427 27216 raft_consensus.cc:482] T c5c2e425a74b4e64ae70b5def750434c P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:53.457868 26719 raft_consensus.cc:482] T 763e58d6ba644eea830a6c1664d39c9b P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:46:53.465514 27076 raft_consensus.cc:2726] T c0741ab44d99439abc3aa3bdd27ee160 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260809 15:46:53.459131 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e14d118b2eaa42aa8a7cd4025aba7128" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
W20260809 15:46:53.523147 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 46040.8us
W20260809 15:46:53.525679 23470 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 47761.5us
I20260809 15:46:53.527540 23228 leader_election.cc:302] T 16ca412a0dae4feebe85dd01bf1a9173 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c
I20260809 15:46:53.528051 23277 raft_consensus.cc:1261] T d0721a4403a440f383318a3cea9d1442 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260809 15:46:53.453707 27201 consensus_queue.cc:1035] T d1a134ba33ea4076bbaac0ae460c17ba P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.070s
I20260809 15:46:53.487370 26527 raft_consensus.cc:2781] T d3be1e356b254d22b995be32f1c12f6f P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260809 15:46:53.533310 23345 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.68:34689, user_credentials={real_user=slave}} blocked reactor thread for 106242us
W20260809 15:46:53.533186 23353 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 83275.9us
I20260809 15:46:53.536671 23228 leader_election.cc:302] T 6ae14a74a26f434db08c652e097385c8 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
W20260809 15:46:53.533304 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 81255.2us
I20260809 15:46:53.537844 23228 leader_election.cc:302] T 953eea268e17491ca76f3a555860a512 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:46:53.524477 26719 raft_consensus.cc:3037] T 763e58d6ba644eea830a6c1664d39c9b P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Advancing to term 6
I20260809 15:46:53.537695 23397 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55320 (request call id 1351) took 1065 ms. Trace:
I20260809 15:46:53.539522 23277 raft_consensus.cc:1226] T d2d4f749fcfd4fb3bfb8115af3471598 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 4. Current term is 5. Ops: []
I20260809 15:46:53.539242 23397 rpcz_store.cc:276] 0809 15:46:52.472587 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:53.080240 (+607653us) service_pool.cc:227] Handling call
0809 15:46:53.080529 (+ 289us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:53.080712 (+ 183us) raft_consensus.cc:1520] Early marking committed up to index 4
0809 15:46:53.346975 (+266263us) log.cc:842] Serialized 14 byte log entry
0809 15:46:53.537174 (+190199us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 15:46:53.537275 (+ 101us) raft_consensus.cc:1631] Marking committed up to 4
0809 15:46:53.537342 (+ 67us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:53.537430 (+ 88us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:53.537618 (+ 188us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3696,"prepare.queue_time_us":242495,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":2886,"thread_start_us":414983,"threads_started":2}
I20260809 15:46:53.540386 23400 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55320 (request call id 1359) took 1065 ms. Trace:
I20260809 15:46:53.473016 27221 consensus_queue.cc:1035] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.058s
I20260809 15:46:53.392160 27213 consensus_queue.cc:237] T d2d4f749fcfd4fb3bfb8115af3471598 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:53.543337 23228 leader_election.cc:302] T f9e7cadcd24d45fcbfa84209225e8f2d P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:46:53.541608 23400 rpcz_store.cc:276] 0809 15:46:52.475427 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:53.112347 (+636920us) service_pool.cc:227] Handling call
0809 15:46:53.112656 (+ 309us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 15:46:53.112986 (+ 330us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:46:53.113012 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 15:46:53.113458 (+ 446us) log.cc:842] Serialized 26 byte log entry
0809 15:46:53.282820 (+169362us) raft_consensus.cc:1631] Marking committed up to 1
0809 15:46:53.282854 (+ 34us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:53.282909 (+ 55us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 15:46:53.540213 (+257304us) raft_consensus.cc:1671] finished
0809 15:46:53.540233 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:53.540336 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":4060,"raft.run_cpu_time_us":128,"raft.run_wall_time_us":4840,"thread_start_us":169021,"threads_started":1,"wal-append.queue_time_us":209897}
I20260809 15:46:53.547894 23353 leader_election.cc:302] T 953eea268e17491ca76f3a555860a512 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
W20260809 15:46:53.552896 27236 log.cc:924] Time spent T 99bdb93503db4de4a1cfb0161ea0c07c P 67b3aeadfa7a432abe8af4d292e04ab9: Append to log took a long time: real 0.080s user 0.000s sys 0.000s
I20260809 15:46:53.553013 23393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3296ef50be254fc29cfc77d22a8f7a23" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:53.553831 27200 raft_consensus.cc:3032] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 61e69695258f48baa984c9d395e8ce3f [term 6 LEADER]: Stepping down as leader of term 6
I20260809 15:46:53.554194 27200 raft_consensus.cc:729] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 61e69695258f48baa984c9d395e8ce3f [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 61e69695258f48baa984c9d395e8ce3f, State: Running, Role: LEADER
I20260809 15:46:53.555205 23518 raft_consensus.cc:1226] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Rejecting Update request from peer 61e69695258f48baa984c9d395e8ce3f for earlier term 6. Current term is 7. Ops: []
W20260809 15:46:53.555718 27246 log.cc:924] Time spent T 0da4d5283c8a46878bd39b85e5c71f66 P 65c36d164b0540f099d592f38307ee6c: Append to log took a long time: real 0.101s user 0.000s sys 0.000s
I20260809 15:46:53.556111 26730 leader_election.cc:288] T 213543cdbe814c89a17a963469922c1b P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:53.561291 23520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73f6f44c259547f1b05a3a434616fa54" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:53.565088 23353 leader_election.cc:302] T 00a1881267a14ba8b78c81040f17a014 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:46:53.536485 27162 raft_consensus.cc:2726] T 16ca412a0dae4feebe85dd01bf1a9173 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260809 15:46:53.536731 26527 raft_consensus.cc:482] T d3be1e356b254d22b995be32f1c12f6f P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:46:53.554960 27200 consensus_queue.cc:260] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 61e69695258f48baa984c9d395e8ce3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:53.568289 26747 raft_consensus.cc:2726] T 953eea268e17491ca76f3a555860a512 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260809 15:46:53.548238 27229 raft_consensus.cc:2781] T 953eea268e17491ca76f3a555860a512 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Leader election won for term 5
I20260809 15:46:53.578119 23521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b30f28311964a0b9149a50495e83d8f" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:53.578715 23396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b30f28311964a0b9149a50495e83d8f" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:53.578194 23518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5c2e425a74b4e64ae70b5def750434c" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
W20260809 15:46:53.579836 23472 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 97698.2us
I20260809 15:46:53.580219 23518 raft_consensus.cc:2379] T c5c2e425a74b4e64ae70b5def750434c P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 in current term 5: Already voted for candidate 65c36d164b0540f099d592f38307ee6c in this term.
I20260809 15:46:53.542668 27226 raft_consensus.cc:2781] T 6ae14a74a26f434db08c652e097385c8 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Leader election won for term 5
I20260809 15:46:53.547082 27011 raft_consensus.cc:482] T 28e17dcb5efe45b8b6a8c42da03b4819 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:46:53.580139 23400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73f6f44c259547f1b05a3a434616fa54" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:53.584092 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1a134ba33ea4076bbaac0ae460c17ba" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:53.584321 23228 leader_election.cc:302] T 998c6cee72cf40228c6dba064c0db2da P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:53.476543 26966 consensus_queue.cc:237] T 30c1740ef6874df0a332fbd6fd4b03ce P 65c36d164b0540f099d592f38307ee6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:53.531698 27219 consensus_queue.cc:1035] T 05ad7d8fdca64e3da527280ca5be9280 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.049s
I20260809 15:46:53.549588 27053 raft_consensus.cc:482] T 73f6f44c259547f1b05a3a434616fa54 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:53.592862 23521 raft_consensus.cc:1226] T 4564dfc6d4094c1b9240b367ccecf3b1 P 65c36d164b0540f099d592f38307ee6c [term 5 LEADER]: Rejecting Update request from peer 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260809 15:46:53.568595 26527 raft_consensus.cc:3037] T d3be1e356b254d22b995be32f1c12f6f P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Advancing to term 6
I20260809 15:46:53.549841 27076 raft_consensus.cc:2781] T f9e7cadcd24d45fcbfa84209225e8f2d P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Leader election won for term 5
I20260809 15:46:53.593530 23394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05ad7d8fdca64e3da527280ca5be9280" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:53.594286 23394 raft_consensus.cc:2454] T 05ad7d8fdca64e3da527280ca5be9280 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 5.
I20260809 15:46:53.592154 23518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05ad7d8fdca64e3da527280ca5be9280" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:53.598591 24475 raft_consensus.cc:2454] T e71a05ba1d4d42bd95c3f85f9313399e P 61e69695258f48baa984c9d395e8ce3f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 6.
I20260809 15:46:53.524374 27216 raft_consensus.cc:504] T c5c2e425a74b4e64ae70b5def750434c P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:46:53.568225 24452 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38724 (request call id 489) took 1213 ms (client timeout 1502 ms). Trace:
W20260809 15:46:53.604846 23193 auto_rebalancer.cc:254] failed to send replica move request: Illegal state: Replica 67b3aeadfa7a432abe8af4d292e04ab9 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 5 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } }
I20260809 15:46:53.605176 23396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e71a05ba1d4d42bd95c3f85f9313399e" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:53.605716 23396 raft_consensus.cc:3037] T e71a05ba1d4d42bd95c3f85f9313399e P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Advancing to term 6
I20260809 15:46:53.602981 23394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1a134ba33ea4076bbaac0ae460c17ba" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:53.606784 23394 raft_consensus.cc:2454] T d1a134ba33ea4076bbaac0ae460c17ba P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 4.
I20260809 15:46:53.589416 27011 raft_consensus.cc:504] T 28e17dcb5efe45b8b6a8c42da03b4819 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:46:53.610745 27254 log.cc:924] Time spent T 4564dfc6d4094c1b9240b367ccecf3b1 P 65c36d164b0540f099d592f38307ee6c: Append to log took a long time: real 0.068s user 0.000s sys 0.000s
I20260809 15:46:53.611085 27011 leader_election.cc:288] T 28e17dcb5efe45b8b6a8c42da03b4819 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:53.611586 27206 consensus_queue.cc:1035] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.006s
I20260809 15:46:53.610652 27216 leader_election.cc:288] T c5c2e425a74b4e64ae70b5def750434c P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:53.588075 24463 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38734 (request call id 304) took 1120 ms. Trace:
I20260809 15:46:53.630474 23276 raft_consensus.cc:2454] T 00a1881267a14ba8b78c81040f17a014 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 7.
I20260809 15:46:53.600822 26772 raft_consensus.cc:2781] T 00a1881267a14ba8b78c81040f17a014 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Leader election won for term 7
I20260809 15:46:53.584827 26826 consensus_queue.cc:1046] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 61e69695258f48baa984c9d395e8ce3f [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.029s
I20260809 15:46:53.642846 23276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5c2e425a74b4e64ae70b5def750434c" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
W20260809 15:46:53.641685 24406 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 59312.7us
I20260809 15:46:53.602780 23393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5c2e425a74b4e64ae70b5def750434c" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:53.642421 23521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5c2e425a74b4e64ae70b5def750434c" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:53.644699 23521 raft_consensus.cc:2454] T c5c2e425a74b4e64ae70b5def750434c P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 5.
I20260809 15:46:53.644106 24406 leader_election.cc:302] T e14d118b2eaa42aa8a7cd4025aba7128 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:53.643612 23276 raft_consensus.cc:2454] T c5c2e425a74b4e64ae70b5def750434c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 4.
I20260809 15:46:53.645763 23353 leader_election.cc:302] T c5c2e425a74b4e64ae70b5def750434c P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:46:53.620169 27076 raft_consensus.cc:686] T f9e7cadcd24d45fcbfa84209225e8f2d P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 LEADER]: Becoming Leader. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Running, Role: LEADER
I20260809 15:46:53.557837 27116 consensus_queue.cc:260] T 73f6f44c259547f1b05a3a434616fa54 P c0f38c1908414566aa9115bb2cbdd85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:53.585500 27263 consensus_queue.cc:1035] T 99bdb93503db4de4a1cfb0161ea0c07c P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.074s
I20260809 15:46:53.601090 27053 raft_consensus.cc:504] T 73f6f44c259547f1b05a3a434616fa54 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:46:53.659003 27234 log.cc:924] Time spent T cef3e7277d7d4760824a3436ed4a847d P c0f38c1908414566aa9115bb2cbdd85d: Append to log took a long time: real 0.130s user 0.000s sys 0.001s
W20260809 15:46:53.660162 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 75914.1us
I20260809 15:46:53.677435 23228 leader_election.cc:302] T a88421dba80d4d3388aa5eb62e23853b P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c
W20260809 15:46:53.683490 23472 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 97956.9us
I20260809 15:46:53.646624 26772 raft_consensus.cc:686] T 00a1881267a14ba8b78c81040f17a014 P c0f38c1908414566aa9115bb2cbdd85d [term 7 LEADER]: Becoming Leader. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:46:53.654215 27196 raft_consensus.cc:3037] T e14d118b2eaa42aa8a7cd4025aba7128 P 61e69695258f48baa984c9d395e8ce3f [term 6 FOLLOWER]: Advancing to term 7
I20260809 15:46:53.771359 23396 raft_consensus.cc:2454] T e71a05ba1d4d42bd95c3f85f9313399e P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 6.
W20260809 15:46:53.732173 24406 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 74897.9us
W20260809 15:46:53.604532 24452 rpcz_store.cc:269] 0809 15:46:52.353059 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:52.365307 (+ 12248us) service_pool.cc:227] Handling call
0809 15:46:53.557228 (+1191921us) spinlock_profiling.cc:243] Waited 177 ms on lock 0x7b54005334d4. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e92242ae 00007f34ee161c27 00007f34e7849c04 00007f34e7849b92 00007f34e7849ac1 00007f34e7849a3c 00007f34e7848cf2 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:46:53.566046 (+ 8818us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":407945728}
W20260809 15:46:53.819801 23470 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 238690us
I20260809 15:46:53.816768 23520 raft_consensus.cc:2379] T 73f6f44c259547f1b05a3a434616fa54 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 in current term 6: Already voted for candidate 65c36d164b0540f099d592f38307ee6c in this term.
W20260809 15:46:53.818897 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 141529us
I20260809 15:46:53.613643 27268 consensus_queue.cc:1046] T 4b30f28311964a0b9149a50495e83d8f P 65c36d164b0540f099d592f38307ee6c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.071s
I20260809 15:46:53.653666 23393 raft_consensus.cc:2454] T c5c2e425a74b4e64ae70b5def750434c P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 5.
W20260809 15:46:53.772436 27261 log.cc:924] Time spent T 297c82ed906d4b48b184985eed3de601 P c0f38c1908414566aa9115bb2cbdd85d: Append to log took a long time: real 0.086s user 0.000s sys 0.001s
I20260809 15:46:53.824134 24406 leader_election.cc:302] T 3d880de657e544468c82157769622a26 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:53.839200 23228 leader_election.cc:302] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:53.817590 27213 raft_consensus.cc:482] T ae6d2c08a7a94f58ac8dd698e3fd456c P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:53.856626 23228 leader_election.cc:302] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d
W20260809 15:46:53.832687 27238 log.cc:924] Time spent T 7baa2492df0340c4a5ac241959ce1ac3 P 67b3aeadfa7a432abe8af4d292e04ab9: Append to log took a long time: real 0.161s user 0.000s sys 0.000s
I20260809 15:46:53.837936 27274 raft_consensus.cc:2726] T 998c6cee72cf40228c6dba064c0db2da P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260809 15:46:53.858150 23228 leader_election.cc:302] T a3622f6e3e3641cea1960e3a31498bb7 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c
I20260809 15:46:53.859205 23228 leader_election.cc:302] T df20671a7b874dbb98c0ef28371c1308 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c
I20260809 15:46:53.860155 23228 leader_election.cc:302] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:53.856309 27213 raft_consensus.cc:504] T ae6d2c08a7a94f58ac8dd698e3fd456c P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:53.863790 23402 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55320 (request call id 1349) took 1392 ms. Trace:
I20260809 15:46:53.865052 23394 raft_consensus.cc:1261] T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Refusing update from remote peer 61e69695258f48baa984c9d395e8ce3f: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260809 15:46:53.869935 23402 rpcz_store.cc:276] 0809 15:46:52.471594 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:53.077895 (+606301us) service_pool.cc:227] Handling call
0809 15:46:53.078184 (+ 289us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 15:46:53.124438 (+ 46254us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:46:53.124466 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 15:46:53.124825 (+ 359us) log.cc:842] Serialized 26 byte log entry
0809 15:46:53.340940 (+216115us) raft_consensus.cc:1631] Marking committed up to 2
0809 15:46:53.340973 (+ 33us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:53.341033 (+ 60us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 15:46:53.863560 (+522527us) raft_consensus.cc:1671] finished
0809 15:46:53.863582 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:53.863727 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":215881,"threads_started":1,"wal-append.queue_time_us":343385}
W20260809 15:46:53.876601 24406 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 52539.9us
I20260809 15:46:53.635775 24463 rpcz_store.cc:276] 0809 15:46:52.467333 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:52.467465 (+ 132us) service_pool.cc:227] Handling call
0809 15:46:53.524247 (+1056782us) spinlock_profiling.cc:243] Waited 2.83 ms on lock 0x7b5800005498. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e613cf11 00007f34e62c0851 00007f34e62c0702 00007f34ef1b0b09 00007f34ee0ec730 00007f34ee1b279b 00007f34ee1b2821 00007f34ee1762b0 00007f34ee1617f9 00007f34ee1612a2 00007f34e7847684
0809 15:46:53.531613 (+ 7366us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:53.544183 (+ 12570us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:46:53.547286 (+ 3103us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 15:46:53.557722 (+ 10436us) raft_consensus.cc:1631] Marking committed up to 6
0809 15:46:53.566943 (+ 9221us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:53.575053 (+ 8110us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:53.585299 (+ 10246us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6500224}
I20260809 15:46:53.893052 23223 leader_election.cc:302] T e71a05ba1d4d42bd95c3f85f9313399e P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:46:53.878363 27266 raft_consensus.cc:2781] T c5c2e425a74b4e64ae70b5def750434c P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260809 15:46:53.888480 27268 raft_consensus.cc:482] T 3d880de657e544468c82157769622a26 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:53.887856 26772 consensus_queue.cc:237] T 00a1881267a14ba8b78c81040f17a014 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:53.891476 27283 raft_consensus.cc:3037] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Advancing to term 5
W20260809 15:46:53.904686 26996 consensus_meta.cc:287] T d3be1e356b254d22b995be32f1c12f6f P 61e69695258f48baa984c9d395e8ce3f: Time spent flushing consensus metadata: real 0.758s user 0.009s sys 0.001s
I20260809 15:46:53.886925 27278 raft_consensus.cc:2781] T a88421dba80d4d3388aa5eb62e23853b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260809 15:46:53.890192 27055 raft_consensus.cc:3037] T 3d880de657e544468c82157769622a26 P 61e69695258f48baa984c9d395e8ce3f [term 1 FOLLOWER]: Advancing to term 2
I20260809 15:46:53.857662 27076 consensus_queue.cc:237] T f9e7cadcd24d45fcbfa84209225e8f2d P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:53.905385 24472 raft_consensus.cc:2454] T d3be1e356b254d22b995be32f1c12f6f P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 5.
W20260809 15:46:53.899071 27262 log.cc:924] Time spent T 16ca412a0dae4feebe85dd01bf1a9173 P 67b3aeadfa7a432abe8af4d292e04ab9: Append to log took a long time: real 0.311s user 0.001s sys 0.000s
W20260809 15:46:53.914358 23345 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.68:34689, user_credentials={real_user=slave}} blocked reactor thread for 48569.3us
I20260809 15:46:53.891019 27229 raft_consensus.cc:686] T 953eea268e17491ca76f3a555860a512 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 LEADER]: Becoming Leader. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Running, Role: LEADER
I20260809 15:46:53.894505 27280 raft_consensus.cc:2726] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260809 15:46:53.932017 27288 log.cc:924] Time spent T d2d4f749fcfd4fb3bfb8115af3471598 P c0f38c1908414566aa9115bb2cbdd85d: Append to log took a long time: real 0.100s user 0.000s sys 0.000s
W20260809 15:46:53.934571 27269 log.cc:924] Time spent T cef3e7277d7d4760824a3436ed4a847d P 67b3aeadfa7a432abe8af4d292e04ab9: Append to log took a long time: real 0.162s user 0.001s sys 0.000s
W20260809 15:46:53.935735 27253 log.cc:924] Time spent T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 61e69695258f48baa984c9d395e8ce3f: Append to log took a long time: real 0.322s user 0.000s sys 0.001s
I20260809 15:46:53.938314 27213 leader_election.cc:288] T ae6d2c08a7a94f58ac8dd698e3fd456c P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:53.938860 27213 raft_consensus.cc:482] T 2424fd9bd7f14e508fd106cbfda786a4 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:53.939286 27213 raft_consensus.cc:504] T 2424fd9bd7f14e508fd106cbfda786a4 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:53.916878 27081 consensus_queue.cc:1035] T d0721a4403a440f383318a3cea9d1442 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.025s
I20260809 15:46:53.893855 27097 consensus_queue.cc:1035] T 3924b648461e463b9d7925e5671d68ae P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.019s
I20260809 15:46:53.941509 23270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae6d2c08a7a94f58ac8dd698e3fd456c" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:53.910454 27278 raft_consensus.cc:482] T a88421dba80d4d3388aa5eb62e23853b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:53.971985 23521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae6d2c08a7a94f58ac8dd698e3fd456c" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:53.959914 27308 raft_consensus.cc:2726] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260809 15:46:53.913560 27011 consensus_queue.cc:1046] T d2d4f749fcfd4fb3bfb8115af3471598 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.042s
W20260809 15:46:54.033371 23472 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 209949us
I20260809 15:46:54.025372 23353 leader_election.cc:302] T ae6d2c08a7a94f58ac8dd698e3fd456c P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
W20260809 15:46:54.026242 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 166129us
I20260809 15:46:53.910849 27268 raft_consensus.cc:504] T 3d880de657e544468c82157769622a26 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:53.956090 27280 raft_consensus.cc:2781] T df20671a7b874dbb98c0ef28371c1308 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260809 15:46:54.030833 27226 raft_consensus.cc:686] T 6ae14a74a26f434db08c652e097385c8 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 LEADER]: Becoming Leader. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Running, Role: LEADER
I20260809 15:46:53.951287 27295 raft_consensus.cc:2781] T a3622f6e3e3641cea1960e3a31498bb7 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260809 15:46:53.951159 27053 leader_election.cc:288] T 73f6f44c259547f1b05a3a434616fa54 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:53.956733 27278 raft_consensus.cc:3037] T a88421dba80d4d3388aa5eb62e23853b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:46:54.034904 23228 leader_election.cc:302] T ba56725e42f74a1baa4c04d9660727f9 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c
I20260809 15:46:55.023759 23392 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55320 (request call id 1367) took 1941 ms. Trace:
I20260809 15:46:55.024333 23392 rpcz_store.cc:276] 0809 15:46:53.082137 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:53.166661 (+ 84524us) service_pool.cc:227] Handling call
0809 15:46:53.203610 (+ 36949us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:53.360677 (+157067us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:55.023447 (+1662770us) spinlock_profiling.cc:243] Waited 82 ms on lock 0x7b54001cf2d4. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e9220f51 00007f34e921f99e 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772 00007f34e764a94c 00007f34e7649d86
0809 15:46:55.023689 (+ 242us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":188539776}
W20260809 15:46:54.036785 24410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 62261.2us
I20260809 15:46:54.037590 27280 raft_consensus.cc:482] T df20671a7b874dbb98c0ef28371c1308 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:54.037465 27292 raft_consensus.cc:482] T 7baa2492df0340c4a5ac241959ce1ac3 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:55.031086 27280 raft_consensus.cc:3037] T df20671a7b874dbb98c0ef28371c1308 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Advancing to term 5
I20260809 15:46:55.031426 27075 raft_consensus.cc:2781] T ba56725e42f74a1baa4c04d9660727f9 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260809 15:46:55.031216 27292 raft_consensus.cc:504] T 7baa2492df0340c4a5ac241959ce1ac3 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:55.031970 27075 raft_consensus.cc:482] T ba56725e42f74a1baa4c04d9660727f9 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:55.032423 27075 raft_consensus.cc:3037] T ba56725e42f74a1baa4c04d9660727f9 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:46:55.033923 27292 leader_election.cc:288] T 7baa2492df0340c4a5ac241959ce1ac3 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:54.019276 27197 raft_consensus.cc:2726] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260809 15:46:54.096645 27294 consensus_queue.cc:1046] T 0acc47ce21c3422698d1ef7277c3ed63 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: INVALID_TERM, Last received: 5.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.027s
I20260809 15:46:54.100035 27268 leader_election.cc:288] T 3d880de657e544468c82157769622a26 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:55.061406 27268 raft_consensus.cc:3032] T 0acc47ce21c3422698d1ef7277c3ed63 P 65c36d164b0540f099d592f38307ee6c [term 5 LEADER]: Stepping down as leader of term 5
I20260809 15:46:55.061780 27268 raft_consensus.cc:729] T 0acc47ce21c3422698d1ef7277c3ed63 P 65c36d164b0540f099d592f38307ee6c [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:46:54.062932 27308 raft_consensus.cc:2781] T e71a05ba1d4d42bd95c3f85f9313399e P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Leader election won for term 6
I20260809 15:46:55.064327 27308 raft_consensus.cc:686] T e71a05ba1d4d42bd95c3f85f9313399e P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 LEADER]: Becoming Leader. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Running, Role: LEADER
I20260809 15:46:55.065383 27308 consensus_queue.cc:237] T e71a05ba1d4d42bd95c3f85f9313399e P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
W20260809 15:46:54.115530 23470 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 231099us
I20260809 15:46:54.062594 27295 raft_consensus.cc:482] T a3622f6e3e3641cea1960e3a31498bb7 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:55.071743 27295 raft_consensus.cc:3037] T a3622f6e3e3641cea1960e3a31498bb7 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:46:55.075567 27280 raft_consensus.cc:504] T df20671a7b874dbb98c0ef28371c1308 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:55.078522 27280 leader_election.cc:288] T df20671a7b874dbb98c0ef28371c1308 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
W20260809 15:46:54.123236 23395 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55332 (request call id 1338) took 2007 ms (client timeout 1914 ms). Trace:
I20260809 15:46:54.129017 27283 raft_consensus.cc:2726] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260809 15:46:54.131132 27316 raft_consensus.cc:2726] T ae6d2c08a7a94f58ac8dd698e3fd456c P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260809 15:46:54.256841 24406 leader_election.cc:334] T cee02fff2c9c48a19f43eb358a89e7b0 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459): Timed out: RequestConsensusVote RPC to 127.20.202.65:35459 timed out after 1.780s (SENT)
I20260809 15:46:55.094920 24406 leader_election.cc:302] T cee02fff2c9c48a19f43eb358a89e7b0 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:54.287339 27322 raft_consensus.cc:482] T 6ae14a74a26f434db08c652e097385c8 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:54.366283 27213 leader_election.cc:288] T 2424fd9bd7f14e508fd106cbfda786a4 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
W20260809 15:46:54.422326 27196 consensus_meta.cc:287] T e14d118b2eaa42aa8a7cd4025aba7128 P 61e69695258f48baa984c9d395e8ce3f: Time spent flushing consensus metadata: real 0.605s user 0.002s sys 0.006s
W20260809 15:46:54.477703 23353 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 452400us
I20260809 15:46:54.595409 27313 raft_consensus.cc:482] T cee02fff2c9c48a19f43eb358a89e7b0 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:54.634106 27079 raft_consensus.cc:482] T 0c48a3049897452383257aa757834fb5 P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:54.663663 27278 raft_consensus.cc:504] T a88421dba80d4d3388aa5eb62e23853b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:46:54.674919 26772 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.754s user 0.004s sys 0.008s
W20260809 15:46:54.679780 27055 consensus_meta.cc:287] T 3d880de657e544468c82157769622a26 P 61e69695258f48baa984c9d395e8ce3f: Time spent flushing consensus metadata: real 0.767s user 0.006s sys 0.015s
I20260809 15:46:54.680052 27279 consensus_queue.cc:1046] T d2d4f749fcfd4fb3bfb8115af3471598 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20260809 15:46:54.680511 23351 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.626s user 0.007s sys 0.000s
I20260809 15:46:54.693638 27226 consensus_queue.cc:237] T 6ae14a74a26f434db08c652e097385c8 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:46:54.697486 27238 log.cc:924] Time spent T 7baa2492df0340c4a5ac241959ce1ac3 P 67b3aeadfa7a432abe8af4d292e04ab9: Append to log took a long time: real 0.643s user 0.000s sys 0.003s
I20260809 15:46:54.708952 23274 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53554 (request call id 1503) took 1334 ms. Trace:
I20260809 15:46:54.731782 27217 raft_consensus.cc:482] T e14d118b2eaa42aa8a7cd4025aba7128 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 15:46:54.738426 26719 consensus_meta.cc:287] T 763e58d6ba644eea830a6c1664d39c9b P c0f38c1908414566aa9115bb2cbdd85d: Time spent flushing consensus metadata: real 1.188s user 0.015s sys 0.006s
W20260809 15:46:54.748296 26527 consensus_meta.cc:287] T d3be1e356b254d22b995be32f1c12f6f P c0f38c1908414566aa9115bb2cbdd85d: Time spent flushing consensus metadata: real 1.143s user 0.015s sys 0.005s
I20260809 15:46:54.754492 23517 raft_consensus.cc:3032] T 4b30f28311964a0b9149a50495e83d8f P 65c36d164b0540f099d592f38307ee6c [term 2 LEADER]: Stepping down as leader of term 2
I20260809 15:46:54.759967 27274 raft_consensus.cc:482] T cee02fff2c9c48a19f43eb358a89e7b0 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 15:46:54.769708 23345 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.741s user 0.012s sys 0.008s
I20260809 15:46:54.816196 27034 raft_consensus.cc:482] T f9e7cadcd24d45fcbfa84209225e8f2d P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:54.827190 27072 raft_consensus.cc:482] T 99bdb93503db4de4a1cfb0161ea0c07c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:46:54.941816 27053 raft_consensus.cc:482] T 3924b648461e463b9d7925e5671d68ae P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:54.946921 27134 raft_consensus.cc:482] T 538a228ddfb54bc6a36528d5ff07597b P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:54.958425 27076 raft_consensus.cc:482] T 2db057fd10e149a9895dff6ef8c9f038 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:54.976318 23404 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55320 (request call id 1365) took 1900 ms. Trace:
I20260809 15:46:54.981961 27336 consensus_queue.cc:1046] T 2424fd9bd7f14e508fd106cbfda786a4 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: INVALID_TERM, Last received: 5.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 15:46:54.982829 26810 raft_consensus.cc:482] T 0acc47ce21c3422698d1ef7277c3ed63 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:54.983748 27334 raft_consensus.cc:482] T d3e94b7e50754a81b3d72a07476cb7b9 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
I20260809 15:46:54.985190 23472 leader_election.cc:398] T 00a1881267a14ba8b78c81040f17a014 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 6 election: Vote denied by peer c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263) with higher term. Message: Invalid argument: T 00a1881267a14ba8b78c81040f17a014 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c for earlier term 6. Current term is 7.
I20260809 15:46:54.985561 27337 raft_consensus.cc:482] T 953eea268e17491ca76f3a555860a512 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:54.987831 27338 consensus_queue.cc:1035] T f7715d71dbac447b9351fc5eb36c5e80 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20260809 15:46:54.992144 27174 leader_election.cc:288] T d1a134ba33ea4076bbaac0ae460c17ba P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:54.998790 27339 raft_consensus.cc:482] T 294d4bcc1e7a477cb56455581277a60f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:54.068814 27229 consensus_queue.cc:237] T 953eea268e17491ca76f3a555860a512 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:55.007238 24455 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38722 (request call id 464) took 1578 ms. Trace:
I20260809 15:46:55.010767 27298 raft_consensus.cc:482] T 907bacfa79d04a55a68c8dbe60056462 P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:55.013315 27266 raft_consensus.cc:482] T c5c2e425a74b4e64ae70b5def750434c P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:46:55.015432 24451 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38724 (request call id 492) took 2501 ms. Trace:
W20260809 15:46:55.049885 27208 log.cc:924] Time spent T 05ad7d8fdca64e3da527280ca5be9280 P 67b3aeadfa7a432abe8af4d292e04ab9: Append to log took a long time: real 1.854s user 0.007s sys 0.013s
I20260809 15:46:55.040889 27194 raft_consensus.cc:482] T 563daf42d7784db195ea99c3cb21b370 P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
I20260809 15:46:55.040263 27162 raft_consensus.cc:482] T c0741ab44d99439abc3aa3bdd27ee160 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
W20260809 15:46:55.055153 24472 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38724 (request call id 493) took 1649 ms (client timeout 1662 ms). Trace:
I20260809 15:46:55.057821 23526 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56766 (request call id 1903) took 1466 ms. Trace:
W20260809 15:46:55.082713 23395 rpcz_store.cc:269] 0809 15:46:52.115883 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:52.289409 (+173526us) service_pool.cc:227] Handling call
0809 15:46:52.377355 (+ 87946us) spinlock_profiling.cc:243] Waited 997 us on lock 0x7b58001d7c98. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e613cf11 00007f34e62c0851 00007f34e62c0702 00007f34ef1b0b09 00007f34ee0ec730 00007f34ee1b279b 00007f34ee1b2821 00007f34ee1621b0 00007f34ee161a94 00007f34e7849c04 00007f34e7849b92
0809 15:46:54.123040 (+1745685us) spinlock_profiling.cc:243] Waited 89.5 ms on lock 0x7b54000a91d4. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e92242ae 00007f34ee161c27 00007f34e7849c04 00007f34e7849b92 00007f34e7849ac1 00007f34e7849a3c 00007f34e7848cf2 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:46:54.123163 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":208046208}
I20260809 15:46:55.092710 23270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73f6f44c259547f1b05a3a434616fa54" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:55.097817 23286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d880de657e544468c82157769622a26" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:55.099611 27097 raft_consensus.cc:2726] T cee02fff2c9c48a19f43eb358a89e7b0 P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20260809 15:46:55.098794 24406 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 841935us
I20260809 15:46:55.102262 23399 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55320 (request call id 1352) took 2629 ms. Trace:
I20260809 15:46:55.088090 27075 raft_consensus.cc:504] T ba56725e42f74a1baa4c04d9660727f9 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:53.468127 21289 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20260809 15:46:55.410332 23282 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53554 (request call id 1504) took 2032 ms (client timeout 1646 ms). Trace:
I20260809 15:46:55.107762 27196 raft_consensus.cc:2726] T e14d118b2eaa42aa8a7cd4025aba7128 P 61e69695258f48baa984c9d395e8ce3f [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260809 15:46:55.224493 23470 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 122390us
W20260809 15:46:55.422060 24472 rpcz_store.cc:269] 0809 15:46:53.405725 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:53.405863 (+ 138us) service_pool.cc:227] Handling call
0809 15:46:55.054998 (+1649135us) spinlock_profiling.cc:243] Waited 493 ms on lock 0x7b54004bd7d4. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e92242ae 00007f34ee161c27 00007f34e7849c04 00007f34e7849b92 00007f34e7849ac1 00007f34e7849a3c 00007f34e7848cf2 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:46:55.055100 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1134361344}
W20260809 15:46:55.423861 23282 rpcz_store.cc:269] 0809 15:46:53.378069 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:53.418274 (+ 40205us) service_pool.cc:227] Handling call
0809 15:46:55.410119 (+1991845us) spinlock_profiling.cc:243] Waited 39.4 ms on lock 0x7b54000bf254. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e92242ae 00007f34ee161c27 00007f34e7849c04 00007f34e7849b92 00007f34e7849ac1 00007f34e7849a3c 00007f34e7848cf2 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:46:55.410259 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":90557184}
I20260809 15:46:55.103785 27322 raft_consensus.cc:504] T 6ae14a74a26f434db08c652e097385c8 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:55.425727 23399 rpcz_store.cc:276] 0809 15:46:52.472955 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:53.081375 (+608420us) service_pool.cc:227] Handling call
0809 15:46:53.100343 (+ 18968us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:53.100518 (+ 175us) raft_consensus.cc:1520] Early marking committed up to index 5
0809 15:46:53.246652 (+146134us) log.cc:842] Serialized 14 byte log entry
0809 15:46:55.012106 (+1765454us) spinlock_profiling.cc:243] Waited 2.5 ms on lock 0x7b1800316d80. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34ef1978b1 00007f34ef1a4661 00007f34e626f4ca 00007f34ef950f99 00007f34e6133990 00007f34e62f5864 00007f34e62f5155 00007f34ecbf5167 00007f34e922a807 00007f34e9239f0f 00007f34e9239e88
0809 15:46:55.012287 (+ 181us) log.cc:842] Serialized 14 byte log entry
0809 15:46:55.100914 (+ 88627us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 15:46:55.100961 (+ 47us) raft_consensus.cc:1631] Marking committed up to 5
0809 15:46:55.101000 (+ 39us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:55.101261 (+ 261us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:55.102210 (+ 949us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":29978,"raft.queue_time_us":12961,"raft.run_cpu_time_us":248,"raft.run_wall_time_us":2602,"spinlock_wait_cycles":5751168,"thread_start_us":270263,"threads_started":3,"wal-append.queue_time_us":228827,"wal-append.run_cpu_time_us":2034,"wal-append.run_wall_time_us":1018291}
I20260809 15:46:55.428310 27217 raft_consensus.cc:504] T e14d118b2eaa42aa8a7cd4025aba7128 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:55.109931 23280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e17dcb5efe45b8b6a8c42da03b4819" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:55.141156 27197 raft_consensus.cc:482] T 7660dfa2ef694927b8eda4100a2bcd9b P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:46:55.430305 27217 leader_election.cc:288] T e14d118b2eaa42aa8a7cd4025aba7128 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:55.429965 27197 raft_consensus.cc:504] T 7660dfa2ef694927b8eda4100a2bcd9b P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
W20260809 15:46:55.303895 23520 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56770 (request call id 1323) took 1743 ms (client timeout 1705 ms). Trace:
W20260809 15:46:55.431353 23520 rpcz_store.cc:269] 0809 15:46:53.560912 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:53.561051 (+ 139us) service_pool.cc:227] Handling call
0809 15:46:55.303687 (+1742636us) spinlock_profiling.cc:243] Waited 221 ms on lock 0x7b54000f82d4. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e92242ae 00007f34ee161c27 00007f34e7849c04 00007f34e7849b92 00007f34e7849ac1 00007f34e7849a3c 00007f34e7848cf2 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:46:55.303823 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":507201792}
I20260809 15:46:55.433229 27197 leader_election.cc:288] T 7660dfa2ef694927b8eda4100a2bcd9b P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:55.186440 23522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2424fd9bd7f14e508fd106cbfda786a4" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 7 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:55.434370 27200 raft_consensus.cc:3037] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 61e69695258f48baa984c9d395e8ce3f [term 6 FOLLOWER]: Advancing to term 7
I20260809 15:46:55.146163 23276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7baa2492df0340c4a5ac241959ce1ac3" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:55.435483 23276 raft_consensus.cc:2396] T 7baa2492df0340c4a5ac241959ce1ac3 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260809 15:46:55.110935 27079 raft_consensus.cc:504] T 0c48a3049897452383257aa757834fb5 P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:55.422961 27055 raft_consensus.cc:2726] T 3d880de657e544468c82157769622a26 P 61e69695258f48baa984c9d395e8ce3f [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 15:46:55.442044 27244 raft_consensus.cc:482] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:55.441979 27334 raft_consensus.cc:504] T d3e94b7e50754a81b3d72a07476cb7b9 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:55.442898 27244 raft_consensus.cc:504] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:55.443647 26719 raft_consensus.cc:504] T 763e58d6ba644eea830a6c1664d39c9b P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:55.446281 26719 leader_election.cc:288] T 763e58d6ba644eea830a6c1664d39c9b P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 6 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:55.447738 23472 leader_election.cc:401] T 00a1881267a14ba8b78c81040f17a014 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 6 election: Cancelling election due to peer responding with higher term
I20260809 15:46:55.448680 23517 raft_consensus.cc:729] T 4b30f28311964a0b9149a50495e83d8f P 65c36d164b0540f099d592f38307ee6c [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
W20260809 15:46:55.348155 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 1.31322e+06us
I20260809 15:46:55.446358 27244 leader_election.cc:288] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:55.422699 23274 rpcz_store.cc:276] 0809 15:46:53.374430 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:53.480866 (+106436us) service_pool.cc:227] Handling call
0809 15:46:53.481127 (+ 261us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:53.527254 (+ 46127us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:46:53.527299 (+ 45us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 15:46:53.527335 (+ 36us) raft_consensus.cc:1631] Marking committed up to 6
0809 15:46:53.527368 (+ 33us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:54.708713 (+1181345us) spinlock_profiling.cc:243] Waited 46 ms on lock 0x7b5400366354. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e9220f51 00007f34e921f99e 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772 00007f34e764a94c 00007f34e7649d86
0809 15:46:54.708758 (+ 45us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:54.708879 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":105704704}
I20260809 15:46:55.422731 23270 raft_consensus.cc:2454] T 73f6f44c259547f1b05a3a434616fa54 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 5.
W20260809 15:46:55.425148 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 1.38912e+06us
W20260809 15:46:55.455447 23345 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.426s user 0.020s sys 0.016s
I20260809 15:46:55.448335 23404 rpcz_store.cc:276] 0809 15:46:53.076026 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:53.132141 (+ 56115us) service_pool.cc:227] Handling call
0809 15:46:53.132370 (+ 229us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:53.193392 (+ 61022us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:54.976056 (+1782664us) spinlock_profiling.cc:243] Waited 60.2 ms on lock 0x7b54000a8cd4. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e9220f51 00007f34e921f99e 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772 00007f34e764a94c 00007f34e7649d86
0809 15:46:54.976252 (+ 196us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":138423040}
I20260809 15:46:55.103386 27283 raft_consensus.cc:482] T 28e17dcb5efe45b8b6a8c42da03b4819 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
W20260809 15:46:55.405579 23172 debug-util.cc:398] Leaking SignalData structure 0x7b0800063f20 after lost signal to thread 21293
W20260809 15:46:55.462266 23472 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 477206us
I20260809 15:46:55.103593 27268 consensus_queue.cc:260] T 0acc47ce21c3422698d1ef7277c3ed63 P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.2, Last appended by leader: 2, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:46:55.109519 26772 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.189s user 0.004s sys 0.009s
I20260809 15:46:55.449957 23395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7660dfa2ef694927b8eda4100a2bcd9b" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:55.450484 23228 leader_election.cc:302] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
W20260809 15:46:55.462513 23172 debug-util.cc:398] Leaking SignalData structure 0x7b0800035000 after lost signal to thread 23176
I20260809 15:46:55.449445 23526 rpcz_store.cc:276] 0809 15:46:53.591106 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:53.591222 (+ 116us) service_pool.cc:227] Handling call
0809 15:46:53.591498 (+ 276us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:53.614023 (+ 22525us) raft_consensus.cc:1520] Early marking committed up to index 4
0809 15:46:53.614242 (+ 219us) log.cc:842] Serialized 14 byte log entry
0809 15:46:53.825062 (+210820us) log.cc:842] Serialized 14 byte log entry
0809 15:46:54.000752 (+175690us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 15:46:54.000827 (+ 75us) raft_consensus.cc:1631] Marking committed up to 4
0809 15:46:54.026679 (+ 25852us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:54.768349 (+741670us) spinlock_profiling.cc:243] Waited 5.11 ms on lock 0x7b4800280758. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34ee0c77f1 00007f34ee0be8e7 00007f34ee0be77a 00007f34ee1b295f 00007f34ee1b2627 00007f34ee1c6b2c 00007f34ee1c6ad8 00007f34ee1c6a58 00007f34ee1c6a13 00007f34ee1c5cbc 00007f34ed147be6
0809 15:46:55.016307 (+247958us) spinlock_profiling.cc:243] Waited 5.09 ms on lock 0x7b180039c8e0. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34ef1978b1 00007f34ef1a4661 00007f34e62f8638 00007f34e62fbb11 00007f34e62fbac9 00007f34e62fba59 00007f34e62fba21 00007f34e62fad1d 00007f34ef55c204 00007f34ef55c039 00007f34e62dadc6
0809 15:46:55.057606 (+ 41299us) spinlock_profiling.cc:243] Waited 22.3 ms on lock 0x7b540020f154. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e9220f51 00007f34e921f99e 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772 00007f34e764a94c 00007f34e7649d86
0809 15:46:55.057656 (+ 50us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:55.057768 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":9555,"prepare.run_cpu_time_us":66,"prepare.run_wall_time_us":68,"raft.queue_time_us":145066,"raft.run_cpu_time_us":9100,"raft.run_wall_time_us":882112,"spinlock_wait_cycles":74818816,"thread_start_us":385595,"threads_started":2,"wal-append.queue_time_us":296746}
W20260809 15:46:55.464649 23172 debug-util.cc:398] Leaking SignalData structure 0x7b08000359a0 after lost signal to thread 23193
W20260809 15:46:55.465026 23172 debug-util.cc:398] Leaking SignalData structure 0x7b0800055020 after lost signal to thread 23200
I20260809 15:46:55.464702 23276 raft_consensus.cc:1261] T 30c1740ef6874df0a332fbd6fd4b03ce P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260809 15:46:55.420408 27032 raft_consensus.cc:3032] T ae6d2c08a7a94f58ac8dd698e3fd456c P 65c36d164b0540f099d592f38307ee6c [term 5 LEADER]: Stepping down as leader of term 5
I20260809 15:46:55.469411 23522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1a134ba33ea4076bbaac0ae460c17ba" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:55.469501 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "907bacfa79d04a55a68c8dbe60056462" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f"
I20260809 15:46:55.471236 23274 raft_consensus.cc:1226] T ba56725e42f74a1baa4c04d9660727f9 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 3. Current term is 4. Ops: []
W20260809 15:46:55.471819 23172 debug-util.cc:398] Leaking SignalData structure 0x7b0800350760 after lost signal to thread 23320
I20260809 15:46:55.466370 27316 raft_consensus.cc:482] T 4564dfc6d4094c1b9240b367ccecf3b1 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:55.448014 26527 raft_consensus.cc:504] T d3be1e356b254d22b995be32f1c12f6f P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:55.460919 26810 raft_consensus.cc:504] T 0acc47ce21c3422698d1ef7277c3ed63 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:55.472694 27316 raft_consensus.cc:504] T 4564dfc6d4094c1b9240b367ccecf3b1 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:55.457047 27076 raft_consensus.cc:504] T 2db057fd10e149a9895dff6ef8c9f038 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:55.474557 23522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "763e58d6ba644eea830a6c1664d39c9b" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:55.474923 27316 leader_election.cc:288] T 4564dfc6d4094c1b9240b367ccecf3b1 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:55.473256 27079 leader_election.cc:288] T 0c48a3049897452383257aa757834fb5 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:55.469493 27283 raft_consensus.cc:504] T 28e17dcb5efe45b8b6a8c42da03b4819 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:55.458704 27274 raft_consensus.cc:504] T cee02fff2c9c48a19f43eb358a89e7b0 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:55.476063 27322 leader_election.cc:288] T 6ae14a74a26f434db08c652e097385c8 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:55.447667 27266 raft_consensus.cc:3037] T c5c2e425a74b4e64ae70b5def750434c P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Advancing to term 6
I20260809 15:46:55.478890 23280 raft_consensus.cc:2454] T 28e17dcb5efe45b8b6a8c42da03b4819 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 6.
I20260809 15:46:55.233808 27278 leader_election.cc:288] T a88421dba80d4d3388aa5eb62e23853b P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:55.452957 23517 consensus_queue.cc:260] T 4b30f28311964a0b9149a50495e83d8f P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:55.479490 27347 raft_consensus.cc:482] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:55.446979 27053 raft_consensus.cc:504] T 3924b648461e463b9d7925e5671d68ae P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
W20260809 15:46:55.486610 23172 debug-util.cc:398] Leaking SignalData structure 0x7b0800331980 after lost signal to thread 23443
W20260809 15:46:55.554534 23172 debug-util.cc:398] Leaking SignalData structure 0x7b0800353f80 after lost signal to thread 23567
W20260809 15:46:55.554854 23172 debug-util.cc:398] Leaking SignalData structure 0x7b0800353f60 after lost signal to thread 23568
I20260809 15:46:55.550837 27136 raft_consensus.cc:482] T 07fcabb569414bddbd39df6cedd24e64 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:55.597903 27136 raft_consensus.cc:504] T 07fcabb569414bddbd39df6cedd24e64 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:55.600572 27136 leader_election.cc:288] T 07fcabb569414bddbd39df6cedd24e64 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:55.601205 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07fcabb569414bddbd39df6cedd24e64" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:55.475350 27076 leader_election.cc:288] T 2db057fd10e149a9895dff6ef8c9f038 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:55.479194 27174 raft_consensus.cc:482] T cef3e7277d7d4760824a3436ed4a847d P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:55.469655 27118 raft_consensus.cc:482] T 2db057fd10e149a9895dff6ef8c9f038 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:55.475086 27166 consensus_queue.cc:1035] T d1a134ba33ea4076bbaac0ae460c17ba P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.084s
I20260809 15:46:55.421573 24451 rpcz_store.cc:276] 0809 15:46:52.514123 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:52.515414 (+ 1291us) service_pool.cc:227] Handling call
0809 15:46:52.538022 (+ 22608us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 15:46:53.097639 (+559617us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:46:53.097680 (+ 41us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 15:46:53.098162 (+ 482us) log.cc:842] Serialized 26 byte log entry
0809 15:46:53.413326 (+315164us) spinlock_profiling.cc:243] Waited 1.47 ms on lock 0x7b18004631e0. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34ef1978b1 00007f34ef1a4661 00007f34e62da345 00007f34ef111a39 00007f34e62f78bb 00007f34e62f5cbf 00007f34e62f7e22 00007f34e8ea15bb 00007f34e8ea7063 00007f34e8ea7411 00007f34e91f45cf
0809 15:46:53.529850 (+116524us) raft_consensus.cc:1631] Marking committed up to 5
0809 15:46:53.870530 (+340680us) log.cc:842] Serialized 14 byte log entry
0809 15:46:53.998677 (+128147us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:55.015011 (+1016334us) spinlock_profiling.cc:243] Waited 559 ms on lock 0x7b540047fc54. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e9220f51 00007f34e921f99e 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772 00007f34e764a94c 00007f34e7649d86
0809 15:46:55.015090 (+ 79us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 15:46:55.015144 (+ 54us) raft_consensus.cc:1671] finished
0809 15:46:55.015160 (+ 16us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:55.015365 (+ 205us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":10449,"prepare.queue_time_us":67540,"prepare.run_cpu_time_us":28,"prepare.run_wall_time_us":29,"raft.queue_time_us":28789,"raft.run_cpu_time_us":183,"raft.run_wall_time_us":3716,"spinlock_wait_cycles":1289465344,"thread_start_us":511384,"threads_started":3,"wal-append.queue_time_us":478146,"wal-append.run_cpu_time_us":4098,"wal-append.run_wall_time_us":1356074}
I20260809 15:46:55.469863 27032 raft_consensus.cc:729] T ae6d2c08a7a94f58ac8dd698e3fd456c P 65c36d164b0540f099d592f38307ee6c [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:46:55.449481 27194 raft_consensus.cc:504] T 563daf42d7784db195ea99c3cb21b370 P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:55.477226 27283 leader_election.cc:288] T 28e17dcb5efe45b8b6a8c42da03b4819 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:55.447439 27072 raft_consensus.cc:504] T 99bdb93503db4de4a1cfb0161ea0c07c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:46:55.685757 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 235310us
I20260809 15:46:55.610785 23522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4564dfc6d4094c1b9240b367ccecf3b1" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:55.446643 27334 leader_election.cc:288] T d3e94b7e50754a81b3d72a07476cb7b9 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:55.458606 27034 raft_consensus.cc:504] T f9e7cadcd24d45fcbfa84209225e8f2d P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:55.631561 27136 raft_consensus.cc:482] T 0acc47ce21c3422698d1ef7277c3ed63 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:55.481513 26772 raft_consensus.cc:482] T 0c29937a80724ddb9d07a27534a146f3 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:55.556540 27316 raft_consensus.cc:482] T f7715d71dbac447b9351fc5eb36c5e80 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:55.613857 23518 raft_consensus.cc:1261] T 3924b648461e463b9d7925e5671d68ae P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Refusing update from remote peer 61e69695258f48baa984c9d395e8ce3f: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
W20260809 15:46:55.596081 23172 debug-util.cc:398] Leaking SignalData structure 0x7b080030c9c0 after lost signal to thread 24513
I20260809 15:46:55.633733 27118 raft_consensus.cc:504] T 2db057fd10e149a9895dff6ef8c9f038 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:46:55.700994 23172 debug-util.cc:398] Leaking SignalData structure 0x7b08003bbc40 after lost signal to thread 24516
I20260809 15:46:55.109475 27313 raft_consensus.cc:504] T cee02fff2c9c48a19f43eb358a89e7b0 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:55.483189 27347 raft_consensus.cc:504] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:55.483773 27213 raft_consensus.cc:482] T 297c82ed906d4b48b184985eed3de601 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
W20260809 15:46:55.606266 23285 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.521s user 0.012s sys 0.018s
W20260809 15:46:55.706247 23285 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.621s user 0.012s sys 0.018s
W20260809 15:46:55.607395 23472 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 143940us
I20260809 15:46:55.685520 23283 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53554 (request call id 1479) took 3238 ms. Trace:
I20260809 15:46:55.453708 27134 raft_consensus.cc:504] T 538a228ddfb54bc6a36528d5ff07597b P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:55.448805 27298 raft_consensus.cc:504] T 907bacfa79d04a55a68c8dbe60056462 P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:55.707247 23283 rpcz_store.cc:276] 0809 15:46:52.402183 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:52.402381 (+ 198us) service_pool.cc:227] Handling call
0809 15:46:52.461471 (+ 59090us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 15:46:52.478537 (+ 17066us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:46:52.478557 (+ 20us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 15:46:52.481466 (+ 2909us) log.cc:842] Serialized 26 byte log entry
0809 15:46:53.089233 (+607767us) raft_consensus.cc:1631] Marking committed up to 3
0809 15:46:53.089283 (+ 50us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:53.089342 (+ 59us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 15:46:55.049176 (+1959834us) spinlock_profiling.cc:243] Waited 6.95 ms on lock 0x7b44000232cc. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e613cf11 00007f34e620aa95 00007f34e620b4b2 00007f34e8eb3511 00007f34e8eaee1a 00007f34e8ea2bee 00007f34e8ea1f36 00007f34e8ea1b58 00007f34e8eab551 00007f34e8eab509 00007f34e8eab499
0809 15:46:55.632712 (+583536us) raft_consensus.cc:1671] finished
0809 15:46:55.632734 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:55.640644 (+ 7910us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":10903,"raft.queue_time_us":12556,"raft.run_cpu_time_us":114,"raft.run_wall_time_us":4779,"spinlock_wait_cycles":15979904,"thread_start_us":188752,"threads_started":1,"wal-append.queue_time_us":656556}
I20260809 15:46:55.689364 27274 leader_election.cc:288] T cee02fff2c9c48a19f43eb358a89e7b0 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:55.610268 27076 raft_consensus.cc:482] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
I20260809 15:46:55.449255 27162 raft_consensus.cc:504] T c0741ab44d99439abc3aa3bdd27ee160 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:55.607311 27216 raft_consensus.cc:482] T a3622f6e3e3641cea1960e3a31498bb7 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:55.642206 23276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1a134ba33ea4076bbaac0ae460c17ba" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:55.449136 27339 raft_consensus.cc:504] T 294d4bcc1e7a477cb56455581277a60f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:55.448504 27337 raft_consensus.cc:504] T 953eea268e17491ca76f3a555860a512 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:55.612097 26719 raft_consensus.cc:482] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
I20260809 15:46:55.483939 27259 raft_consensus.cc:482] T df20671a7b874dbb98c0ef28371c1308 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:55.633427 27174 raft_consensus.cc:504] T cef3e7277d7d4760824a3436ed4a847d P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:55.685721 26405 raft_consensus.cc:482] T 6ced718c1a5c4da79cb8c1ebd1435b87 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:55.742398 24472 raft_consensus.cc:2362] T 907bacfa79d04a55a68c8dbe60056462 P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 5. Re-sending same reply.
I20260809 15:46:55.711874 23397 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55332 (request call id 1342) took 2165 ms. Trace:
I20260809 15:46:55.687615 23228 leader_election.cc:302] T 07fcabb569414bddbd39df6cedd24e64 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c
I20260809 15:46:55.696406 26757 raft_consensus.cc:482] T a88421dba80d4d3388aa5eb62e23853b P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:55.703194 27217 raft_consensus.cc:482] T d0721a4403a440f383318a3cea9d1442 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:55.734143 23276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "763e58d6ba644eea830a6c1664d39c9b" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:55.715914 27278 raft_consensus.cc:2726] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260809 15:46:55.737211 23283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ae14a74a26f434db08c652e097385c8" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:55.787442 23283 raft_consensus.cc:1261] T 00a1881267a14ba8b78c81040f17a014 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260809 15:46:55.787804 27053 leader_election.cc:288] T 3924b648461e463b9d7925e5671d68ae P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:55.788100 23393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "563daf42d7784db195ea99c3cb21b370" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:55.740773 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c48a3049897452383257aa757834fb5" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:55.723671 27283 raft_consensus.cc:482] T 30c1740ef6874df0a332fbd6fd4b03ce P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:55.796990 27283 raft_consensus.cc:504] T 30c1740ef6874df0a332fbd6fd4b03ce P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:55.701824 27136 raft_consensus.cc:504] T 0acc47ce21c3422698d1ef7277c3ed63 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:55.745671 26405 raft_consensus.cc:504] T 6ced718c1a5c4da79cb8c1ebd1435b87 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:55.751587 26810 leader_election.cc:288] T 0acc47ce21c3422698d1ef7277c3ed63 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:55.760905 23282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2424fd9bd7f14e508fd106cbfda786a4" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 7 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:55.763334 27174 leader_election.cc:288] T cef3e7277d7d4760824a3436ed4a847d P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:55.764065 27194 leader_election.cc:288] T 563daf42d7784db195ea99c3cb21b370 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:55.764617 27072 leader_election.cc:288] T 99bdb93503db4de4a1cfb0161ea0c07c P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:55.766552 27339 leader_election.cc:288] T 294d4bcc1e7a477cb56455581277a60f P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:55.708405 27316 raft_consensus.cc:504] T f7715d71dbac447b9351fc5eb36c5e80 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:46:55.753180 27357 log.cc:924] Time spent T d1a134ba33ea4076bbaac0ae460c17ba P 67b3aeadfa7a432abe8af4d292e04ab9: Append to log took a long time: real 0.144s user 0.000s sys 0.000s
I20260809 15:46:55.705555 26772 raft_consensus.cc:504] T 0c29937a80724ddb9d07a27534a146f3 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:55.723785 27213 raft_consensus.cc:504] T 297c82ed906d4b48b184985eed3de601 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:55.755970 26527 leader_election.cc:288] T d3be1e356b254d22b995be32f1c12f6f P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 6 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:55.770397 23522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7660dfa2ef694927b8eda4100a2bcd9b" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:55.448125 24455 rpcz_store.cc:276] 0809 15:46:53.429110 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:53.436466 (+ 7356us) service_pool.cc:227] Handling call
0809 15:46:53.436758 (+ 292us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:53.436962 (+ 204us) raft_consensus.cc:1520] Early marking committed up to index 5
0809 15:46:53.596100 (+159138us) log.cc:842] Serialized 14 byte log entry
0809 15:46:54.995578 (+1399478us) spinlock_profiling.cc:243] Waited 8.81 ms on lock 0x7b180031be20. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34ef1978b1 00007f34ef1a4661 00007f34e62da345 00007f34ef111a39 00007f34e62f78bb 00007f34e62f5cbf 00007f34e62f7e22 00007f34e8ea15bb 00007f34e8ea7063 00007f34e8ea757b 00007f34e922a88a
0809 15:46:55.006883 (+ 11305us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 15:46:55.006932 (+ 49us) raft_consensus.cc:1631] Marking committed up to 5
0809 15:46:55.006981 (+ 49us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:55.007058 (+ 77us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:55.007174 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":167451,"prepare.run_cpu_time_us":29,"prepare.run_wall_time_us":30,"raft.queue_time_us":850,"raft.run_cpu_time_us":100,"raft.run_wall_time_us":100,"spinlock_wait_cycles":20251904,"thread_start_us":156111,"threads_started":2,"wal-append.queue_time_us":1407372}
I20260809 15:46:55.752344 26747 raft_consensus.cc:482] T ba56725e42f74a1baa4c04d9660727f9 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:55.756880 27280 raft_consensus.cc:2781] T 07fcabb569414bddbd39df6cedd24e64 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260809 15:46:55.782786 27162 leader_election.cc:288] T c0741ab44d99439abc3aa3bdd27ee160 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:55.744477 27259 raft_consensus.cc:504] T df20671a7b874dbb98c0ef28371c1308 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:55.844877 27174 raft_consensus.cc:482] T 998c6cee72cf40228c6dba064c0db2da P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:55.845510 27174 raft_consensus.cc:504] T 998c6cee72cf40228c6dba064c0db2da P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:55.734977 27032 consensus_queue.cc:260] T ae6d2c08a7a94f58ac8dd698e3fd456c P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:55.851405 27280 raft_consensus.cc:482] T 07fcabb569414bddbd39df6cedd24e64 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:55.851899 27280 raft_consensus.cc:3037] T 07fcabb569414bddbd39df6cedd24e64 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Advancing to term 6
I20260809 15:46:55.800738 27217 raft_consensus.cc:504] T d0721a4403a440f383318a3cea9d1442 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:55.855965 26747 raft_consensus.cc:504] T ba56725e42f74a1baa4c04d9660727f9 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:55.856706 27118 leader_election.cc:288] T 2db057fd10e149a9895dff6ef8c9f038 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
W20260809 15:46:55.858608 27295 consensus_meta.cc:287] T a3622f6e3e3641cea1960e3a31498bb7 P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent flushing consensus metadata: real 0.786s user 0.006s sys 0.005s
I20260809 15:46:55.859100 27295 raft_consensus.cc:504] T a3622f6e3e3641cea1960e3a31498bb7 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:55.846271 27259 leader_election.cc:288] T df20671a7b874dbb98c0ef28371c1308 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
W20260809 15:46:55.789813 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 102237us
I20260809 15:46:55.868950 23228 leader_election.cc:302] T ae6d2c08a7a94f58ac8dd698e3fd456c P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:55.790880 23276 raft_consensus.cc:1226] T a88421dba80d4d3388aa5eb62e23853b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 3. Current term is 4. Ops: []
I20260809 15:46:55.744874 23397 rpcz_store.cc:276] 0809 15:46:53.546077 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:53.546914 (+ 837us) service_pool.cc:227] Handling call
0809 15:46:53.547158 (+ 244us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:53.547352 (+ 194us) raft_consensus.cc:1520] Early marking committed up to index 6
0809 15:46:55.545406 (+1998054us) log.cc:842] Serialized 14 byte log entry
0809 15:46:55.700244 (+154838us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 15:46:55.700324 (+ 80us) raft_consensus.cc:1631] Marking committed up to 6
0809 15:46:55.700399 (+ 75us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:55.711714 (+ 11315us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:55.711820 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1866589,"prepare.queue_time_us":1930584,"prepare.run_cpu_time_us":27,"prepare.run_wall_time_us":27,"raft.queue_time_us":33766,"raft.run_cpu_time_us":98,"raft.run_wall_time_us":5246,"thread_start_us":226153,"threads_started":2}
I20260809 15:46:55.796334 27298 leader_election.cc:288] T 907bacfa79d04a55a68c8dbe60056462 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:55.788759 23393 raft_consensus.cc:2454] T 563daf42d7784db195ea99c3cb21b370 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 4.
I20260809 15:46:55.738559 27075 leader_election.cc:288] T ba56725e42f74a1baa4c04d9660727f9 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
W20260809 15:46:55.811841 23353 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 78347.8us
I20260809 15:46:55.742178 27263 raft_consensus.cc:482] T 6ae14a74a26f434db08c652e097385c8 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 15:46:55.819312 23472 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 94438.5us
I20260809 15:46:55.824832 23399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "907bacfa79d04a55a68c8dbe60056462" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:55.888389 23399 raft_consensus.cc:2454] T 907bacfa79d04a55a68c8dbe60056462 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 5.
I20260809 15:46:55.736604 27216 raft_consensus.cc:504] T a3622f6e3e3641cea1960e3a31498bb7 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:55.799697 27390 raft_consensus.cc:482] T 953eea268e17491ca76f3a555860a512 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:55.896584 27313 leader_election.cc:288] T cee02fff2c9c48a19f43eb358a89e7b0 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:55.788139 27134 leader_election.cc:288] T 538a228ddfb54bc6a36528d5ff07597b P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:55.897336 26405 leader_election.cc:288] T 6ced718c1a5c4da79cb8c1ebd1435b87 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:55.898396 23522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "538a228ddfb54bc6a36528d5ff07597b" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:55.898993 27174 leader_election.cc:288] T 998c6cee72cf40228c6dba064c0db2da P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:55.831216 23283 raft_consensus.cc:3032] T 4564dfc6d4094c1b9240b367ccecf3b1 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 LEADER]: Stepping down as leader of term 4
I20260809 15:46:55.735908 27076 raft_consensus.cc:504] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:55.901166 26772 leader_election.cc:288] T 0c29937a80724ddb9d07a27534a146f3 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:55.901193 25863 raft_consensus.cc:3032] T 2424fd9bd7f14e508fd106cbfda786a4 P 65c36d164b0540f099d592f38307ee6c [term 5 LEADER]: Stepping down as leader of term 5
I20260809 15:46:55.902158 25863 raft_consensus.cc:729] T 2424fd9bd7f14e508fd106cbfda786a4 P 65c36d164b0540f099d592f38307ee6c [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:46:55.902709 27076 leader_election.cc:288] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:55.903477 23283 raft_consensus.cc:729] T 4564dfc6d4094c1b9240b367ccecf3b1 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Running, Role: LEADER
I20260809 15:46:55.744203 26719 raft_consensus.cc:504] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:55.815364 27283 leader_election.cc:288] T 30c1740ef6874df0a332fbd6fd4b03ce P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:55.880105 23228 leader_election.cc:302] T f7715d71dbac447b9351fc5eb36c5e80 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:55.804949 26757 raft_consensus.cc:504] T a88421dba80d4d3388aa5eb62e23853b P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:55.889889 24410 leader_election.cc:302] T 563daf42d7784db195ea99c3cb21b370 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:46:55.892059 23520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0acc47ce21c3422698d1ef7277c3ed63" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 7 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:55.892875 27347 leader_election.cc:288] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:55.887228 27162 raft_consensus.cc:3037] T ae6d2c08a7a94f58ac8dd698e3fd456c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Advancing to term 6
I20260809 15:46:55.893504 27116 raft_consensus.cc:3037] T 73f6f44c259547f1b05a3a434616fa54 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Advancing to term 6
I20260809 15:46:55.880788 27244 raft_consensus.cc:2726] T 00a1881267a14ba8b78c81040f17a014 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Leader election lost for term 6. Reason: Vote denied by peer c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263) with higher term. Message: Invalid argument: T 00a1881267a14ba8b78c81040f17a014 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c for earlier term 6. Current term is 7.
I20260809 15:46:55.920162 26747 leader_election.cc:288] T ba56725e42f74a1baa4c04d9660727f9 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:55.920586 26719 leader_election.cc:288] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:55.895195 27378 consensus_queue.cc:1046] T 4564dfc6d4094c1b9240b367ccecf3b1 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260809 15:46:55.896320 27390 raft_consensus.cc:504] T 953eea268e17491ca76f3a555860a512 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:55.836488 27136 leader_election.cc:288] T 0acc47ce21c3422698d1ef7277c3ed63 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:55.908552 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df20671a7b874dbb98c0ef28371c1308" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:55.910530 24410 leader_election.cc:302] T 7660dfa2ef694927b8eda4100a2bcd9b P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:55.909648 26228 consensus_queue.cc:1035] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.009s
I20260809 15:46:55.916491 27210 raft_consensus.cc:2781] T 563daf42d7784db195ea99c3cb21b370 P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260809 15:46:55.916785 27347 raft_consensus.cc:2726] T f7715d71dbac447b9351fc5eb36c5e80 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260809 15:46:55.903316 25863 consensus_queue.cc:260] T 2424fd9bd7f14e508fd106cbfda786a4 P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.2, Last appended by leader: 2, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:55.919426 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2db057fd10e149a9895dff6ef8c9f038" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:55.887013 26730 raft_consensus.cc:482] T 563daf42d7784db195ea99c3cb21b370 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
I20260809 15:46:55.939181 26730 raft_consensus.cc:504] T 563daf42d7784db195ea99c3cb21b370 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:55.909232 27266 raft_consensus.cc:504] T c5c2e425a74b4e64ae70b5def750434c P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:55.921020 27034 leader_election.cc:288] T f9e7cadcd24d45fcbfa84209225e8f2d P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:55.926282 23228 leader_election.cc:302] T 2424fd9bd7f14e508fd106cbfda786a4 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:55.927690 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998c6cee72cf40228c6dba064c0db2da" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:55.928900 24410 leader_election.cc:302] T 907bacfa79d04a55a68c8dbe60056462 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:46:55.920090 27081 raft_consensus.cc:3032] T d2d4f749fcfd4fb3bfb8115af3471598 P 65c36d164b0540f099d592f38307ee6c [term 4 LEADER]: Stepping down as leader of term 4
I20260809 15:46:55.931035 23522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ced718c1a5c4da79cb8c1ebd1435b87" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:55.965353 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a823d35f4f348d1b52ba9e6d80b00d2" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:55.935815 27216 leader_election.cc:288] T a3622f6e3e3641cea1960e3a31498bb7 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:55.929471 27210 raft_consensus.cc:482] T 563daf42d7784db195ea99c3cb21b370 P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Starting leader election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
I20260809 15:46:55.938488 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c29937a80724ddb9d07a27534a146f3" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:55.947943 27316 leader_election.cc:288] T f7715d71dbac447b9351fc5eb36c5e80 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:55.923796 23528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cef3e7277d7d4760824a3436ed4a847d" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:55.966105 23526 raft_consensus.cc:2379] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d in current term 7: Already voted for candidate 65c36d164b0540f099d592f38307ee6c in this term.
I20260809 15:46:55.924450 27295 leader_election.cc:288] T a3622f6e3e3641cea1960e3a31498bb7 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:55.930615 27206 raft_consensus.cc:2726] T 7660dfa2ef694927b8eda4100a2bcd9b P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260809 15:46:55.970892 23228 leader_election.cc:302] T d2d4f749fcfd4fb3bfb8115af3471598 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:55.947963 27213 leader_election.cc:288] T 297c82ed906d4b48b184985eed3de601 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:55.964861 27081 raft_consensus.cc:729] T d2d4f749fcfd4fb3bfb8115af3471598 P 65c36d164b0540f099d592f38307ee6c [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:46:55.975572 27210 raft_consensus.cc:3037] T 563daf42d7784db195ea99c3cb21b370 P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Advancing to term 5
I20260809 15:46:55.970235 27347 consensus_queue.cc:1035] T 0a51c0baaa6e432997c9d3552af4f254 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.006s
I20260809 15:46:55.978493 27390 leader_election.cc:288] T 953eea268e17491ca76f3a555860a512 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:55.968277 27378 raft_consensus.cc:3037] T 2424fd9bd7f14e508fd106cbfda786a4 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Advancing to term 6
I20260809 15:46:55.979753 27266 leader_election.cc:288] T c5c2e425a74b4e64ae70b5def750434c P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 6 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:55.968350 27136 raft_consensus.cc:482] T 0da4d5283c8a46878bd39b85e5c71f66 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
W20260809 15:46:55.998221 23470 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 116873us
I20260809 15:46:55.981212 23528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "297c82ed906d4b48b184985eed3de601" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:55.981513 27280 raft_consensus.cc:504] T 07fcabb569414bddbd39df6cedd24e64 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:55.968441 27298 raft_consensus.cc:2781] T 907bacfa79d04a55a68c8dbe60056462 P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260809 15:46:56.010598 27298 raft_consensus.cc:482] T 907bacfa79d04a55a68c8dbe60056462 P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:46:56.011154 27298 raft_consensus.cc:3037] T 907bacfa79d04a55a68c8dbe60056462 P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Advancing to term 6
I20260809 15:46:56.012161 27280 leader_election.cc:288] T 07fcabb569414bddbd39df6cedd24e64 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 6 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:56.011873 27081 consensus_queue.cc:260] T d2d4f749fcfd4fb3bfb8115af3471598 P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:56.013499 27081 raft_consensus.cc:3037] T d2d4f749fcfd4fb3bfb8115af3471598 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Advancing to term 5
I20260809 15:46:56.015394 27266 raft_consensus.cc:482] T 213543cdbe814c89a17a963469922c1b P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:56.016063 27266 raft_consensus.cc:504] T 213543cdbe814c89a17a963469922c1b P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:55.984107 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7715d71dbac447b9351fc5eb36c5e80" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:56.018905 27266 leader_election.cc:288] T 213543cdbe814c89a17a963469922c1b P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:56.019385 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "953eea268e17491ca76f3a555860a512" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:56.019969 23526 raft_consensus.cc:2454] T 953eea268e17491ca76f3a555860a512 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 5.
I20260809 15:46:55.986171 27178 raft_consensus.cc:482] T f9e7cadcd24d45fcbfa84209225e8f2d P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:56.020820 23470 leader_election.cc:302] T 73f6f44c259547f1b05a3a434616fa54 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:46:56.021615 27334 raft_consensus.cc:2781] T 73f6f44c259547f1b05a3a434616fa54 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260809 15:46:56.021307 27178 raft_consensus.cc:504] T f9e7cadcd24d45fcbfa84209225e8f2d P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:56.022009 27334 raft_consensus.cc:482] T 73f6f44c259547f1b05a3a434616fa54 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:46:56.022504 27334 raft_consensus.cc:3037] T 73f6f44c259547f1b05a3a434616fa54 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Advancing to term 7
I20260809 15:46:56.024643 23353 leader_election.cc:302] T 953eea268e17491ca76f3a555860a512 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:46:56.025683 24455 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3be1e356b254d22b995be32f1c12f6f" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f"
I20260809 15:46:56.026522 24455 raft_consensus.cc:3037] T d3be1e356b254d22b995be32f1c12f6f P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Advancing to term 6
I20260809 15:46:56.027478 27394 consensus_queue.cc:1035] T 763e58d6ba644eea830a6c1664d39c9b P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260809 15:46:56.026120 24451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a823d35f4f348d1b52ba9e6d80b00d2" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:56.032194 24451 raft_consensus.cc:2454] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 61e69695258f48baa984c9d395e8ce3f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 7.
I20260809 15:46:55.997125 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba56725e42f74a1baa4c04d9660727f9" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:55.999722 23522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3622f6e3e3641cea1960e3a31498bb7" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:56.034257 27334 raft_consensus.cc:504] T 73f6f44c259547f1b05a3a434616fa54 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:56.000903 23228 leader_election.cc:302] T 00a1881267a14ba8b78c81040f17a014 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:55.911528 27263 raft_consensus.cc:504] T 6ae14a74a26f434db08c652e097385c8 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:55.985136 27395 raft_consensus.cc:482] T d0721a4403a440f383318a3cea9d1442 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
W20260809 15:46:56.080667 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 79841.5us
W20260809 15:46:56.081123 24406 connection.cc:232] RPC call timeout handler was delayed by -1.6409872809890658s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21000000000000002s for a response
I20260809 15:46:56.003795 23528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5c2e425a74b4e64ae70b5def750434c" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
W20260809 15:46:56.008580 27365 log.cc:924] Time spent T 763e58d6ba644eea830a6c1664d39c9b P 65c36d164b0540f099d592f38307ee6c: Append to log took a long time: real 0.540s user 0.001s sys 0.003s
I20260809 15:46:56.037101 27350 consensus_queue.cc:1035] T 30c1740ef6874df0a332fbd6fd4b03ce P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260809 15:46:56.109741 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9e7cadcd24d45fcbfa84209225e8f2d" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:56.110386 23527 raft_consensus.cc:2454] T f9e7cadcd24d45fcbfa84209225e8f2d P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 5.
W20260809 15:46:56.110759 24406 connection.cc:232] RPC call timeout handler was delayed by -1.2448820159770548s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1563s for a response
W20260809 15:46:56.111122 24406 connection.cc:232] RPC call timeout handler was delayed by -1.1494820159859955s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16690000000000002s for a response
I20260809 15:46:55.968633 27337 leader_election.cc:288] T 953eea268e17491ca76f3a555860a512 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:56.041658 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "213543cdbe814c89a17a963469922c1b" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:56.042225 23399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e17dcb5efe45b8b6a8c42da03b4819" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:56.045950 25863 raft_consensus.cc:3037] T 2424fd9bd7f14e508fd106cbfda786a4 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Advancing to term 6
I20260809 15:46:56.037050 27268 raft_consensus.cc:3037] T 0acc47ce21c3422698d1ef7277c3ed63 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Advancing to term 6
I20260809 15:46:56.052320 23517 raft_consensus.cc:3037] T 4b30f28311964a0b9149a50495e83d8f P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:56.025660 27266 raft_consensus.cc:2781] T 953eea268e17491ca76f3a555860a512 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260809 15:46:56.000201 27295 raft_consensus.cc:2726] T d2d4f749fcfd4fb3bfb8115af3471598 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260809 15:46:56.056823 26757 leader_election.cc:288] T a88421dba80d4d3388aa5eb62e23853b P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:56.057960 23521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a88421dba80d4d3388aa5eb62e23853b" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:56.055543 27178 leader_election.cc:288] T f9e7cadcd24d45fcbfa84209225e8f2d P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:56.060256 23273 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:53552 (request call id 917) took 2450 ms. Trace:
I20260809 15:46:56.060426 23397 raft_consensus.cc:1226] T ae6d2c08a7a94f58ac8dd698e3fd456c P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 5. Current term is 6. Ops: []
I20260809 15:46:56.063678 23404 raft_consensus.cc:1261] T 4564dfc6d4094c1b9240b367ccecf3b1 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260809 15:46:56.064698 23395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3e94b7e50754a81b3d72a07476cb7b9" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:56.078480 23394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a823d35f4f348d1b52ba9e6d80b00d2" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:56.002020 27136 raft_consensus.cc:504] T 0da4d5283c8a46878bd39b85e5c71f66 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:56.109160 23528 raft_consensus.cc:3037] T c5c2e425a74b4e64ae70b5def750434c P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Advancing to term 6
I20260809 15:46:56.107846 27395 raft_consensus.cc:504] T d0721a4403a440f383318a3cea9d1442 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:56.130342 24463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a823d35f4f348d1b52ba9e6d80b00d2" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:56.112990 27055 raft_consensus.cc:482] T 0a51c0baaa6e432997c9d3552af4f254 P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
I20260809 15:46:56.061627 27162 raft_consensus.cc:2726] T ae6d2c08a7a94f58ac8dd698e3fd456c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260809 15:46:56.108510 27347 raft_consensus.cc:2726] T 00a1881267a14ba8b78c81040f17a014 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260809 15:46:56.140709 23399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9e7cadcd24d45fcbfa84209225e8f2d" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:56.211158 23228 leader_election.cc:302] T 3296ef50be254fc29cfc77d22a8f7a23 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:56.210558 23395 raft_consensus.cc:2454] T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 6.
I20260809 15:46:56.212821 23345 leader_election.cc:302] T 907bacfa79d04a55a68c8dbe60056462 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d; no voters: 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:56.215636 23395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ae14a74a26f434db08c652e097385c8" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:56.215902 23345 leader_election.cc:302] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c
I20260809 15:46:56.209879 23273 rpcz_store.cc:276] 0809 15:46:53.610117 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:53.613631 (+ 3514us) service_pool.cc:227] Handling call
0809 15:46:56.059705 (+2446074us) spinlock_profiling.cc:243] Waited 1.96 s on lock 0x7b54003a98d4. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5af0 00007f34e9228f2a 00007f34ee165c6e 00007f34e785c807 00007f34e785c792 00007f34e785c6c1 00007f34e785c63c 00007f34e785b8f2 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:46:56.060140 (+ 435us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4512186240}
I20260809 15:46:56.222055 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73f6f44c259547f1b05a3a434616fa54" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:56.220229 23472 leader_election.cc:302] T d3e94b7e50754a81b3d72a07476cb7b9 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:46:56.217306 27055 raft_consensus.cc:504] T 0a51c0baaa6e432997c9d3552af4f254 P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:56.223618 24452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3924b648461e463b9d7925e5671d68ae" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 9 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:56.217940 27283 raft_consensus.cc:2726] T 3296ef50be254fc29cfc77d22a8f7a23 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 15:46:56.225806 23470 leader_election.cc:302] T 28e17dcb5efe45b8b6a8c42da03b4819 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters: c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:56.226565 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3e94b7e50754a81b3d72a07476cb7b9" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:56.213994 23353 leader_election.cc:302] T f9e7cadcd24d45fcbfa84209225e8f2d P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:46:56.216303 23228 leader_election.cc:302] T 05ad7d8fdca64e3da527280ca5be9280 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:46:56.140082 27266 raft_consensus.cc:482] T 953eea268e17491ca76f3a555860a512 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:46:56.211514 23394 raft_consensus.cc:2454] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 6.
I20260809 15:46:56.207823 27116 raft_consensus.cc:482] T 30c1740ef6874df0a332fbd6fd4b03ce P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:56.211959 23399 raft_consensus.cc:2454] T f9e7cadcd24d45fcbfa84209225e8f2d P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 5.
I20260809 15:46:56.181180 27222 raft_consensus.cc:482] T 73f6f44c259547f1b05a3a434616fa54 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:56.143621 24455 raft_consensus.cc:2454] T d3be1e356b254d22b995be32f1c12f6f P 61e69695258f48baa984c9d395e8ce3f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 6.
I20260809 15:46:56.218382 23274 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53554 (request call id 1521) took 1136 ms. Trace:
I20260809 15:46:56.214634 24451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d880de657e544468c82157769622a26" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:56.239212 23274 rpcz_store.cc:276] 0809 15:46:55.081562 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:55.462555 (+380993us) service_pool.cc:227] Handling call
0809 15:46:55.462798 (+ 243us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:55.471700 (+ 8902us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:56.218186 (+746486us) spinlock_profiling.cc:243] Waited 3.68 ms on lock 0x7b54002f49d4. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e9220f51 00007f34e921f99e 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772 00007f34e764a94c 00007f34e7649d86
0809 15:46:56.218330 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":8470400}
I20260809 15:46:56.213292 23396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7baa2492df0340c4a5ac241959ce1ac3" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:56.219197 23517 raft_consensus.cc:1261] T 4b30f28311964a0b9149a50495e83d8f P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260809 15:46:56.216311 23395 raft_consensus.cc:2454] T 6ae14a74a26f434db08c652e097385c8 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 5.
I20260809 15:46:56.221385 24463 raft_consensus.cc:2454] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 61e69695258f48baa984c9d395e8ce3f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 7.
I20260809 15:46:56.223773 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee02fff2c9c48a19f43eb358a89e7b0" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:56.225523 27055 leader_election.cc:288] T 0a51c0baaa6e432997c9d3552af4f254 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:56.185580 27378 raft_consensus.cc:2726] T 2424fd9bd7f14e508fd106cbfda786a4 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260809 15:46:56.219920 23404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "953eea268e17491ca76f3a555860a512" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:56.320890 23517 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56766 (request call id 1899) took 2952 ms. Trace:
I20260809 15:46:56.321269 23517 rpcz_store.cc:276] 0809 15:46:53.368342 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:53.368511 (+ 169us) service_pool.cc:227] Handling call
0809 15:46:53.380889 (+ 12378us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:54.754404 (+1373515us) spinlock_profiling.cc:243] Waited 34 ms on lock 0x7b5400176ea0. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e91b0730 00007f34e9222300 00007f34e9223371 00007f34e921fedf 00007f34e921f99e 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc
0809 15:46:56.051834 (+1297430us) spinlock_profiling.cc:243] Waited 497 ms on lock 0x7b580047eab0. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e919bf63 00007f34e91ffaba 00007f34e92173af 00007f34e9218b4e 00007f34e9222c89 00007f34e9223556 00007f34e921fedf 00007f34e921f99e 00007f34ee1612e9 00007f34e7847684
0809 15:46:56.320639 (+268805us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:56.320841 (+ 202us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1220660736}
I20260809 15:46:56.269623 23520 raft_consensus.cc:2454] T 0acc47ce21c3422698d1ef7277c3ed63 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 6.
I20260809 15:46:56.240113 23399 raft_consensus.cc:1226] T d2d4f749fcfd4fb3bfb8115af3471598 P c0f38c1908414566aa9115bb2cbdd85d [term 5 LEADER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 4. Current term is 5. Ops: []
I20260809 15:46:56.324460 23399 raft_consensus.cc:1226] T 4b30f28311964a0b9149a50495e83d8f P c0f38c1908414566aa9115bb2cbdd85d [term 3 LEADER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 2. Current term is 3. Ops: []
I20260809 15:46:56.317209 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3924b648461e463b9d7925e5671d68ae" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 9 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
W20260809 15:46:56.328766 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 112522us
I20260809 15:46:56.329428 23228 leader_election.cc:302] T d1a134ba33ea4076bbaac0ae460c17ba P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:46:56.331786 23472 leader_election.cc:302] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:46:56.237605 27383 raft_consensus.cc:2781] T 28e17dcb5efe45b8b6a8c42da03b4819 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260809 15:46:56.237317 27244 raft_consensus.cc:2781] T d3e94b7e50754a81b3d72a07476cb7b9 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260809 15:46:56.376178 23193 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 6ae14a74a26f434db08c652e097385c8, TS c0f38c1908414566aa9115bb2cbdd85d -> TS 61e69695258f48baa984c9d395e8ce3f move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:46:56.264008 27413 log.cc:924] Time spent T 99bdb93503db4de4a1cfb0161ea0c07c P 67b3aeadfa7a432abe8af4d292e04ab9: Append to log took a long time: real 0.279s user 0.001s sys 0.000s
I20260809 15:46:56.238512 26757 raft_consensus.cc:2781] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260809 15:46:56.230154 27178 raft_consensus.cc:2781] T 907bacfa79d04a55a68c8dbe60056462 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Leader election won for term 5
I20260809 15:46:56.265419 27418 raft_consensus.cc:2781] T f9e7cadcd24d45fcbfa84209225e8f2d P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260809 15:46:56.380049 23522 raft_consensus.cc:1261] T d2d4f749fcfd4fb3bfb8115af3471598 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260809 15:46:56.237785 27266 raft_consensus.cc:3037] T 953eea268e17491ca76f3a555860a512 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Advancing to term 6
I20260809 15:46:56.264876 26719 raft_consensus.cc:482] T 0a51c0baaa6e432997c9d3552af4f254 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 15:46:56.332638 26399 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 3 current term: 3 is higher.
W20260809 15:46:56.381784 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 52427us
I20260809 15:46:56.382316 23273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9e7cadcd24d45fcbfa84209225e8f2d" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
W20260809 15:46:56.395030 23193 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 6ae14a74a26f434db08c652e097385c8, TS c0f38c1908414566aa9115bb2cbdd85d -> TS 61e69695258f48baa984c9d395e8ce3f move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:56.443711 23228 leader_election.cc:302] T c5c2e425a74b4e64ae70b5def750434c P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:46:56.385217 23276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "953eea268e17491ca76f3a555860a512" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:56.265086 27327 consensus_queue.cc:1035] T 4564dfc6d4094c1b9240b367ccecf3b1 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.156s
I20260809 15:46:56.319316 27334 leader_election.cc:288] T 73f6f44c259547f1b05a3a434616fa54 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 7 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
W20260809 15:46:56.385998 23472 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 54276.9us
I20260809 15:46:56.401203 23285 raft_consensus.cc:1226] T 2424fd9bd7f14e508fd106cbfda786a4 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 5. Current term is 6. Ops: []
I20260809 15:46:56.447371 23472 leader_election.cc:302] T f9e7cadcd24d45fcbfa84209225e8f2d P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:46:56.447788 23285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5712f6b6fffc4cc5a8ddfbe63e4097df" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:56.399478 27283 raft_consensus.cc:2781] T d1a134ba33ea4076bbaac0ae460c17ba P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260809 15:46:56.417435 23280 raft_consensus.cc:1226] T df20671a7b874dbb98c0ef28371c1308 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 4. Current term is 5. Ops: []
I20260809 15:46:56.265249 27372 consensus_queue.cc:1046] T a88421dba80d4d3388aa5eb62e23853b P 65c36d164b0540f099d592f38307ee6c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.156s
I20260809 15:46:56.418576 23396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a51c0baaa6e432997c9d3552af4f254" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:56.421931 23277 raft_consensus.cc:1226] T a3622f6e3e3641cea1960e3a31498bb7 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 3. Current term is 4. Ops: []
I20260809 15:46:56.239176 27116 raft_consensus.cc:504] T 30c1740ef6874df0a332fbd6fd4b03ce P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:56.239207 27222 raft_consensus.cc:504] T 73f6f44c259547f1b05a3a434616fa54 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:56.426513 27382 raft_consensus.cc:2781] T 05ad7d8fdca64e3da527280ca5be9280 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260809 15:46:56.427174 23270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee02fff2c9c48a19f43eb358a89e7b0" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:56.431623 23520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ae14a74a26f434db08c652e097385c8" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:56.469574 23270 raft_consensus.cc:2454] T cee02fff2c9c48a19f43eb358a89e7b0 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 5.
I20260809 15:46:56.469722 23520 raft_consensus.cc:2454] T 6ae14a74a26f434db08c652e097385c8 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 5.
I20260809 15:46:56.424479 23518 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44018 (request call id 239) took 1382 ms. Trace:
I20260809 15:46:56.470420 23518 rpcz_store.cc:276] 0809 15:46:55.042554 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:55.043017 (+ 463us) service_pool.cc:227] Handling call
0809 15:46:55.043308 (+ 291us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:55.700757 (+657449us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:56.424262 (+723505us) spinlock_profiling.cc:243] Waited 570 ms on lock 0x7b54003bb0d4. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e9220f51 00007f34e921f99e 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772 00007f34e764a94c 00007f34e7649d86
0809 15:46:56.424423 (+ 161us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1311422720}
I20260809 15:46:56.399935 27338 raft_consensus.cc:2781] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260809 15:46:56.273366 27210 raft_consensus.cc:504] T 563daf42d7784db195ea99c3cb21b370 P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:56.396108 27383 raft_consensus.cc:482] T 28e17dcb5efe45b8b6a8c42da03b4819 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Starting leader election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:46:56.376663 24468 raft_consensus.cc:2454] T cee02fff2c9c48a19f43eb358a89e7b0 P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 5.
I20260809 15:46:56.445868 23276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee02fff2c9c48a19f43eb358a89e7b0" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:56.427760 23283 consensus_queue.cc:260] T 4564dfc6d4094c1b9240b367ccecf3b1 P 67b3aeadfa7a432abe8af4d292e04ab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:56.449795 23286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "294d4bcc1e7a477cb56455581277a60f" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:56.402275 27244 raft_consensus.cc:482] T d3e94b7e50754a81b3d72a07476cb7b9 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Starting leader election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
I20260809 15:46:56.452595 23353 leader_election.cc:302] T 0acc47ce21c3422698d1ef7277c3ed63 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:46:56.453642 23285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0741ab44d99439abc3aa3bdd27ee160" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:56.480237 23285 raft_consensus.cc:2454] T c0741ab44d99439abc3aa3bdd27ee160 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 4.
W20260809 15:46:56.443703 23351 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.710s user 0.015s sys 0.018s
W20260809 15:46:56.482237 23351 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.748s user 0.015s sys 0.019s
I20260809 15:46:56.483366 23351 leader_election.cc:302] T d1a134ba33ea4076bbaac0ae460c17ba P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:56.373802 26981 consensus_queue.cc:1035] T 4b30f28311964a0b9149a50495e83d8f P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.054s
I20260809 15:46:56.486617 26981 raft_consensus.cc:482] T 8e4ffcf83c234ae7910ddef3ec0b4bda P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
I20260809 15:46:56.487372 26981 raft_consensus.cc:504] T 8e4ffcf83c234ae7910ddef3ec0b4bda P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:56.452137 27283 raft_consensus.cc:482] T d1a134ba33ea4076bbaac0ae460c17ba P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:56.432745 27263 leader_election.cc:288] T 6ae14a74a26f434db08c652e097385c8 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:56.452828 27347 raft_consensus.cc:2781] T c5c2e425a74b4e64ae70b5def750434c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260809 15:46:56.477888 23283 raft_consensus.cc:3037] T 4564dfc6d4094c1b9240b367ccecf3b1 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Advancing to term 5
I20260809 15:46:56.446667 27178 raft_consensus.cc:686] T 907bacfa79d04a55a68c8dbe60056462 P c0f38c1908414566aa9115bb2cbdd85d [term 5 LEADER]: Becoming Leader. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:46:56.479198 23286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e71a05ba1d4d42bd95c3f85f9313399e" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
W20260809 15:46:56.493117 23472 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 45855.2us
I20260809 15:46:56.468093 27382 raft_consensus.cc:482] T 05ad7d8fdca64e3da527280ca5be9280 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:56.478389 27338 raft_consensus.cc:482] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:46:56.381212 24451 raft_consensus.cc:2454] T 3d880de657e544468c82157769622a26 P 61e69695258f48baa984c9d395e8ce3f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 2.
I20260809 15:46:56.496480 23400 raft_consensus.cc:2454] T 73f6f44c259547f1b05a3a434616fa54 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 6.
I20260809 15:46:56.497794 23270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16ca412a0dae4feebe85dd01bf1a9173" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:56.499871 23273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73f6f44c259547f1b05a3a434616fa54" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:56.512027 23273 raft_consensus.cc:3037] T 73f6f44c259547f1b05a3a434616fa54 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Advancing to term 7
I20260809 15:46:56.490425 27216 raft_consensus.cc:2726] T d1a134ba33ea4076bbaac0ae460c17ba P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260809 15:46:56.494762 27244 raft_consensus.cc:3037] T d3e94b7e50754a81b3d72a07476cb7b9 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Advancing to term 7
I20260809 15:46:56.494156 23353 leader_election.cc:302] T 6ae14a74a26f434db08c652e097385c8 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:46:56.525385 23273 raft_consensus.cc:2454] T 73f6f44c259547f1b05a3a434616fa54 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 7.
I20260809 15:46:56.508610 24452 raft_consensus.cc:1226] T 563daf42d7784db195ea99c3cb21b370 P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Rejecting Update request from peer 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 4. Current term is 5. Ops: []
W20260809 15:46:56.497269 27429 log.cc:924] Time spent T d0721a4403a440f383318a3cea9d1442 P 65c36d164b0540f099d592f38307ee6c: Append to log took a long time: real 0.071s user 0.001s sys 0.000s
I20260809 15:46:56.480082 27383 raft_consensus.cc:3037] T 28e17dcb5efe45b8b6a8c42da03b4819 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Advancing to term 7
I20260809 15:46:56.512135 23286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e14d118b2eaa42aa8a7cd4025aba7128" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:56.509927 27263 raft_consensus.cc:2781] T 0acc47ce21c3422698d1ef7277c3ed63 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260809 15:46:56.452443 27322 consensus_queue.cc:1046] T ba56725e42f74a1baa4c04d9660727f9 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.045s
I20260809 15:46:56.516029 27338 raft_consensus.cc:3037] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Advancing to term 8
I20260809 15:46:56.507925 23276 raft_consensus.cc:2454] T cee02fff2c9c48a19f43eb358a89e7b0 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 5.
I20260809 15:46:56.549189 23276 raft_consensus.cc:1226] T ae6d2c08a7a94f58ac8dd698e3fd456c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 5. Current term is 6. Ops: []
I20260809 15:46:56.522083 27395 leader_election.cc:288] T d0721a4403a440f383318a3cea9d1442 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:56.501325 27283 raft_consensus.cc:3037] T d1a134ba33ea4076bbaac0ae460c17ba P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Advancing to term 5
I20260809 15:46:56.503320 27268 raft_consensus.cc:2781] T f9e7cadcd24d45fcbfa84209225e8f2d P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260809 15:46:56.500808 27347 raft_consensus.cc:482] T c5c2e425a74b4e64ae70b5def750434c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting leader election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:46:56.501282 24448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e14d118b2eaa42aa8a7cd4025aba7128" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:56.532722 23283 raft_consensus.cc:1261] T 4564dfc6d4094c1b9240b367ccecf3b1 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
I20260809 15:46:56.535524 23286 raft_consensus.cc:2396] T e14d118b2eaa42aa8a7cd4025aba7128 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c for term 8 because replica has last-logged OpId of term: 6 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20260809 15:46:56.537461 23399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "563daf42d7784db195ea99c3cb21b370" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:56.539664 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0da4d5283c8a46878bd39b85e5c71f66" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
W20260809 15:46:56.520147 27422 log.cc:924] Time spent T 3924b648461e463b9d7925e5671d68ae P 61e69695258f48baa984c9d395e8ce3f: Append to log took a long time: real 0.137s user 0.000s sys 0.001s
I20260809 15:46:56.452632 26719 raft_consensus.cc:504] T 0a51c0baaa6e432997c9d3552af4f254 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:56.479244 27200 consensus_queue.cc:1035] T 3924b648461e463b9d7925e5671d68ae P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 5, Time since last communication: 0.044s
I20260809 15:46:56.516624 27382 raft_consensus.cc:3037] T 05ad7d8fdca64e3da527280ca5be9280 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Advancing to term 6
I20260809 15:46:56.527382 23472 leader_election.cc:302] T 6ae14a74a26f434db08c652e097385c8 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters: 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:56.551165 23402 raft_consensus.cc:1261] T 30c1740ef6874df0a332fbd6fd4b03ce P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260809 15:46:56.527814 26747 raft_consensus.cc:2781] T 6ae14a74a26f434db08c652e097385c8 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260809 15:46:56.539659 27298 consensus_meta.cc:287] T 907bacfa79d04a55a68c8dbe60056462 P 61e69695258f48baa984c9d395e8ce3f: Time spent flushing consensus metadata: real 0.528s user 0.008s sys 0.003s
I20260809 15:46:56.553306 23521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0721a4403a440f383318a3cea9d1442" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:56.527275 27210 leader_election.cc:288] T 563daf42d7784db195ea99c3cb21b370 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 5 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
W20260809 15:46:56.545562 27420 log.cc:924] Time spent T 00a1881267a14ba8b78c81040f17a014 P 65c36d164b0540f099d592f38307ee6c: Append to log took a long time: real 0.172s user 0.000s sys 0.000s
I20260809 15:46:56.500763 27213 consensus_queue.cc:1035] T 00a1881267a14ba8b78c81040f17a014 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.033s
I20260809 15:46:56.538244 27263 raft_consensus.cc:482] T 0acc47ce21c3422698d1ef7277c3ed63 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:46:56.533253 27216 consensus_queue.cc:1035] T d2d4f749fcfd4fb3bfb8115af3471598 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.032s
W20260809 15:46:56.683506 23193 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 2424fd9bd7f14e508fd106cbfda786a4, TS 65c36d164b0540f099d592f38307ee6c -> TS 61e69695258f48baa984c9d395e8ce3f move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:56.682654 23277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "563daf42d7784db195ea99c3cb21b370" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:56.714753 23283 pending_rounds.cc:76] T 4564dfc6d4094c1b9240b367ccecf3b1 P 67b3aeadfa7a432abe8af4d292e04ab9: Aborting all ops after (but not including) 2
I20260809 15:46:56.697348 24472 raft_consensus.cc:1261] T e71a05ba1d4d42bd95c3f85f9313399e P 61e69695258f48baa984c9d395e8ce3f [term 6 FOLLOWER]: Refusing update from remote peer 67b3aeadfa7a432abe8af4d292e04ab9: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260809 15:46:56.715440 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df20671a7b874dbb98c0ef28371c1308" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
W20260809 15:46:56.763677 23193 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 2424fd9bd7f14e508fd106cbfda786a4, TS 65c36d164b0540f099d592f38307ee6c -> TS 61e69695258f48baa984c9d395e8ce3f move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:56.622184 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5712f6b6fffc4cc5a8ddfbe63e4097df" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
W20260809 15:46:56.694669 23470 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 128769us
I20260809 15:46:56.686419 23345 leader_election.cc:302] T d3be1e356b254d22b995be32f1c12f6f P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:46:56.764037 23283 pending_rounds.cc:97] T 4564dfc6d4094c1b9240b367ccecf3b1 P 67b3aeadfa7a432abe8af4d292e04ab9: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20260809 15:46:56.765730 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73f6f44c259547f1b05a3a434616fa54" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:56.764154 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e17dcb5efe45b8b6a8c42da03b4819" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:56.644315 23392 raft_consensus.cc:1226] T 763e58d6ba644eea830a6c1664d39c9b P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20260809 15:46:56.657262 23528 raft_consensus.cc:2454] T c5c2e425a74b4e64ae70b5def750434c P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 6.
I20260809 15:46:56.767251 23283 raft_consensus.cc:2866] T 4564dfc6d4094c1b9240b367ccecf3b1 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 15:46:56.589006 23273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c48a3049897452383257aa757834fb5" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:56.768741 23528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e14d118b2eaa42aa8a7cd4025aba7128" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:56.769008 23283 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53554 (request call id 1527) took 1219 ms. Trace:
I20260809 15:46:56.698588 23468 leader_election.cc:302] T cee02fff2c9c48a19f43eb358a89e7b0 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c; no voters:
I20260809 15:46:56.769454 23528 raft_consensus.cc:2454] T e14d118b2eaa42aa8a7cd4025aba7128 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 7.
I20260809 15:46:56.538771 27136 leader_election.cc:288] T 0da4d5283c8a46878bd39b85e5c71f66 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:56.770188 27334 raft_consensus.cc:2781] T cee02fff2c9c48a19f43eb358a89e7b0 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260809 15:46:56.770668 27334 raft_consensus.cc:482] T cee02fff2c9c48a19f43eb358a89e7b0 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:46:56.771153 27334 raft_consensus.cc:3037] T cee02fff2c9c48a19f43eb358a89e7b0 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Advancing to term 6
I20260809 15:46:56.770438 27298 raft_consensus.cc:504] T 907bacfa79d04a55a68c8dbe60056462 P 61e69695258f48baa984c9d395e8ce3f [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:56.771766 23519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2db057fd10e149a9895dff6ef8c9f038" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:56.772769 23519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee02fff2c9c48a19f43eb358a89e7b0" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
W20260809 15:46:56.660123 27423 log.cc:924] Time spent T f7715d71dbac447b9351fc5eb36c5e80 P 65c36d164b0540f099d592f38307ee6c: Append to log took a long time: real 0.217s user 0.000s sys 0.001s
I20260809 15:46:56.554013 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee02fff2c9c48a19f43eb358a89e7b0" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:56.768927 23273 raft_consensus.cc:2454] T 0c48a3049897452383257aa757834fb5 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 3.
I20260809 15:46:56.779742 23517 raft_consensus.cc:1226] T 3d880de657e544468c82157769622a26 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Rejecting Update request from peer 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 1. Current term is 2. Ops: []
I20260809 15:46:56.775952 23286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "907bacfa79d04a55a68c8dbe60056462" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:56.787132 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df20671a7b874dbb98c0ef28371c1308" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
W20260809 15:46:56.775471 23468 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.68:34689, user_credentials={real_user=slave}} blocked reactor thread for 76951.3us
I20260809 15:46:56.820225 23286 raft_consensus.cc:2454] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 5.
I20260809 15:46:56.799710 23396 raft_consensus.cc:2454] T 0a51c0baaa6e432997c9d3552af4f254 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 4.
I20260809 15:46:56.781145 23521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba56725e42f74a1baa4c04d9660727f9" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:56.762640 23399 raft_consensus.cc:3037] T 563daf42d7784db195ea99c3cb21b370 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Advancing to term 5
W20260809 15:46:56.800846 23472 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 273544us
I20260809 15:46:56.785600 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0acc47ce21c3422698d1ef7277c3ed63" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:56.790805 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "907bacfa79d04a55a68c8dbe60056462" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:56.819836 23528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99bdb93503db4de4a1cfb0161ea0c07c" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
W20260809 15:46:56.825129 23345 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.68:34689, user_credentials={real_user=slave}} blocked reactor thread for 138784us
I20260809 15:46:56.825941 23528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "294d4bcc1e7a477cb56455581277a60f" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:56.781195 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e4ffcf83c234ae7910ddef3ec0b4bda" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:56.653337 26981 leader_election.cc:288] T 8e4ffcf83c234ae7910ddef3ec0b4bda P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:56.822655 23526 raft_consensus.cc:2454] T 0acc47ce21c3422698d1ef7277c3ed63 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 6.
I20260809 15:46:56.769418 23283 rpcz_store.cc:276] 0809 15:46:55.549524 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:55.823085 (+273561us) service_pool.cc:227] Handling call
0809 15:46:55.831108 (+ 8023us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:56.427472 (+596364us) spinlock_profiling.cc:243] Waited 38.3 ms on lock 0x7b54002402a0. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e91b6951 00007f34e9217321 00007f34e9218b4e 00007f34e9222c89 00007f34e9223556 00007f34e921fedf 00007f34e921f99e 00007f34ee1612e9 00007f34e7847684 00007f34e7847612
0809 15:46:56.768783 (+341311us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:56.768960 (+ 177us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":88068352}
W20260809 15:46:56.780793 27431 log.cc:924] Time spent T d1a134ba33ea4076bbaac0ae460c17ba P c0f38c1908414566aa9115bb2cbdd85d: Append to log took a long time: real 0.127s user 0.000s sys 0.000s
W20260809 15:46:56.786444 25863 consensus_meta.cc:287] T 2424fd9bd7f14e508fd106cbfda786a4 P 65c36d164b0540f099d592f38307ee6c: Time spent flushing consensus metadata: real 0.658s user 0.005s sys 0.002s
I20260809 15:46:56.634579 24455 raft_consensus.cc:1226] T 3d880de657e544468c82157769622a26 P 61e69695258f48baa984c9d395e8ce3f [term 2 FOLLOWER]: Rejecting Update request from peer 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 1. Current term is 2. Ops: []
W20260809 15:46:56.867519 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 383072us
I20260809 15:46:56.608143 24451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0741ab44d99439abc3aa3bdd27ee160" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:56.659950 24453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e4ffcf83c234ae7910ddef3ec0b4bda" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:56.894109 23351 leader_election.cc:302] T 763e58d6ba644eea830a6c1664d39c9b P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:56.766268 27263 raft_consensus.cc:3037] T 0acc47ce21c3422698d1ef7277c3ed63 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Advancing to term 7
I20260809 15:46:56.766021 27347 raft_consensus.cc:3037] T c5c2e425a74b4e64ae70b5def750434c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Advancing to term 5
I20260809 15:46:56.799367 24448 raft_consensus.cc:2454] T e14d118b2eaa42aa8a7cd4025aba7128 P 61e69695258f48baa984c9d395e8ce3f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 7.
I20260809 15:46:56.774703 27298 leader_election.cc:288] T 907bacfa79d04a55a68c8dbe60056462 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 6 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:56.765950 26747 raft_consensus.cc:482] T 6ae14a74a26f434db08c652e097385c8 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:46:56.886154 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a88421dba80d4d3388aa5eb62e23853b" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:56.795346 26619 raft_consensus.cc:482] T c5c2e425a74b4e64ae70b5def750434c P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:56.789500 27322 raft_consensus.cc:2781] T 6ae14a74a26f434db08c652e097385c8 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260809 15:46:56.697628 27217 leader_election.cc:288] T d0721a4403a440f383318a3cea9d1442 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:56.559602 27268 raft_consensus.cc:482] T f9e7cadcd24d45fcbfa84209225e8f2d P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:46:56.891567 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "563daf42d7784db195ea99c3cb21b370" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:56.697546 27395 raft_consensus.cc:482] T ae6d2c08a7a94f58ac8dd698e3fd456c P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:56.780202 26730 leader_election.cc:288] T 563daf42d7784db195ea99c3cb21b370 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:56.899700 27378 raft_consensus.cc:482] T 7baa2492df0340c4a5ac241959ce1ac3 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:46:56.863355 27136 raft_consensus.cc:2781] T d3be1e356b254d22b995be32f1c12f6f P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Leader election won for term 6
I20260809 15:46:56.781534 27363 consensus_queue.cc:1046] T a3622f6e3e3641cea1960e3a31498bb7 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.112s
I20260809 15:46:56.903398 23351 leader_election.cc:302] T 2424fd9bd7f14e508fd106cbfda786a4 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:56.911351 27322 raft_consensus.cc:482] T 6ae14a74a26f434db08c652e097385c8 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:46:56.914269 27268 raft_consensus.cc:3037] T f9e7cadcd24d45fcbfa84209225e8f2d P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Advancing to term 6
I20260809 15:46:56.853180 24468 raft_consensus.cc:2454] T cee02fff2c9c48a19f43eb358a89e7b0 P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 5.
I20260809 15:46:56.780272 27334 raft_consensus.cc:504] T cee02fff2c9c48a19f43eb358a89e7b0 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:56.913936 23397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e17dcb5efe45b8b6a8c42da03b4819" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:56.595057 27178 consensus_queue.cc:237] T 907bacfa79d04a55a68c8dbe60056462 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:56.942214 23273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07fcabb569414bddbd39df6cedd24e64" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:56.942987 23273 raft_consensus.cc:2379] T 07fcabb569414bddbd39df6cedd24e64 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d in current term 6: Already voted for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in this term.
I20260809 15:46:56.943696 23273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cef3e7277d7d4760824a3436ed4a847d" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:56.945487 23397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2db057fd10e149a9895dff6ef8c9f038" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:56.595479 27394 consensus_queue.cc:1046] T df20671a7b874dbb98c0ef28371c1308 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.278s
I20260809 15:46:56.907671 27216 raft_consensus.cc:2726] T 763e58d6ba644eea830a6c1664d39c9b P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260809 15:46:56.898793 24451 raft_consensus.cc:2454] T c0741ab44d99439abc3aa3bdd27ee160 P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 4.
I20260809 15:46:56.909711 26619 raft_consensus.cc:504] T c5c2e425a74b4e64ae70b5def750434c P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:56.904186 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0741ab44d99439abc3aa3bdd27ee160" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:56.924618 23286 raft_consensus.cc:1261] T d2d4f749fcfd4fb3bfb8115af3471598 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260809 15:46:56.964131 23286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c29937a80724ddb9d07a27534a146f3" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:56.962951 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07fcabb569414bddbd39df6cedd24e64" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:56.877678 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3622f6e3e3641cea1960e3a31498bb7" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:56.964675 23286 raft_consensus.cc:2454] T 0c29937a80724ddb9d07a27534a146f3 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 5.
I20260809 15:46:56.967329 23286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba56725e42f74a1baa4c04d9660727f9" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:56.967993 23286 raft_consensus.cc:2379] T ba56725e42f74a1baa4c04d9660727f9 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d in current term 4: Already voted for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in this term.
I20260809 15:46:56.968958 23526 raft_consensus.cc:1261] T 953eea268e17491ca76f3a555860a512 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Refusing update from remote peer 67b3aeadfa7a432abe8af4d292e04ab9: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260809 15:46:56.935029 23399 raft_consensus.cc:2454] T 563daf42d7784db195ea99c3cb21b370 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 5.
I20260809 15:46:56.939657 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30c1740ef6874df0a332fbd6fd4b03ce" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:56.941679 23528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5712f6b6fffc4cc5a8ddfbe63e4097df" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:56.933049 27136 raft_consensus.cc:686] T d3be1e356b254d22b995be32f1c12f6f P c0f38c1908414566aa9115bb2cbdd85d [term 6 LEADER]: Becoming Leader. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:46:56.948607 23283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4564dfc6d4094c1b9240b367ccecf3b1" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:56.948843 23273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998c6cee72cf40228c6dba064c0db2da" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:56.951545 23397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99bdb93503db4de4a1cfb0161ea0c07c" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:56.952344 23517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3622f6e3e3641cea1960e3a31498bb7" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:56.908237 26747 raft_consensus.cc:3037] T 6ae14a74a26f434db08c652e097385c8 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Advancing to term 6
I20260809 15:46:56.933588 27322 raft_consensus.cc:3037] T 6ae14a74a26f434db08c652e097385c8 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Advancing to term 6
I20260809 15:46:56.961760 23519 raft_consensus.cc:2379] T cee02fff2c9c48a19f43eb358a89e7b0 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 in current term 6: Already voted for candidate 65c36d164b0540f099d592f38307ee6c in this term.
I20260809 15:46:56.926379 26981 raft_consensus.cc:2726] T 2424fd9bd7f14e508fd106cbfda786a4 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260809 15:46:56.947830 23277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3be1e356b254d22b995be32f1c12f6f" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:56.997287 23277 raft_consensus.cc:3037] T d3be1e356b254d22b995be32f1c12f6f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Advancing to term 6
W20260809 15:46:56.995458 24406 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 307694us
I20260809 15:46:56.919925 27378 raft_consensus.cc:504] T 7baa2492df0340c4a5ac241959ce1ac3 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:56.972880 23286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "297c82ed906d4b48b184985eed3de601" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:56.993100 23397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0acc47ce21c3422698d1ef7277c3ed63" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:56.993000 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba56725e42f74a1baa4c04d9660727f9" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:56.994303 23273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "538a228ddfb54bc6a36528d5ff07597b" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:56.994819 23283 raft_consensus.cc:2454] T 4564dfc6d4094c1b9240b367ccecf3b1 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 5.
I20260809 15:46:56.917384 27395 raft_consensus.cc:504] T ae6d2c08a7a94f58ac8dd698e3fd456c P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:56.971493 24410 leader_election.cc:302] T 563daf42d7784db195ea99c3cb21b370 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d; no voters:
W20260809 15:46:57.032347 24410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 60940.1us
I20260809 15:46:56.972040 23399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07fcabb569414bddbd39df6cedd24e64" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
W20260809 15:46:57.026711 24406 leader_election.cc:334] T 16ca412a0dae4feebe85dd01bf1a9173 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459): Timed out: RequestConsensusVote RPC to 127.20.202.65:35459 timed out after 2.094s (SENT)
W20260809 15:46:56.995668 27266 consensus_meta.cc:287] T 953eea268e17491ca76f3a555860a512 P c0f38c1908414566aa9115bb2cbdd85d: Time spent flushing consensus metadata: real 0.544s user 0.009s sys 0.000s
I20260809 15:46:57.033077 24406 leader_election.cc:302] T 16ca412a0dae4feebe85dd01bf1a9173 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:57.033531 23399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a88421dba80d4d3388aa5eb62e23853b" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:57.034965 23283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5c2e425a74b4e64ae70b5def750434c" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:57.031169 23273 raft_consensus.cc:2454] T 538a228ddfb54bc6a36528d5ff07597b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 4.
W20260809 15:46:57.042755 24406 leader_election.cc:334] T e71a05ba1d4d42bd95c3f85f9313399e P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459): Timed out: RequestConsensusVote RPC to 127.20.202.65:35459 timed out after 2.628s (SENT)
I20260809 15:46:57.043161 24406 leader_election.cc:302] T e71a05ba1d4d42bd95c3f85f9313399e P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f; no voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:57.043227 23273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3622f6e3e3641cea1960e3a31498bb7" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:57.045063 23273 raft_consensus.cc:2379] T a3622f6e3e3641cea1960e3a31498bb7 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d in current term 4: Already voted for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in this term.
I20260809 15:46:57.086189 27032 raft_consensus.cc:3037] T ae6d2c08a7a94f58ac8dd698e3fd456c P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Advancing to term 6
I20260809 15:46:57.088558 27053 raft_consensus.cc:3032] T a88421dba80d4d3388aa5eb62e23853b P 65c36d164b0540f099d592f38307ee6c [term 3 LEADER]: Stepping down as leader of term 3
I20260809 15:46:57.088922 27053 raft_consensus.cc:729] T a88421dba80d4d3388aa5eb62e23853b P 65c36d164b0540f099d592f38307ee6c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:46:57.089650 27053 consensus_queue.cc:260] T a88421dba80d4d3388aa5eb62e23853b P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:57.063138 27382 raft_consensus.cc:504] T 05ad7d8fdca64e3da527280ca5be9280 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:57.092381 23276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9e7cadcd24d45fcbfa84209225e8f2d" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:57.092835 27162 consensus_queue.cc:1035] T e71a05ba1d4d42bd95c3f85f9313399e P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260809 15:46:57.048262 27266 raft_consensus.cc:504] T 953eea268e17491ca76f3a555860a512 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:57.093967 23280 raft_consensus.cc:1226] T 05ad7d8fdca64e3da527280ca5be9280 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 5. Current term is 6. Ops: []
I20260809 15:46:57.094241 27268 raft_consensus.cc:504] T f9e7cadcd24d45fcbfa84209225e8f2d P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:57.093452 27382 leader_election.cc:288] T 05ad7d8fdca64e3da527280ca5be9280 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 6 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:57.046456 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e4ffcf83c234ae7910ddef3ec0b4bda" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:57.095278 23521 raft_consensus.cc:1226] T f9e7cadcd24d45fcbfa84209225e8f2d P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Rejecting Update request from peer 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 5. Current term is 6. Ops: []
I20260809 15:46:57.040475 27283 raft_consensus.cc:504] T d1a134ba33ea4076bbaac0ae460c17ba P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:57.096007 27372 raft_consensus.cc:3032] T a3622f6e3e3641cea1960e3a31498bb7 P 65c36d164b0540f099d592f38307ee6c [term 3 LEADER]: Stepping down as leader of term 3
I20260809 15:46:57.096370 27372 raft_consensus.cc:729] T a3622f6e3e3641cea1960e3a31498bb7 P 65c36d164b0540f099d592f38307ee6c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:46:57.097079 27283 leader_election.cc:288] T d1a134ba33ea4076bbaac0ae460c17ba P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:57.097715 23285 raft_consensus.cc:1226] T d1a134ba33ea4076bbaac0ae460c17ba P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 4. Current term is 5. Ops: []
I20260809 15:46:57.048686 23399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30c1740ef6874df0a332fbd6fd4b03ce" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:57.098420 23399 raft_consensus.cc:2454] T 30c1740ef6874df0a332fbd6fd4b03ce P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 5.
I20260809 15:46:57.101091 23399 raft_consensus.cc:1261] T f9e7cadcd24d45fcbfa84209225e8f2d P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Refusing update from remote peer 67b3aeadfa7a432abe8af4d292e04ab9: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260809 15:46:56.958309 27383 raft_consensus.cc:504] T 28e17dcb5efe45b8b6a8c42da03b4819 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:57.103475 23520 raft_consensus.cc:1226] T 28e17dcb5efe45b8b6a8c42da03b4819 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Rejecting Update request from peer c0f38c1908414566aa9115bb2cbdd85d for earlier term 6. Current term is 7. Ops: []
I20260809 15:46:57.103963 24463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "294d4bcc1e7a477cb56455581277a60f" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:57.044075 27426 raft_consensus.cc:2781] T 563daf42d7784db195ea99c3cb21b370 P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Leader election won for term 5
I20260809 15:46:57.105260 27426 raft_consensus.cc:686] T 563daf42d7784db195ea99c3cb21b370 P 61e69695258f48baa984c9d395e8ce3f [term 5 LEADER]: Becoming Leader. State: Replica: 61e69695258f48baa984c9d395e8ce3f, State: Running, Role: LEADER
I20260809 15:46:57.106156 27426 consensus_queue.cc:237] T 563daf42d7784db195ea99c3cb21b370 P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
W20260809 15:46:57.054800 27437 log.cc:924] Time spent T 6ced718c1a5c4da79cb8c1ebd1435b87 P c0f38c1908414566aa9115bb2cbdd85d: Append to log took a long time: real 0.157s user 0.001s sys 0.000s
I20260809 15:46:57.049367 27298 raft_consensus.cc:2726] T e71a05ba1d4d42bd95c3f85f9313399e P 61e69695258f48baa984c9d395e8ce3f [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260809 15:46:57.057402 23351 leader_election.cc:302] T 07fcabb569414bddbd39df6cedd24e64 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:57.058527 27334 leader_election.cc:288] T cee02fff2c9c48a19f43eb358a89e7b0 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 6 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
W20260809 15:46:57.063524 27432 log.cc:924] Time spent T cef3e7277d7d4760824a3436ed4a847d P c0f38c1908414566aa9115bb2cbdd85d: Append to log took a long time: real 0.173s user 0.000s sys 0.001s
W20260809 15:46:57.109045 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 51724.8us
I20260809 15:46:57.110049 23351 leader_election.cc:302] T cef3e7277d7d4760824a3436ed4a847d P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:57.030689 26719 leader_election.cc:288] T 0a51c0baaa6e432997c9d3552af4f254 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
W20260809 15:46:57.119246 23193 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 6ced718c1a5c4da79cb8c1ebd1435b87, TS 67b3aeadfa7a432abe8af4d292e04ab9 -> TS 61e69695258f48baa984c9d395e8ce3f move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:46:57.119601 23193 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 6ced718c1a5c4da79cb8c1ebd1435b87, TS 67b3aeadfa7a432abe8af4d292e04ab9 -> TS 61e69695258f48baa984c9d395e8ce3f move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:46:57.073452 23468 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.68:34689, user_credentials={real_user=slave}} blocked reactor thread for 252767us
I20260809 15:46:57.121313 23468 leader_election.cc:302] T 3d880de657e544468c82157769622a26 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c; no voters: 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:57.122053 27327 raft_consensus.cc:2781] T 3d880de657e544468c82157769622a26 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260809 15:46:57.049149 27434 log.cc:924] Time spent T d0721a4403a440f383318a3cea9d1442 P c0f38c1908414566aa9115bb2cbdd85d: Append to log took a long time: real 0.130s user 0.000s sys 0.001s
I20260809 15:46:57.048076 27079 raft_consensus.cc:2726] T 16ca412a0dae4feebe85dd01bf1a9173 P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260809 15:46:57.093448 23276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "563daf42d7784db195ea99c3cb21b370" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:57.128190 23276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "953eea268e17491ca76f3a555860a512" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:57.111604 23351 leader_election.cc:302] T 0c29937a80724ddb9d07a27534a146f3 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c
I20260809 15:46:57.113498 24410 leader_election.cc:302] T 0a51c0baaa6e432997c9d3552af4f254 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:46:57.111621 23404 raft_consensus.cc:2379] T 953eea268e17491ca76f3a555860a512 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c in current term 6: Already voted for candidate c0f38c1908414566aa9115bb2cbdd85d in this term.
I20260809 15:46:57.031622 23286 raft_consensus.cc:2454] T 297c82ed906d4b48b184985eed3de601 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 6.
I20260809 15:46:57.133828 23394 raft_consensus.cc:1226] T 4564dfc6d4094c1b9240b367ccecf3b1 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Rejecting Update request from peer 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260809 15:46:57.134719 23286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0da4d5283c8a46878bd39b85e5c71f66" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:57.078964 23277 raft_consensus.cc:2454] T d3be1e356b254d22b995be32f1c12f6f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 6.
I20260809 15:46:57.135124 23276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df20671a7b874dbb98c0ef28371c1308" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:57.136256 23276 raft_consensus.cc:2379] T df20671a7b874dbb98c0ef28371c1308 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d in current term 5: Already voted for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in this term.
I20260809 15:46:57.136234 23277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2db057fd10e149a9895dff6ef8c9f038" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:57.137863 23277 raft_consensus.cc:2454] T 2db057fd10e149a9895dff6ef8c9f038 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 4.
I20260809 15:46:57.139204 23353 leader_election.cc:302] T c5c2e425a74b4e64ae70b5def750434c P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:46:57.139977 23276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0acc47ce21c3422698d1ef7277c3ed63" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 7 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:57.141723 23276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a88421dba80d4d3388aa5eb62e23853b" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:57.142520 23276 raft_consensus.cc:2379] T a88421dba80d4d3388aa5eb62e23853b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d in current term 4: Already voted for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in this term.
I20260809 15:46:57.117535 26981 raft_consensus.cc:3037] T 07fcabb569414bddbd39df6cedd24e64 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Advancing to term 6
I20260809 15:46:57.143522 23276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e4ffcf83c234ae7910ddef3ec0b4bda" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
W20260809 15:46:57.123070 23345 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.68:34689, user_credentials={real_user=slave}} blocked reactor thread for 228148us
I20260809 15:46:57.132325 23351 leader_election.cc:302] T ba56725e42f74a1baa4c04d9660727f9 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
W20260809 15:46:57.118021 23470 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 264957us
I20260809 15:46:57.148941 23470 leader_election.cc:302] T 73f6f44c259547f1b05a3a434616fa54 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:46:57.119896 23273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0721a4403a440f383318a3cea9d1442" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:57.145198 23286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "953eea268e17491ca76f3a555860a512" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:57.095157 27266 leader_election.cc:288] T 953eea268e17491ca76f3a555860a512 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 6 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:57.151048 23286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "213543cdbe814c89a17a963469922c1b" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:57.123308 27447 raft_consensus.cc:2726] T cef3e7277d7d4760824a3436ed4a847d P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260809 15:46:57.125739 27327 raft_consensus.cc:482] T 3d880de657e544468c82157769622a26 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:46:57.155226 23277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ced718c1a5c4da79cb8c1ebd1435b87" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
W20260809 15:46:57.144971 23228 connection.cc:232] RPC call timeout handler was delayed by -1.5801548808813095s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16610000000000003s for a response
W20260809 15:46:57.156296 23228 connection.cc:232] RPC call timeout handler was delayed by -1.5405548808630556s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1705s for a response
W20260809 15:46:57.156623 23228 connection.cc:232] RPC call timeout handler was delayed by -1.5405548808630556s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1705s for a response
W20260809 15:46:57.156903 23228 connection.cc:232] RPC call timeout handler was delayed by -1.515354880830273s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1733s for a response
W20260809 15:46:57.157158 23228 connection.cc:232] RPC call timeout handler was delayed by -1.4181548808701336s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1841s for a response
I20260809 15:46:57.157614 23273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7715d71dbac447b9351fc5eb36c5e80" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:57.162830 23286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ae14a74a26f434db08c652e097385c8" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:57.164196 23286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a51c0baaa6e432997c9d3552af4f254" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:57.139317 26719 raft_consensus.cc:2781] T 0c29937a80724ddb9d07a27534a146f3 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260809 15:46:57.144469 27338 consensus_meta.cc:287] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 65c36d164b0540f099d592f38307ee6c: Time spent flushing consensus metadata: real 0.586s user 0.006s sys 0.002s
W20260809 15:46:57.327399 23193 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet ba56725e42f74a1baa4c04d9660727f9, TS 67b3aeadfa7a432abe8af4d292e04ab9 -> TS 61e69695258f48baa984c9d395e8ce3f move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:46:57.327785 23193 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet ba56725e42f74a1baa4c04d9660727f9, TS 67b3aeadfa7a432abe8af4d292e04ab9 -> TS 61e69695258f48baa984c9d395e8ce3f move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:57.157676 23228 leader_election.cc:302] T 4b30f28311964a0b9149a50495e83d8f P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
W20260809 15:46:57.143927 27390 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.534s user 0.010s sys 0.015s
I20260809 15:46:57.144150 27206 raft_consensus.cc:2781] T 0a51c0baaa6e432997c9d3552af4f254 P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260809 15:46:57.125216 27216 consensus_queue.cc:1035] T d2d4f749fcfd4fb3bfb8115af3471598 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.037s
I20260809 15:46:57.334259 23470 leader_election.cc:302] T e14d118b2eaa42aa8a7cd4025aba7128 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 61e69695258f48baa984c9d395e8ce3f, 67b3aeadfa7a432abe8af4d292e04ab9
W20260809 15:46:57.329816 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 172210us
W20260809 15:46:57.339085 23400 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55332 (request call id 1345) took 3778 ms (client timeout 1705 ms). Trace:
I20260809 15:46:57.341471 23395 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55320 (request call id 1392) took 1234 ms. Trace:
I20260809 15:46:57.340986 23277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30c1740ef6874df0a332fbd6fd4b03ce" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
W20260809 15:46:57.351920 23400 rpcz_store.cc:269] 0809 15:46:53.560314 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:53.560424 (+ 110us) service_pool.cc:227] Handling call
0809 15:46:57.338925 (+3778501us) spinlock_profiling.cc:243] Waited 2.91 s on lock 0x7b5400193054. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e92242ae 00007f34ee161c27 00007f34e7849c04 00007f34e7849b92 00007f34e7849ac1 00007f34e7849a3c 00007f34e7848cf2 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:46:57.339020 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6693709568}
I20260809 15:46:57.352032 23395 rpcz_store.cc:276] 0809 15:46:56.107070 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:56.318360 (+211290us) service_pool.cc:227] Handling call
0809 15:46:56.320088 (+ 1728us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 15:46:56.320268 (+ 180us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:46:56.320290 (+ 22us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 15:46:56.331429 (+ 11139us) log.cc:842] Serialized 26 byte log entry
0809 15:46:56.734045 (+402616us) raft_consensus.cc:1631] Marking committed up to 3
0809 15:46:56.734083 (+ 38us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:56.734148 (+ 65us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 15:46:57.341261 (+607113us) raft_consensus.cc:1671] finished
0809 15:46:57.341285 (+ 24us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:57.341409 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":340228,"threads_started":1,"wal-append.queue_time_us":540296}
I20260809 15:46:57.160665 27363 raft_consensus.cc:2781] T 73f6f44c259547f1b05a3a434616fa54 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Leader election won for term 7
I20260809 15:46:57.162153 27327 raft_consensus.cc:3037] T 3d880de657e544468c82157769622a26 P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:57.097110 27372 consensus_queue.cc:260] T a3622f6e3e3641cea1960e3a31498bb7 P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:57.160972 27383 leader_election.cc:288] T 28e17dcb5efe45b8b6a8c42da03b4819 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 7 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:57.375550 23228 leader_election.cc:302] T e14d118b2eaa42aa8a7cd4025aba7128 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
W20260809 15:46:57.336313 27390 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.739s user 0.010s sys 0.015s
W20260809 15:46:57.379884 24406 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 328087us
W20260809 15:46:57.374998 27414 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.002s user 0.059s sys 0.063s
I20260809 15:46:57.122912 27136 consensus_queue.cc:237] T d3be1e356b254d22b995be32f1c12f6f P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:57.324721 26719 raft_consensus.cc:482] T 0c29937a80724ddb9d07a27534a146f3 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Starting leader election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:57.386181 23228 leader_election.cc:302] T 0acc47ce21c3422698d1ef7277c3ed63 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:46:57.338865 27206 raft_consensus.cc:482] T 0a51c0baaa6e432997c9d3552af4f254 P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Starting leader election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
I20260809 15:46:57.350461 27447 raft_consensus.cc:3037] T ba56725e42f74a1baa4c04d9660727f9 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:46:57.339082 24463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a51c0baaa6e432997c9d3552af4f254" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
W20260809 15:46:57.389361 24410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 256839us
I20260809 15:46:57.342229 27476 raft_consensus.cc:2781] T c5c2e425a74b4e64ae70b5def750434c P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Leader election won for term 6
I20260809 15:46:57.384318 27334 raft_consensus.cc:482] T d2d4f749fcfd4fb3bfb8115af3471598 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:57.342092 27338 raft_consensus.cc:504] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
W20260809 15:46:57.389402 27414 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.057s user 0.060s sys 0.063s
I20260809 15:46:57.388988 27382 raft_consensus.cc:2781] T e14d118b2eaa42aa8a7cd4025aba7128 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260809 15:46:57.419796 27347 consensus_meta.cc:287] T c5c2e425a74b4e64ae70b5def750434c P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent flushing consensus metadata: real 0.501s user 0.008s sys 0.000s
I20260809 15:46:57.387986 27363 raft_consensus.cc:686] T 73f6f44c259547f1b05a3a434616fa54 P 65c36d164b0540f099d592f38307ee6c [term 7 LEADER]: Becoming Leader. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
W20260809 15:46:57.429490 23353 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 276865us
I20260809 15:46:57.411604 26719 raft_consensus.cc:3037] T 0c29937a80724ddb9d07a27534a146f3 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Advancing to term 6
I20260809 15:46:57.430627 27283 raft_consensus.cc:2726] T 4b30f28311964a0b9149a50495e83d8f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 15:46:57.428733 27395 leader_election.cc:288] T ae6d2c08a7a94f58ac8dd698e3fd456c P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:57.427424 27382 raft_consensus.cc:482] T e14d118b2eaa42aa8a7cd4025aba7128 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:46:57.429960 27372 raft_consensus.cc:3037] T a3622f6e3e3641cea1960e3a31498bb7 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:46:57.420534 27206 raft_consensus.cc:3037] T 0a51c0baaa6e432997c9d3552af4f254 P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Advancing to term 5
W20260809 15:46:57.445604 23228 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.20.202.66:36263 timed out after 1.705s (SENT) [suppressed 7 similar messages]
I20260809 15:46:57.412663 27383 raft_consensus.cc:2726] T e14d118b2eaa42aa8a7cd4025aba7128 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260809 15:46:57.430148 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 297888us
I20260809 15:46:57.468096 23351 leader_election.cc:302] T 998c6cee72cf40228c6dba064c0db2da P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
W20260809 15:46:57.439998 23223 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.68:34689, user_credentials={real_user=slave}} blocked reactor thread for 392146us
I20260809 15:46:57.442327 23392 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55320 (request call id 1402) took 1222 ms. Trace:
I20260809 15:46:57.439620 27268 leader_election.cc:288] T f9e7cadcd24d45fcbfa84209225e8f2d P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 6 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:57.469156 23351 leader_election.cc:302] T 4564dfc6d4094c1b9240b367ccecf3b1 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c
W20260809 15:46:57.471632 23470 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 137446us
I20260809 15:46:57.426786 27334 raft_consensus.cc:504] T d2d4f749fcfd4fb3bfb8115af3471598 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:57.470917 23392 rpcz_store.cc:276] 0809 15:46:56.220035 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:56.809628 (+589593us) service_pool.cc:227] Handling call
0809 15:46:56.810455 (+ 827us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 15:46:56.864377 (+ 53922us) raft_consensus.cc:1520] Early marking committed up to index 2
0809 15:46:56.864405 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 15:46:57.066090 (+201685us) spinlock_profiling.cc:243] Waited 5.92 ms on lock 0x7b2c0005c9b8. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e7c03e11 00007f34e7c13442 00007f34e925bc11 00007f34e922058a 00007f34e921f99e 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772
0809 15:46:57.066415 (+ 325us) log.cc:842] Serialized 26 byte log entry
0809 15:46:57.393873 (+327458us) raft_consensus.cc:1631] Marking committed up to 2
0809 15:46:57.393916 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:57.393979 (+ 63us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 15:46:57.442073 (+ 48094us) raft_consensus.cc:1671] finished
0809 15:46:57.442092 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:57.442272 (+ 180us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":55820,"raft.queue_time_us":58172,"raft.run_cpu_time_us":114,"raft.run_wall_time_us":10560,"spinlock_wait_cycles":13624192,"thread_start_us":327104,"threads_started":1,"wal-append.queue_time_us":374208}
W20260809 15:46:57.476300 23472 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 313868us
I20260809 15:46:57.464471 27382 raft_consensus.cc:3037] T e14d118b2eaa42aa8a7cd4025aba7128 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Advancing to term 8
I20260809 15:46:57.478329 23472 leader_election.cc:302] T 3924b648461e463b9d7925e5671d68ae P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:57.439399 27347 raft_consensus.cc:504] T c5c2e425a74b4e64ae70b5def750434c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:57.484256 23351 leader_election.cc:302] T 538a228ddfb54bc6a36528d5ff07597b P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c
W20260809 15:46:57.485690 27468 consensus_queue.cc:1159] T a3622f6e3e3641cea1960e3a31498bb7 P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "c0f38c1908414566aa9115bb2cbdd85d" responder_term: 3 status { last_received { term: 3 index: 2 } last_committed_idx: 2 last_received_current_leader { term: 3 index: 2 } }
I20260809 15:46:57.486428 27477 consensus_queue.cc:1046] T 3d880de657e544468c82157769622a26 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.066s
I20260809 15:46:57.466727 27378 leader_election.cc:288] T 7baa2492df0340c4a5ac241959ce1ac3 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:57.489198 23472 leader_election.cc:302] T 7baa2492df0340c4a5ac241959ce1ac3 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:57.493076 23273 raft_consensus.cc:1226] T 07fcabb569414bddbd39df6cedd24e64 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 5. Current term is 6. Ops: []
I20260809 15:46:57.493840 23520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0721a4403a440f383318a3cea9d1442" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:57.501003 23273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e17dcb5efe45b8b6a8c42da03b4819" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:57.502213 23273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9e7cadcd24d45fcbfa84209225e8f2d" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:57.498055 23351 leader_election.cc:302] T a3622f6e3e3641cea1960e3a31498bb7 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:57.499075 23286 raft_consensus.cc:1226] T ba56725e42f74a1baa4c04d9660727f9 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 3. Current term is 4. Ops: []
I20260809 15:46:57.504601 23286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee02fff2c9c48a19f43eb358a89e7b0" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:57.505146 23286 raft_consensus.cc:3037] T cee02fff2c9c48a19f43eb358a89e7b0 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Advancing to term 6
I20260809 15:46:57.509507 23283 raft_consensus.cc:3037] T c5c2e425a74b4e64ae70b5def750434c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Advancing to term 6
I20260809 15:46:57.512727 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1a134ba33ea4076bbaac0ae460c17ba" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:57.520234 23517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05ad7d8fdca64e3da527280ca5be9280" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:57.522811 23283 raft_consensus.cc:2454] T c5c2e425a74b4e64ae70b5def750434c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 6.
I20260809 15:46:57.528072 26619 leader_election.cc:288] T c5c2e425a74b4e64ae70b5def750434c P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:57.529496 26619 raft_consensus.cc:482] T 00a1881267a14ba8b78c81040f17a014 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:46:57.529930 26619 raft_consensus.cc:504] T 00a1881267a14ba8b78c81040f17a014 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:57.484723 26757 raft_consensus.cc:482] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Starting leader election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
W20260809 15:46:57.506433 27450 log.cc:924] Time spent T 4b30f28311964a0b9149a50495e83d8f P 65c36d164b0540f099d592f38307ee6c: Append to log took a long time: real 0.062s user 0.000s sys 0.000s
I20260809 15:46:57.533370 23223 leader_election.cc:302] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c
I20260809 15:46:57.505944 27459 raft_consensus.cc:482] T 2424fd9bd7f14e508fd106cbfda786a4 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:57.546461 23223 leader_election.cc:302] T cee02fff2c9c48a19f43eb358a89e7b0 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
W20260809 15:46:57.532596 26747 consensus_meta.cc:287] T 6ae14a74a26f434db08c652e097385c8 P c0f38c1908414566aa9115bb2cbdd85d: Time spent flushing consensus metadata: real 0.520s user 0.007s sys 0.001s
I20260809 15:46:57.506340 27462 raft_consensus.cc:2781] T 0acc47ce21c3422698d1ef7277c3ed63 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260809 15:46:57.547695 23286 raft_consensus.cc:2454] T cee02fff2c9c48a19f43eb358a89e7b0 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 6.
I20260809 15:46:57.524299 27378 raft_consensus.cc:3032] T 3d880de657e544468c82157769622a26 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 LEADER]: Stepping down as leader of term 1
W20260809 15:46:57.555239 27410 log.cc:924] Time spent T 7baa2492df0340c4a5ac241959ce1ac3 P 65c36d164b0540f099d592f38307ee6c: Append to log took a long time: real 1.415s user 0.000s sys 0.014s
I20260809 15:46:57.552106 27492 raft_consensus.cc:2726] T 3924b648461e463b9d7925e5671d68ae P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260809 15:46:57.552618 27347 leader_election.cc:288] T c5c2e425a74b4e64ae70b5def750434c P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:57.545756 27395 raft_consensus.cc:2726] T 998c6cee72cf40228c6dba064c0db2da P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260809 15:46:57.553435 26619 leader_election.cc:288] T 00a1881267a14ba8b78c81040f17a014 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:57.499235 27470 consensus_queue.cc:1046] T 28e17dcb5efe45b8b6a8c42da03b4819 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.138s
I20260809 15:46:57.491070 27268 consensus_queue.cc:1035] T 4564dfc6d4094c1b9240b367ccecf3b1 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.050s
I20260809 15:46:57.505604 27483 consensus_queue.cc:1035] T 953eea268e17491ca76f3a555860a512 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.094s
I20260809 15:46:57.537628 26757 raft_consensus.cc:3037] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Advancing to term 7
I20260809 15:46:57.581894 27490 raft_consensus.cc:2781] T 538a228ddfb54bc6a36528d5ff07597b P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260809 15:46:57.556348 27458 log.cc:924] Time spent T 4b30f28311964a0b9149a50495e83d8f P c0f38c1908414566aa9115bb2cbdd85d: Append to log took a long time: real 0.137s user 0.000s sys 0.001s
W20260809 15:46:57.555621 27472 log.cc:924] Time spent T d0721a4403a440f383318a3cea9d1442 P 67b3aeadfa7a432abe8af4d292e04ab9: Append to log took a long time: real 0.134s user 0.000s sys 0.001s
I20260809 15:46:57.546844 27477 raft_consensus.cc:2726] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260809 15:46:57.582239 23392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0721a4403a440f383318a3cea9d1442" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:57.583549 23395 raft_consensus.cc:1226] T 563daf42d7784db195ea99c3cb21b370 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Rejecting Update request from peer 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 4. Current term is 5. Ops: []
I20260809 15:46:57.550238 27383 raft_consensus.cc:2726] T 7baa2492df0340c4a5ac241959ce1ac3 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260809 15:46:57.571527 26747 raft_consensus.cc:504] T 6ae14a74a26f434db08c652e097385c8 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:57.585235 24406 leader_election.cc:302] T 0c48a3049897452383257aa757834fb5 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c
I20260809 15:46:57.586937 23522 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56766 (request call id 1944) took 1090 ms. Trace:
I20260809 15:46:57.572835 27462 raft_consensus.cc:482] T 0acc47ce21c3422698d1ef7277c3ed63 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:57.500399 27363 consensus_queue.cc:237] T 73f6f44c259547f1b05a3a434616fa54 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:57.588414 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7baa2492df0340c4a5ac241959ce1ac3" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:57.648710 23522 rpcz_store.cc:276] 0809 15:46:56.496243 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:56.496635 (+ 392us) service_pool.cc:227] Handling call
0809 15:46:56.510617 (+ 13982us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 15:46:56.521793 (+ 11176us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:46:56.521822 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 15:46:56.522205 (+ 383us) log.cc:842] Serialized 26 byte log entry
0809 15:46:57.027121 (+504916us) raft_consensus.cc:1631] Marking committed up to 2
0809 15:46:57.027156 (+ 35us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:57.027216 (+ 60us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 15:46:57.586762 (+559546us) raft_consensus.cc:1671] finished
0809 15:46:57.586780 (+ 18us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:57.586884 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":20933,"raft.queue_time_us":18433,"raft.run_cpu_time_us":105,"raft.run_wall_time_us":11061,"thread_start_us":474040,"threads_started":1,"wal-append.queue_time_us":601322}
W20260809 15:46:57.549639 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 104060us
I20260809 15:46:57.652765 23228 leader_election.cc:302] T df20671a7b874dbb98c0ef28371c1308 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:57.653847 23228 leader_election.cc:302] T 28e17dcb5efe45b8b6a8c42da03b4819 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:57.575840 27378 raft_consensus.cc:729] T 3d880de657e544468c82157769622a26 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Running, Role: LEADER
W20260809 15:46:57.592461 23472 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 103329us
I20260809 15:46:57.509991 27322 raft_consensus.cc:504] T 6ae14a74a26f434db08c652e097385c8 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:57.596767 23394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05ad7d8fdca64e3da527280ca5be9280" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:57.660684 23394 raft_consensus.cc:3037] T 05ad7d8fdca64e3da527280ca5be9280 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Advancing to term 6
W20260809 15:46:57.660959 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 162978us
I20260809 15:46:57.550376 27206 raft_consensus.cc:504] T 0a51c0baaa6e432997c9d3552af4f254 P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:57.550872 27459 raft_consensus.cc:504] T 2424fd9bd7f14e508fd106cbfda786a4 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:57.600525 23399 raft_consensus.cc:1226] T 953eea268e17491ca76f3a555860a512 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Rejecting Update request from peer 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 5. Current term is 6. Ops: []
I20260809 15:46:57.600203 23404 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55320 (request call id 1395) took 1466 ms. Trace:
I20260809 15:46:57.587817 27338 leader_election.cc:288] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 8 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:57.591137 27347 raft_consensus.cc:2741] T cee02fff2c9c48a19f43eb358a89e7b0 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20260809 15:46:57.680575 27418 raft_consensus.cc:482] T f9e7cadcd24d45fcbfa84209225e8f2d P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:46:57.681005 27418 raft_consensus.cc:3037] T f9e7cadcd24d45fcbfa84209225e8f2d P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Advancing to term 6
W20260809 15:46:57.646919 24406 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 61754.4us
I20260809 15:46:57.680593 27347 raft_consensus.cc:2726] T 28e17dcb5efe45b8b6a8c42da03b4819 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260809 15:46:57.592466 27032 consensus_meta.cc:287] T ae6d2c08a7a94f58ac8dd698e3fd456c P 65c36d164b0540f099d592f38307ee6c: Time spent flushing consensus metadata: real 0.506s user 0.007s sys 0.003s
I20260809 15:46:57.594645 23393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1a134ba33ea4076bbaac0ae460c17ba" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
W20260809 15:46:57.687588 23223 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.68:34689, user_credentials={real_user=slave}} blocked reactor thread for 141182us
I20260809 15:46:57.688218 23223 leader_election.cc:302] T c0741ab44d99439abc3aa3bdd27ee160 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c
I20260809 15:46:57.593989 27499 raft_consensus.cc:3037] T a3622f6e3e3641cea1960e3a31498bb7 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:46:57.595284 24463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee02fff2c9c48a19f43eb358a89e7b0" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f"
W20260809 15:46:57.648485 27244 consensus_meta.cc:287] T d3e94b7e50754a81b3d72a07476cb7b9 P 65c36d164b0540f099d592f38307ee6c: Time spent flushing consensus metadata: real 1.115s user 0.004s sys 0.007s
I20260809 15:46:57.596220 27490 raft_consensus.cc:482] T 538a228ddfb54bc6a36528d5ff07597b P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Starting leader election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:57.672186 23404 rpcz_store.cc:276] 0809 15:46:56.133705 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:56.219710 (+ 86005us) service_pool.cc:227] Handling call
0809 15:46:57.600004 (+1380294us) spinlock_profiling.cc:243] Waited 792 ms on lock 0x7b54002ff154. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e92242ae 00007f34ee161c27 00007f34e7849c04 00007f34e7849b92 00007f34e7849ac1 00007f34e7849a3c 00007f34e7848cf2 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:46:57.600143 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1821174656}
I20260809 15:46:57.674104 27506 raft_consensus.cc:2726] T df20671a7b874dbb98c0ef28371c1308 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260809 15:46:57.651707 27302 raft_consensus.cc:2781] T 0c48a3049897452383257aa757834fb5 P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260809 15:46:57.675927 27334 leader_election.cc:288] T d2d4f749fcfd4fb3bfb8115af3471598 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
W20260809 15:46:57.660277 27263 consensus_meta.cc:287] T 0acc47ce21c3422698d1ef7277c3ed63 P c0f38c1908414566aa9115bb2cbdd85d: Time spent flushing consensus metadata: real 0.750s user 0.008s sys 0.002s
I20260809 15:46:57.676441 27322 leader_election.cc:288] T 6ae14a74a26f434db08c652e097385c8 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 6 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:57.677721 26747 leader_election.cc:288] T 6ae14a74a26f434db08c652e097385c8 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 6 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
W20260809 15:46:57.678267 27501 log.cc:924] Time spent T 907bacfa79d04a55a68c8dbe60056462 P c0f38c1908414566aa9115bb2cbdd85d: Append to log took a long time: real 0.094s user 0.000s sys 0.000s
I20260809 15:46:57.591288 27479 consensus_queue.cc:1046] T 3d880de657e544468c82157769622a26 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.094s
I20260809 15:46:57.685176 23526 raft_consensus.cc:1226] T 6ae14a74a26f434db08c652e097385c8 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Rejecting Update request from peer 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 5. Current term is 6. Ops: []
I20260809 15:46:57.662264 23351 leader_election.cc:302] T 563daf42d7784db195ea99c3cb21b370 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 61e69695258f48baa984c9d395e8ce3f, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:57.690078 23223 leader_election.cc:302] T 294d4bcc1e7a477cb56455581277a60f P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c
I20260809 15:46:57.691625 23529 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56766 (request call id 1911) took 2578 ms. Trace:
I20260809 15:46:57.651743 27462 raft_consensus.cc:3037] T 0acc47ce21c3422698d1ef7277c3ed63 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Advancing to term 6
I20260809 15:46:57.650000 27476 raft_consensus.cc:686] T c5c2e425a74b4e64ae70b5def750434c P c0f38c1908414566aa9115bb2cbdd85d [term 6 LEADER]: Becoming Leader. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
W20260809 15:46:57.713572 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 59793.5us
I20260809 15:46:57.709347 27347 raft_consensus.cc:2781] T c0741ab44d99439abc3aa3bdd27ee160 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260809 15:46:57.730510 23472 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 44013.2us
I20260809 15:46:57.729444 23529 rpcz_store.cc:276] 0809 15:46:55.113514 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:55.113633 (+ 119us) service_pool.cc:227] Handling call
0809 15:46:55.206691 (+ 93058us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 15:46:55.645477 (+438786us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:46:55.645507 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0809 15:46:55.700239 (+ 54732us) log.cc:842] Serialized 52 byte log entry
0809 15:46:55.839029 (+138790us) raft_consensus.cc:1631] Marking committed up to 1
0809 15:46:55.839068 (+ 39us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:55.839127 (+ 59us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 15:46:57.554719 (+1715592us) spinlock_profiling.cc:243] Waited 22 ms on lock 0x7b44000232cc. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e613cf11 00007f34e620aa95 00007f34e620b4b2 00007f34e8eb3511 00007f34e8eaee1a 00007f34e8ea2bee 00007f34e8ea1f36 00007f34e8ea1b58 00007f34e8eab551 00007f34e8eab509 00007f34e8eab499
0809 15:46:57.685510 (+130791us) raft_consensus.cc:1671] finished
0809 15:46:57.685531 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:57.691556 (+ 6025us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":234078,"raft.queue_time_us":251545,"raft.run_cpu_time_us":381,"raft.run_wall_time_us":171263,"spinlock_wait_cycles":50541312,"thread_start_us":311291,"threads_started":2,"wal-append.queue_time_us":290087}
I20260809 15:46:57.712399 24451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a823d35f4f348d1b52ba9e6d80b00d2" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f"
I20260809 15:46:57.713709 27302 raft_consensus.cc:482] T 0c48a3049897452383257aa757834fb5 P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Starting leader election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:57.732414 26757 raft_consensus.cc:504] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:58.072219 27537 raft_consensus.cc:482] T 3d880de657e544468c82157769622a26 P 61e69695258f48baa984c9d395e8ce3f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:58.073032 27537 raft_consensus.cc:504] T 3d880de657e544468c82157769622a26 P 61e69695258f48baa984c9d395e8ce3f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:57.731099 27032 consensus_queue.cc:1035] T 30c1740ef6874df0a332fbd6fd4b03ce P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20260809 15:46:57.742446 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 80246.8us
I20260809 15:46:58.076318 23351 leader_election.cc:302] T 297c82ed906d4b48b184985eed3de601 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c
I20260809 15:46:58.078158 23351 leader_election.cc:302] T 2db057fd10e149a9895dff6ef8c9f038 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c
I20260809 15:46:57.712761 27490 raft_consensus.cc:3037] T 538a228ddfb54bc6a36528d5ff07597b P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Advancing to term 5
I20260809 15:46:57.743120 23228 leader_election.cc:302] T 2db057fd10e149a9895dff6ef8c9f038 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:57.728305 27216 raft_consensus.cc:2781] T 4564dfc6d4094c1b9240b367ccecf3b1 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260809 15:46:58.038954 23223 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.68:34689, user_credentials={real_user=slave}} blocked reactor thread for 348929us
I20260809 15:46:57.862771 27116 leader_election.cc:288] T 30c1740ef6874df0a332fbd6fd4b03ce P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:58.046556 27382 raft_consensus.cc:504] T e14d118b2eaa42aa8a7cd4025aba7128 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:57.737946 26747 raft_consensus.cc:2726] T 563daf42d7784db195ea99c3cb21b370 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260809 15:46:57.985453 27479 raft_consensus.cc:482] T ae6d2c08a7a94f58ac8dd698e3fd456c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:58.076957 27302 raft_consensus.cc:3037] T 0c48a3049897452383257aa757834fb5 P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Advancing to term 4
W20260809 15:46:57.947304 27505 log.cc:924] Time spent T 563daf42d7784db195ea99c3cb21b370 P 61e69695258f48baa984c9d395e8ce3f: Append to log took a long time: real 0.320s user 0.001s sys 0.000s
W20260809 15:46:57.849483 27500 log.cc:924] Time spent T 0a51c0baaa6e432997c9d3552af4f254 P c0f38c1908414566aa9115bb2cbdd85d: Append to log took a long time: real 0.119s user 0.000s sys 0.000s
I20260809 15:46:57.761205 24451 raft_consensus.cc:3037] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 61e69695258f48baa984c9d395e8ce3f [term 7 FOLLOWER]: Advancing to term 8
W20260809 15:46:57.831408 23472 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 78398.6us
I20260809 15:46:58.024112 27476 consensus_queue.cc:237] T c5c2e425a74b4e64ae70b5def750434c P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:57.726410 27263 raft_consensus.cc:504] T 0acc47ce21c3422698d1ef7277c3ed63 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:57.886941 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae6d2c08a7a94f58ac8dd698e3fd456c" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:57.745306 27483 raft_consensus.cc:2726] T 294d4bcc1e7a477cb56455581277a60f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260809 15:46:57.838388 23394 raft_consensus.cc:2454] T 05ad7d8fdca64e3da527280ca5be9280 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 6.
I20260809 15:46:57.711843 27244 raft_consensus.cc:504] T d3e94b7e50754a81b3d72a07476cb7b9 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:58.039602 23517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "953eea268e17491ca76f3a555860a512" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:58.144411 23277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2424fd9bd7f14e508fd106cbfda786a4" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 7 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:58.144933 23517 raft_consensus.cc:3037] T 953eea268e17491ca76f3a555860a512 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Advancing to term 6
I20260809 15:46:58.145036 23277 raft_consensus.cc:2454] T 2424fd9bd7f14e508fd106cbfda786a4 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 6.
I20260809 15:46:57.735074 27477 consensus_queue.cc:1046] T 563daf42d7784db195ea99c3cb21b370 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.102s
I20260809 15:46:57.955502 27206 leader_election.cc:288] T 0a51c0baaa6e432997c9d3552af4f254 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 5 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:57.912842 23520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ae14a74a26f434db08c652e097385c8" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:57.764468 27383 consensus_queue.cc:1046] T 763e58d6ba644eea830a6c1664d39c9b P 65c36d164b0540f099d592f38307ee6c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.103s
W20260809 15:46:58.020658 27372 consensus_meta.cc:287] T a3622f6e3e3641cea1960e3a31498bb7 P 65c36d164b0540f099d592f38307ee6c: Time spent flushing consensus metadata: real 0.552s user 0.005s sys 0.005s
I20260809 15:46:57.745411 27347 raft_consensus.cc:482] T c0741ab44d99439abc3aa3bdd27ee160 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:58.166813 27347 raft_consensus.cc:3037] T c0741ab44d99439abc3aa3bdd27ee160 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Advancing to term 5
I20260809 15:46:58.168205 27222 leader_election.cc:288] T 73f6f44c259547f1b05a3a434616fa54 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:58.169013 27483 raft_consensus.cc:482] T 2424fd9bd7f14e508fd106cbfda786a4 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:58.169618 27483 raft_consensus.cc:504] T 2424fd9bd7f14e508fd106cbfda786a4 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:46:57.956849 27502 log.cc:924] Time spent T 6ced718c1a5c4da79cb8c1ebd1435b87 P 65c36d164b0540f099d592f38307ee6c: Append to log took a long time: real 0.230s user 0.000s sys 0.001s
I20260809 15:46:57.969508 23404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a51c0baaa6e432997c9d3552af4f254" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:58.043242 27518 raft_consensus.cc:482] T 3296ef50be254fc29cfc77d22a8f7a23 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
I20260809 15:46:58.036995 27499 raft_consensus.cc:2726] T a3622f6e3e3641cea1960e3a31498bb7 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260809 15:46:57.959721 27484 log.cc:924] Time spent T 3924b648461e463b9d7925e5671d68ae P 65c36d164b0540f099d592f38307ee6c: Append to log took a long time: real 0.363s user 0.001s sys 0.000s
I20260809 15:46:58.174456 23517 raft_consensus.cc:2454] T 953eea268e17491ca76f3a555860a512 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 6.
I20260809 15:46:57.887151 23529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30c1740ef6874df0a332fbd6fd4b03ce" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:58.177577 23517 raft_consensus.cc:1226] T 28e17dcb5efe45b8b6a8c42da03b4819 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Rejecting Update request from peer c0f38c1908414566aa9115bb2cbdd85d for earlier term 6. Current term is 7. Ops: []
I20260809 15:46:58.179571 23528 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44018 (request call id 249) took 1215 ms. Trace:
W20260809 15:46:58.180683 23472 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 61784.8us
I20260809 15:46:58.179952 23528 rpcz_store.cc:276] 0809 15:46:56.963788 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:56.974151 (+ 10363us) service_pool.cc:227] Handling call
0809 15:46:56.997676 (+ 23525us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 15:46:56.998026 (+ 350us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:46:56.998052 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0809 15:46:57.027620 (+ 29568us) log.cc:842] Serialized 52 byte log entry
0809 15:46:57.440251 (+412631us) raft_consensus.cc:1631] Marking committed up to 9
0809 15:46:57.462805 (+ 22554us) log.cc:842] Serialized 14 byte log entry
0809 15:46:57.494907 (+ 32102us) log.cc:842] Serialized 14 byte log entry
0809 15:46:57.550264 (+ 55357us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:57.550408 (+ 144us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 15:46:58.179348 (+628940us) raft_consensus.cc:1671] finished
0809 15:46:58.179370 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:58.179512 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":23024,"prepare.queue_time_us":298628,"prepare.run_cpu_time_us":82,"prepare.run_wall_time_us":3062,"raft.queue_time_us":92043,"raft.run_cpu_time_us":212,"raft.run_wall_time_us":25176,"spinlock_wait_cycles":1592192,"thread_start_us":414419,"threads_started":2,"wal-append.queue_time_us":496049}
I20260809 15:46:57.712880 24463 raft_consensus.cc:3037] T cee02fff2c9c48a19f43eb358a89e7b0 P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Advancing to term 6
I20260809 15:46:58.182046 23472 leader_election.cc:302] T 953eea268e17491ca76f3a555860a512 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d
W20260809 15:46:58.187425 23468 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.68:34689, user_credentials={real_user=slave}} blocked reactor thread for 74138.8us
I20260809 15:46:58.079730 23351 leader_election.cc:302] T df20671a7b874dbb98c0ef28371c1308 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
W20260809 15:46:58.192114 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 112855us
I20260809 15:46:58.193323 23351 leader_election.cc:302] T a88421dba80d4d3388aa5eb62e23853b P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:58.194480 23351 leader_election.cc:302] T 0da4d5283c8a46878bd39b85e5c71f66 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:58.194725 27493 raft_consensus.cc:3037] T df20671a7b874dbb98c0ef28371c1308 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Advancing to term 5
I20260809 15:46:58.195214 27116 raft_consensus.cc:3037] T a88421dba80d4d3388aa5eb62e23853b P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Advancing to term 4
W20260809 15:46:58.195401 23470 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 48807us
W20260809 15:46:58.081143 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 338062us
I20260809 15:46:58.197619 23397 raft_consensus.cc:2354] T 0acc47ce21c3422698d1ef7277c3ed63 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 6. Current term is 7.
I20260809 15:46:58.082537 23286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5c2e425a74b4e64ae70b5def750434c" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:58.198567 23518 raft_consensus.cc:1226] T d3e94b7e50754a81b3d72a07476cb7b9 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Rejecting Update request from peer 61e69695258f48baa984c9d395e8ce3f for earlier term 6. Current term is 7. Ops: []
I20260809 15:46:58.083616 23274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ae14a74a26f434db08c652e097385c8" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:58.084419 23273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2d4f749fcfd4fb3bfb8115af3471598" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
W20260809 15:46:58.040405 26719 consensus_meta.cc:287] T 0c29937a80724ddb9d07a27534a146f3 P c0f38c1908414566aa9115bb2cbdd85d: Time spent flushing consensus metadata: real 0.590s user 0.009s sys 0.001s
I20260809 15:46:58.074225 26757 leader_election.cc:288] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 7 election: Requested vote from peers 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:58.107702 23270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00a1881267a14ba8b78c81040f17a014" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:58.203575 23270 raft_consensus.cc:2454] T 00a1881267a14ba8b78c81040f17a014 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 7.
I20260809 15:46:58.085801 27459 leader_election.cc:288] T 2424fd9bd7f14e508fd106cbfda786a4 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:58.094573 27470 raft_consensus.cc:2781] T 2db057fd10e149a9895dff6ef8c9f038 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260809 15:46:58.142552 23527 raft_consensus.cc:2454] T ae6d2c08a7a94f58ac8dd698e3fd456c P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 6.
I20260809 15:46:58.200989 23353 leader_election.cc:302] T 953eea268e17491ca76f3a555860a512 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters: 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:58.088835 27506 raft_consensus.cc:2726] T 2db057fd10e149a9895dff6ef8c9f038 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260809 15:46:58.087438 27516 raft_consensus.cc:2781] T 297c82ed906d4b48b184985eed3de601 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260809 15:46:58.152714 24410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 166301us
I20260809 15:46:58.110778 27011 raft_consensus.cc:482] T 4bcac4b24e7a4448abdc934bba5a624e P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
I20260809 15:46:58.199525 23273 raft_consensus.cc:2454] T d2d4f749fcfd4fb3bfb8115af3471598 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 5.
I20260809 15:46:58.197592 23228 leader_election.cc:302] T a3622f6e3e3641cea1960e3a31498bb7 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:58.205569 23528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73f6f44c259547f1b05a3a434616fa54" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:58.205897 23520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a823d35f4f348d1b52ba9e6d80b00d2" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:58.207741 23351 leader_election.cc:302] T 6ced718c1a5c4da79cb8c1ebd1435b87 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:58.217984 23520 raft_consensus.cc:2354] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d for earlier term 7. Current term is 8.
I20260809 15:46:58.218566 23351 leader_election.cc:302] T d0721a4403a440f383318a3cea9d1442 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:58.221661 24475 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38734 (request call id 319) took 1803 ms. Trace:
I20260809 15:46:58.223673 24475 rpcz_store.cc:276] 0809 15:46:56.418533 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:56.418642 (+ 109us) service_pool.cc:227] Handling call
0809 15:46:58.221057 (+1802415us) spinlock_profiling.cc:243] Waited 2.08 ms on lock 0x7b5800005498. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e613cf11 00007f34e62c0851 00007f34e62c0702 00007f34ef1b0b09 00007f34ee0ec730 00007f34ee1b279b 00007f34ee1b2821 00007f34ee1762b0 00007f34ee1617f9 00007f34ee1612a2 00007f34e7847684
0809 15:46:58.221238 (+ 181us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:58.221393 (+ 155us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:46:58.221414 (+ 21us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 15:46:58.221448 (+ 34us) raft_consensus.cc:1631] Marking committed up to 6
0809 15:46:58.221477 (+ 29us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:58.221525 (+ 48us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:58.221612 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4785920}
W20260809 15:46:58.212585 27500 log.cc:924] Time spent T 0a51c0baaa6e432997c9d3552af4f254 P c0f38c1908414566aa9115bb2cbdd85d: Append to log took a long time: real 0.052s user 0.000s sys 0.000s
I20260809 15:46:58.122984 27479 raft_consensus.cc:504] T ae6d2c08a7a94f58ac8dd698e3fd456c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:58.242130 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a823d35f4f348d1b52ba9e6d80b00d2" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f"
I20260809 15:46:58.212929 27372 raft_consensus.cc:2726] T 953eea268e17491ca76f3a555860a512 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260809 15:46:58.209165 27499 raft_consensus.cc:2726] T 0da4d5283c8a46878bd39b85e5c71f66 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260809 15:46:58.207974 27470 raft_consensus.cc:482] T 2db057fd10e149a9895dff6ef8c9f038 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Starting leader election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
W20260809 15:46:58.251217 23468 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.68:34689, user_credentials={real_user=slave}} blocked reactor thread for 44282.4us
I20260809 15:46:58.216475 27222 raft_consensus.cc:2781] T 953eea268e17491ca76f3a555860a512 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Leader election won for term 6
W20260809 15:46:58.227747 26981 consensus_meta.cc:287] T 07fcabb569414bddbd39df6cedd24e64 P c0f38c1908414566aa9115bb2cbdd85d: Time spent flushing consensus metadata: real 1.073s user 0.006s sys 0.006s
I20260809 15:46:58.214646 27516 raft_consensus.cc:482] T 297c82ed906d4b48b184985eed3de601 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Starting leader election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:58.173142 27518 raft_consensus.cc:504] T 3296ef50be254fc29cfc77d22a8f7a23 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:58.208847 26719 raft_consensus.cc:504] T 0c29937a80724ddb9d07a27534a146f3 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:58.224500 26757 raft_consensus.cc:2726] T 6ced718c1a5c4da79cb8c1ebd1435b87 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260809 15:46:58.148900 27414 consensus_queue.cc:1046] T ba56725e42f74a1baa4c04d9660727f9 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.110s
I20260809 15:46:58.261847 24475 raft_consensus.cc:1261] T d3be1e356b254d22b995be32f1c12f6f P 61e69695258f48baa984c9d395e8ce3f [term 6 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260809 15:46:58.220854 27390 consensus_queue.cc:1046] T 28e17dcb5efe45b8b6a8c42da03b4819 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.044s
I20260809 15:46:58.257093 27470 raft_consensus.cc:3037] T 2db057fd10e149a9895dff6ef8c9f038 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Advancing to term 5
I20260809 15:46:58.174826 27378 consensus_queue.cc:260] T 3d880de657e544468c82157769622a26 P 67b3aeadfa7a432abe8af4d292e04ab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
W20260809 15:46:58.269972 27418 consensus_meta.cc:287] T f9e7cadcd24d45fcbfa84209225e8f2d P c0f38c1908414566aa9115bb2cbdd85d: Time spent flushing consensus metadata: real 0.589s user 0.009s sys 0.001s
I20260809 15:46:58.264103 27499 raft_consensus.cc:2726] T d0721a4403a440f383318a3cea9d1442 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260809 15:46:58.231771 27011 raft_consensus.cc:504] T 4bcac4b24e7a4448abdc934bba5a624e P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:58.271787 26981 raft_consensus.cc:2726] T 07fcabb569414bddbd39df6cedd24e64 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260809 15:46:58.278411 27327 consensus_meta.cc:287] T 3d880de657e544468c82157769622a26 P 65c36d164b0540f099d592f38307ee6c: Time spent flushing consensus metadata: real 0.885s user 0.011s sys 0.000s
I20260809 15:46:58.271483 27516 raft_consensus.cc:3037] T 297c82ed906d4b48b184985eed3de601 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Advancing to term 7
I20260809 15:46:58.272917 27222 raft_consensus.cc:686] T 953eea268e17491ca76f3a555860a512 P c0f38c1908414566aa9115bb2cbdd85d [term 6 LEADER]: Becoming Leader. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:46:58.313820 27222 consensus_queue.cc:237] T 953eea268e17491ca76f3a555860a512 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:46:58.315358 27527 log.cc:924] Time spent T d3be1e356b254d22b995be32f1c12f6f P c0f38c1908414566aa9115bb2cbdd85d: Append to log took a long time: real 0.328s user 0.000s sys 0.001s
I20260809 15:46:58.296149 27507 consensus_queue.cc:1046] T 07fcabb569414bddbd39df6cedd24e64 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.175s
I20260809 15:46:58.321735 27338 raft_consensus.cc:3032] T 763e58d6ba644eea830a6c1664d39c9b P 65c36d164b0540f099d592f38307ee6c [term 5 LEADER]: Stepping down as leader of term 5
W20260809 15:46:58.322109 27549 log.cc:924] Time spent T d2d4f749fcfd4fb3bfb8115af3471598 P 65c36d164b0540f099d592f38307ee6c: Append to log took a long time: real 0.050s user 0.000s sys 0.000s
I20260809 15:46:58.322271 27338 raft_consensus.cc:729] T 763e58d6ba644eea830a6c1664d39c9b P 65c36d164b0540f099d592f38307ee6c [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:46:58.319854 27483 leader_election.cc:288] T 2424fd9bd7f14e508fd106cbfda786a4 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
W20260809 15:46:58.326494 27534 log.cc:924] Time spent T 05ad7d8fdca64e3da527280ca5be9280 P 65c36d164b0540f099d592f38307ee6c: Append to log took a long time: real 0.098s user 0.000s sys 0.001s
I20260809 15:46:58.326794 27575 raft_consensus.cc:482] T d3be1e356b254d22b995be32f1c12f6f P 61e69695258f48baa984c9d395e8ce3f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:58.327315 27575 raft_consensus.cc:504] T d3be1e356b254d22b995be32f1c12f6f P 61e69695258f48baa984c9d395e8ce3f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:58.327941 27338 consensus_queue.cc:260] T 763e58d6ba644eea830a6c1664d39c9b P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:58.329103 27338 raft_consensus.cc:3037] T 763e58d6ba644eea830a6c1664d39c9b P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Advancing to term 6
I20260809 15:46:58.329255 27575 leader_election.cc:288] T d3be1e356b254d22b995be32f1c12f6f P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:58.330127 27493 raft_consensus.cc:2726] T df20671a7b874dbb98c0ef28371c1308 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260809 15:46:58.333699 27462 consensus_meta.cc:287] T 0acc47ce21c3422698d1ef7277c3ed63 P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent flushing consensus metadata: real 0.590s user 0.005s sys 0.000s
I20260809 15:46:58.333954 27347 raft_consensus.cc:504] T c0741ab44d99439abc3aa3bdd27ee160 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:58.334025 27462 raft_consensus.cc:504] T 0acc47ce21c3422698d1ef7277c3ed63 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:58.336145 27347 leader_election.cc:288] T c0741ab44d99439abc3aa3bdd27ee160 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 election: Requested vote from peers 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:58.336359 27462 leader_election.cc:288] T 0acc47ce21c3422698d1ef7277c3ed63 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 6 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:58.339012 27518 leader_election.cc:288] T 3296ef50be254fc29cfc77d22a8f7a23 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:58.340008 27479 leader_election.cc:288] T ae6d2c08a7a94f58ac8dd698e3fd456c P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:58.310608 27382 leader_election.cc:288] T e14d118b2eaa42aa8a7cd4025aba7128 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 8 election: Requested vote from peers 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:58.340201 27490 raft_consensus.cc:504] T 538a228ddfb54bc6a36528d5ff07597b P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:58.342221 27490 leader_election.cc:288] T 538a228ddfb54bc6a36528d5ff07597b P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:58.293663 27327 raft_consensus.cc:504] T 3d880de657e544468c82157769622a26 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
W20260809 15:46:58.343590 27447 consensus_meta.cc:287] T ba56725e42f74a1baa4c04d9660727f9 P c0f38c1908414566aa9115bb2cbdd85d: Time spent flushing consensus metadata: real 0.922s user 0.011s sys 0.000s
I20260809 15:46:58.343942 27447 raft_consensus.cc:2726] T ba56725e42f74a1baa4c04d9660727f9 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260809 15:46:58.336977 27537 leader_election.cc:288] T 3d880de657e544468c82157769622a26 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:58.289108 27418 raft_consensus.cc:504] T f9e7cadcd24d45fcbfa84209225e8f2d P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:58.344834 27327 leader_election.cc:288] T 3d880de657e544468c82157769622a26 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:58.345422 27470 raft_consensus.cc:504] T 2db057fd10e149a9895dff6ef8c9f038 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:58.346313 27418 leader_election.cc:288] T f9e7cadcd24d45fcbfa84209225e8f2d P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 6 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:58.347334 27470 leader_election.cc:288] T 2db057fd10e149a9895dff6ef8c9f038 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
W20260809 15:46:58.383319 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 185598us
I20260809 15:46:58.385094 23228 leader_election.cc:302] T 99bdb93503db4de4a1cfb0161ea0c07c P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:58.398607 23228 leader_election.cc:302] T 07fcabb569414bddbd39df6cedd24e64 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:58.400408 27582 raft_consensus.cc:3032] T 563daf42d7784db195ea99c3cb21b370 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 LEADER]: Stepping down as leader of term 4
I20260809 15:46:58.401580 27582 raft_consensus.cc:729] T 563daf42d7784db195ea99c3cb21b370 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Running, Role: LEADER
I20260809 15:46:58.403785 27584 raft_consensus.cc:2726] T a3622f6e3e3641cea1960e3a31498bb7 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260809 15:46:58.403328 27582 consensus_queue.cc:260] T 563daf42d7784db195ea99c3cb21b370 P 67b3aeadfa7a432abe8af4d292e04ab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:58.406205 23286 raft_consensus.cc:1226] T c0741ab44d99439abc3aa3bdd27ee160 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 4. Current term is 5. Ops: []
I20260809 15:46:58.408073 27582 raft_consensus.cc:3037] T 563daf42d7784db195ea99c3cb21b370 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Advancing to term 5
I20260809 15:46:58.404913 27584 raft_consensus.cc:2726] T 99bdb93503db4de4a1cfb0161ea0c07c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260809 15:46:58.410143 27584 raft_consensus.cc:482] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
I20260809 15:46:58.411725 27584 raft_consensus.cc:504] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:58.415364 27584 leader_election.cc:288] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:58.420095 23396 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55332 (request call id 1363) took 1525 ms. Trace:
I20260809 15:46:58.420549 23396 rpcz_store.cc:276] 0809 15:46:56.894227 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:57.107954 (+213727us) service_pool.cc:227] Handling call
0809 15:46:57.108245 (+ 291us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 15:46:57.108555 (+ 310us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:46:57.108579 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 15:46:57.108987 (+ 408us) log.cc:842] Serialized 26 byte log entry
0809 15:46:57.472957 (+363970us) raft_consensus.cc:1631] Marking committed up to 5
0809 15:46:57.504942 (+ 31985us) log.cc:842] Serialized 14 byte log entry
0809 15:46:57.583957 (+ 79015us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:57.584023 (+ 66us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 15:46:58.419554 (+835531us) raft_consensus.cc:1671] finished
0809 15:46:58.419585 (+ 31us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:58.419865 (+ 280us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":17915,"prepare.queue_time_us":173095,"prepare.run_cpu_time_us":27,"prepare.run_wall_time_us":28,"raft.queue_time_us":46256,"raft.run_cpu_time_us":227,"raft.run_wall_time_us":17037,"spinlock_wait_cycles":1792,"thread_start_us":447294,"threads_started":3,"wal-append.queue_time_us":552362}
I20260809 15:46:58.422734 27582 raft_consensus.cc:2726] T 07fcabb569414bddbd39df6cedd24e64 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260809 15:46:58.404942 23228 leader_election.cc:302] T ba56725e42f74a1baa4c04d9660727f9 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:58.425261 23228 leader_election.cc:302] T a88421dba80d4d3388aa5eb62e23853b P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:58.427512 23228 leader_election.cc:302] T 30c1740ef6874df0a332fbd6fd4b03ce P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c
I20260809 15:46:58.430155 23228 leader_election.cc:302] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d
W20260809 15:46:58.260092 23472 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 59319.2us
I20260809 15:46:58.441499 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5c2e425a74b4e64ae70b5def750434c" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:58.444690 23393 raft_consensus.cc:1226] T df20671a7b874dbb98c0ef28371c1308 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 4. Current term is 5. Ops: []
I20260809 15:46:58.447700 23400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9e7cadcd24d45fcbfa84209225e8f2d" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
W20260809 15:46:58.448369 23468 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.68:34689, user_credentials={real_user=slave}} blocked reactor thread for 192803us
I20260809 15:46:58.448870 23400 raft_consensus.cc:2379] T f9e7cadcd24d45fcbfa84209225e8f2d P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c in current term 6: Already voted for candidate c0f38c1908414566aa9115bb2cbdd85d in this term.
I20260809 15:46:58.323983 27244 leader_election.cc:288] T d3e94b7e50754a81b3d72a07476cb7b9 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 7 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:58.453678 26719 leader_election.cc:288] T 0c29937a80724ddb9d07a27534a146f3 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 6 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:58.458555 27011 leader_election.cc:288] T 4bcac4b24e7a4448abdc934bba5a624e P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
W20260809 15:46:58.467784 23470 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 271050us
I20260809 15:46:58.468945 23470 leader_election.cc:302] T cee02fff2c9c48a19f43eb358a89e7b0 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:46:58.351779 27516 raft_consensus.cc:504] T 297c82ed906d4b48b184985eed3de601 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:58.432821 27582 raft_consensus.cc:2726] T ba56725e42f74a1baa4c04d9660727f9 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260809 15:46:58.473304 27347 raft_consensus.cc:2781] T 30c1740ef6874df0a332fbd6fd4b03ce P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260809 15:46:58.473843 27347 raft_consensus.cc:482] T 30c1740ef6874df0a332fbd6fd4b03ce P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:58.474336 27347 raft_consensus.cc:3037] T 30c1740ef6874df0a332fbd6fd4b03ce P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Advancing to term 6
I20260809 15:46:58.474769 27516 leader_election.cc:288] T 297c82ed906d4b48b184985eed3de601 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 7 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:58.474910 23404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e17dcb5efe45b8b6a8c42da03b4819" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:58.475857 27322 raft_consensus.cc:482] T 0acc47ce21c3422698d1ef7277c3ed63 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:58.476811 27322 raft_consensus.cc:504] T 0acc47ce21c3422698d1ef7277c3ed63 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:58.477953 23396 raft_consensus.cc:1226] T 2db057fd10e149a9895dff6ef8c9f038 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 4. Current term is 5. Ops: []
I20260809 15:46:58.479656 23285 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53554 (request call id 1545) took 2005 ms. Trace:
I20260809 15:46:58.480087 23285 rpcz_store.cc:276] 0809 15:46:56.474236 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:56.481723 (+ 7487us) service_pool.cc:227] Handling call
0809 15:46:56.481975 (+ 252us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:57.098077 (+616102us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:58.479073 (+1380996us) spinlock_profiling.cc:243] Waited 616 ms on lock 0x7b540008aa54. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e9220f51 00007f34e921f99e 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772 00007f34e764a94c 00007f34e7649d86
0809 15:46:58.479438 (+ 365us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1415693056}
I20260809 15:46:58.480816 27322 leader_election.cc:288] T 0acc47ce21c3422698d1ef7277c3ed63 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:58.482704 27382 consensus_queue.cc:1035] T f9e7cadcd24d45fcbfa84209225e8f2d P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.001s
I20260809 15:46:58.484572 27382 raft_consensus.cc:2726] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260809 15:46:58.488332 23522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d880de657e544468c82157769622a26" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:58.452713 27263 leader_election.cc:288] T 0acc47ce21c3422698d1ef7277c3ed63 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 7 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
W20260809 15:46:58.505820 23397 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55332 (request call id 1356) took 1638 ms (client timeout 2010 ms). Trace:
W20260809 15:46:58.506253 23397 rpcz_store.cc:269] 0809 15:46:56.867923 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:56.992904 (+124981us) service_pool.cc:227] Handling call
0809 15:46:58.505368 (+1512464us) spinlock_profiling.cc:243] Waited 1.16 s on lock 0x7b540025e254. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e92242ae 00007f34ee161c27 00007f34e7849c04 00007f34e7849b92 00007f34e7849ac1 00007f34e7849a3c 00007f34e7848cf2 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:46:58.505597 (+ 229us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2668901120}
I20260809 15:46:58.472414 27584 raft_consensus.cc:2726] T a88421dba80d4d3388aa5eb62e23853b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260809 15:46:58.509076 23518 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44018 (request call id 247) took 1995 ms. Trace:
I20260809 15:46:58.512449 23518 rpcz_store.cc:276] 0809 15:46:56.513223 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:56.513477 (+ 254us) service_pool.cc:227] Handling call
0809 15:46:56.680667 (+167190us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:58.198921 (+1518254us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:58.508605 (+309684us) spinlock_profiling.cc:243] Waited 1.52 s on lock 0x7b5400056ed4. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e9220f51 00007f34e921f99e 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772 00007f34e764a94c 00007f34e7649d86
0809 15:46:58.508853 (+ 248us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3490524416}
W20260809 15:46:58.514603 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 295947us
I20260809 15:46:58.515887 23351 leader_election.cc:302] T 213543cdbe814c89a17a963469922c1b P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:58.517107 23522 raft_consensus.cc:2379] T 3d880de657e544468c82157769622a26 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 61e69695258f48baa984c9d395e8ce3f in current term 3: Already voted for candidate 65c36d164b0540f099d592f38307ee6c in this term.
I20260809 15:46:58.520299 26611 raft_consensus.cc:3032] T ba56725e42f74a1baa4c04d9660727f9 P 65c36d164b0540f099d592f38307ee6c [term 3 LEADER]: Stepping down as leader of term 3
I20260809 15:46:58.520753 26611 raft_consensus.cc:729] T ba56725e42f74a1baa4c04d9660727f9 P 65c36d164b0540f099d592f38307ee6c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:46:58.522171 26611 consensus_queue.cc:260] T ba56725e42f74a1baa4c04d9660727f9 P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:58.525084 27213 raft_consensus.cc:2726] T 213543cdbe814c89a17a963469922c1b P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260809 15:46:58.526429 23396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3be1e356b254d22b995be32f1c12f6f" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:58.527338 27516 consensus_queue.cc:1035] T d3be1e356b254d22b995be32f1c12f6f P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260809 15:46:58.527724 24451 raft_consensus.cc:2454] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 61e69695258f48baa984c9d395e8ce3f [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 8.
I20260809 15:46:58.530208 23394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00a1881267a14ba8b78c81040f17a014" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:58.532382 27514 raft_consensus.cc:482] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 61e69695258f48baa984c9d395e8ce3f [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:58.526439 27267 raft_consensus.cc:3032] T df20671a7b874dbb98c0ef28371c1308 P 65c36d164b0540f099d592f38307ee6c [term 4 LEADER]: Stepping down as leader of term 4
I20260809 15:46:58.533303 27267 raft_consensus.cc:729] T df20671a7b874dbb98c0ef28371c1308 P 65c36d164b0540f099d592f38307ee6c [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:46:58.488157 27302 raft_consensus.cc:504] T 0c48a3049897452383257aa757834fb5 P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:58.532428 23404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a823d35f4f348d1b52ba9e6d80b00d2" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:58.538103 23404 raft_consensus.cc:3037] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Advancing to term 8
W20260809 15:46:58.539920 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 67369.7us
I20260809 15:46:58.540303 23396 raft_consensus.cc:1226] T 05ad7d8fdca64e3da527280ca5be9280 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 5. Current term is 6. Ops: []
I20260809 15:46:58.538612 27267 consensus_queue.cc:260] T df20671a7b874dbb98c0ef28371c1308 P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:58.540899 27507 raft_consensus.cc:3032] T 07fcabb569414bddbd39df6cedd24e64 P 65c36d164b0540f099d592f38307ee6c [term 5 LEADER]: Stepping down as leader of term 5
I20260809 15:46:58.541308 27267 raft_consensus.cc:3037] T df20671a7b874dbb98c0ef28371c1308 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Advancing to term 5
I20260809 15:46:58.546098 24463 raft_consensus.cc:2454] T cee02fff2c9c48a19f43eb358a89e7b0 P 61e69695258f48baa984c9d395e8ce3f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 6.
I20260809 15:46:58.546769 23228 leader_election.cc:302] T 05ad7d8fdca64e3da527280ca5be9280 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:46:58.545383 23393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ae14a74a26f434db08c652e097385c8" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:58.541352 27507 raft_consensus.cc:729] T 07fcabb569414bddbd39df6cedd24e64 P 65c36d164b0540f099d592f38307ee6c [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:46:58.547839 23393 raft_consensus.cc:2379] T 6ae14a74a26f434db08c652e097385c8 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c in current term 6: Already voted for candidate c0f38c1908414566aa9115bb2cbdd85d in this term.
I20260809 15:46:58.552385 23280 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53554 (request call id 1541) took 2101 ms. Trace:
I20260809 15:46:58.552788 23280 rpcz_store.cc:276] 0809 15:46:56.450469 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:56.454881 (+ 4412us) service_pool.cc:227] Handling call
0809 15:46:56.473209 (+ 18328us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:57.094319 (+621110us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:58.551890 (+1457571us) spinlock_profiling.cc:243] Waited 621 ms on lock 0x7b5400153bd4. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e9220f51 00007f34e921f99e 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772 00007f34e764a94c 00007f34e7649d86
0809 15:46:58.552178 (+ 288us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1427240832}
I20260809 15:46:58.546382 27196 consensus_queue.cc:1046] T d3e94b7e50754a81b3d72a07476cb7b9 P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.008s
I20260809 15:46:58.554751 24452 raft_consensus.cc:1226] T 0c48a3049897452383257aa757834fb5 P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 3. Current term is 4. Ops: []
I20260809 15:46:58.557375 27395 raft_consensus.cc:3032] T 28e17dcb5efe45b8b6a8c42da03b4819 P c0f38c1908414566aa9115bb2cbdd85d [term 6 LEADER]: Stepping down as leader of term 6
W20260809 15:46:58.557485 23470 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 88794.1us
I20260809 15:46:58.557785 27395 raft_consensus.cc:729] T 28e17dcb5efe45b8b6a8c42da03b4819 P c0f38c1908414566aa9115bb2cbdd85d [term 6 LEADER]: Becoming Follower/Learner. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:46:58.560079 23470 leader_election.cc:302] T 2424fd9bd7f14e508fd106cbfda786a4 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:46:58.560637 27196 raft_consensus.cc:3032] T d3e94b7e50754a81b3d72a07476cb7b9 P 61e69695258f48baa984c9d395e8ce3f [term 6 LEADER]: Stepping down as leader of term 6
I20260809 15:46:58.561112 27196 raft_consensus.cc:729] T d3e94b7e50754a81b3d72a07476cb7b9 P 61e69695258f48baa984c9d395e8ce3f [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 61e69695258f48baa984c9d395e8ce3f, State: Running, Role: LEADER
W20260809 15:46:58.561580 27427 log.cc:924] Time spent T 0c29937a80724ddb9d07a27534a146f3 P c0f38c1908414566aa9115bb2cbdd85d: Append to log took a long time: real 1.691s user 0.006s sys 0.002s
I20260809 15:46:58.562258 27196 consensus_queue.cc:260] T d3e94b7e50754a81b3d72a07476cb7b9 P 61e69695258f48baa984c9d395e8ce3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:58.562995 27081 raft_consensus.cc:2781] T cee02fff2c9c48a19f43eb358a89e7b0 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Leader election won for term 6
I20260809 15:46:58.532971 27514 raft_consensus.cc:504] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 61e69695258f48baa984c9d395e8ce3f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:58.564042 23472 leader_election.cc:302] T c5c2e425a74b4e64ae70b5def750434c P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:58.565052 23394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2d4f749fcfd4fb3bfb8115af3471598" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:58.565459 27347 raft_consensus.cc:504] T 30c1740ef6874df0a332fbd6fd4b03ce P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:58.566330 27267 raft_consensus.cc:482] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:46:58.567463 26399 raft_consensus.cc:482] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
I20260809 15:46:58.565670 23404 raft_consensus.cc:2454] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 8.
I20260809 15:46:58.568173 26399 raft_consensus.cc:504] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:58.569072 27267 raft_consensus.cc:504] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:58.570987 26399 leader_election.cc:288] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:58.575786 27294 raft_consensus.cc:2781] T 2424fd9bd7f14e508fd106cbfda786a4 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260809 15:46:58.573024 27081 raft_consensus.cc:686] T cee02fff2c9c48a19f43eb358a89e7b0 P 65c36d164b0540f099d592f38307ee6c [term 6 LEADER]: Becoming Leader. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:46:58.576968 27217 raft_consensus.cc:2781] T 05ad7d8fdca64e3da527280ca5be9280 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Leader election won for term 6
I20260809 15:46:58.577584 27217 raft_consensus.cc:686] T 05ad7d8fdca64e3da527280ca5be9280 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 LEADER]: Becoming Leader. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Running, Role: LEADER
I20260809 15:46:58.578298 25863 raft_consensus.cc:2726] T c5c2e425a74b4e64ae70b5def750434c P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260809 15:46:58.578733 23472 leader_election.cc:302] T f9e7cadcd24d45fcbfa84209225e8f2d P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:58.579913 27383 raft_consensus.cc:2726] T f9e7cadcd24d45fcbfa84209225e8f2d P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260809 15:46:58.578975 27217 consensus_queue.cc:237] T 05ad7d8fdca64e3da527280ca5be9280 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:46:58.581390 27494 log.cc:924] Time spent T 4b30f28311964a0b9149a50495e83d8f P 67b3aeadfa7a432abe8af4d292e04ab9: Append to log took a long time: real 0.985s user 0.000s sys 0.002s
I20260809 15:46:58.576233 27294 raft_consensus.cc:482] T 2424fd9bd7f14e508fd106cbfda786a4 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:46:58.584595 23472 leader_election.cc:302] T 28e17dcb5efe45b8b6a8c42da03b4819 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:58.560101 27302 leader_election.cc:288] T 0c48a3049897452383257aa757834fb5 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 4 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:58.585409 23522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c48a3049897452383257aa757834fb5" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:58.584728 27294 raft_consensus.cc:3037] T 2424fd9bd7f14e508fd106cbfda786a4 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Advancing to term 7
I20260809 15:46:58.575244 23470 leader_election.cc:302] T 00a1881267a14ba8b78c81040f17a014 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:46:58.590373 23404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2424fd9bd7f14e508fd106cbfda786a4" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 7 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:58.586863 23351 leader_election.cc:302] T 0a51c0baaa6e432997c9d3552af4f254 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 61e69695258f48baa984c9d395e8ce3f, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:58.577540 27081 consensus_queue.cc:237] T cee02fff2c9c48a19f43eb358a89e7b0 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:58.592363 27531 raft_consensus.cc:2781] T 00a1881267a14ba8b78c81040f17a014 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260809 15:46:58.592844 27531 raft_consensus.cc:482] T 00a1881267a14ba8b78c81040f17a014 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Starting leader election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:46:58.593230 27531 raft_consensus.cc:3037] T 00a1881267a14ba8b78c81040f17a014 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Advancing to term 8
I20260809 15:46:58.596024 23399 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55332 (request call id 1361) took 1702 ms. Trace:
I20260809 15:46:58.596400 23399 rpcz_store.cc:276] 0809 15:46:56.893379 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:57.101663 (+208284us) service_pool.cc:227] Handling call
0809 15:46:57.600337 (+498674us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:57.672883 (+ 72546us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:58.595625 (+922742us) spinlock_profiling.cc:243] Waited 498 ms on lock 0x7b54002ff150. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e921fa09 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:46:58.595833 (+ 208us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1146248448}
I20260809 15:46:58.598078 26399 raft_consensus.cc:2726] T 28e17dcb5efe45b8b6a8c42da03b4819 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260809 15:46:58.622959 23404 raft_consensus.cc:2454] T 2424fd9bd7f14e508fd106cbfda786a4 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 6.
I20260809 15:46:58.577612 27116 raft_consensus.cc:2726] T a88421dba80d4d3388aa5eb62e23853b P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260809 15:46:58.548322 27507 consensus_queue.cc:260] T 07fcabb569414bddbd39df6cedd24e64 P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:58.559012 27395 consensus_queue.cc:260] T 28e17dcb5efe45b8b6a8c42da03b4819 P c0f38c1908414566aa9115bb2cbdd85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:58.650717 27520 raft_consensus.cc:2726] T 0a51c0baaa6e432997c9d3552af4f254 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260809 15:46:58.662124 23470 leader_election.cc:302] T d2d4f749fcfd4fb3bfb8115af3471598 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:46:58.639509 27267 leader_election.cc:288] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
W20260809 15:46:58.682658 24406 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 134303us
I20260809 15:46:58.704150 27116 raft_consensus.cc:482] T cef3e7277d7d4760824a3436ed4a847d P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:58.704825 27116 raft_consensus.cc:504] T cef3e7277d7d4760824a3436ed4a847d P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:58.705590 27216 raft_consensus.cc:482] T 4564dfc6d4094c1b9240b367ccecf3b1 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Starting leader election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:58.706112 27216 raft_consensus.cc:3037] T 4564dfc6d4094c1b9240b367ccecf3b1 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Advancing to term 6
I20260809 15:46:58.706887 27116 leader_election.cc:288] T cef3e7277d7d4760824a3436ed4a847d P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:58.708002 23392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3e94b7e50754a81b3d72a07476cb7b9" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:58.709527 23392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bcac4b24e7a4448abdc934bba5a624e" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:58.710292 23392 raft_consensus.cc:2454] T 4bcac4b24e7a4448abdc934bba5a624e P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 2.
W20260809 15:46:58.710139 23116 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:37326 (request call id 149) took 13663 ms (client timeout 15000 ms). Trace:
I20260809 15:46:58.711056 23228 leader_election.cc:398] T 0acc47ce21c3422698d1ef7277c3ed63 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 6 pre-election: Vote denied by peer c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263) with higher term. Message: Invalid argument: T 0acc47ce21c3422698d1ef7277c3ed63 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 6. Current term is 7.
I20260809 15:46:58.712428 24452 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38734 (request call id 323) took 1027 ms. Trace:
I20260809 15:46:58.729019 27514 leader_election.cc:288] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:58.714123 23115 catalog_manager.cc:5495] T 297c82ed906d4b48b184985eed3de601 P 65c36d164b0540f099d592f38307ee6c reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "65c36d164b0540f099d592f38307ee6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } health_report { overall_health: HEALTHY } } }
I20260809 15:46:58.732290 27428 raft_consensus.cc:482] T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
I20260809 15:46:58.732846 23526 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56770 (request call id 1345) took 1779 ms. Trace:
I20260809 15:46:58.733237 23526 rpcz_store.cc:276] 0809 15:46:56.953336 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:56.969719 (+ 16383us) service_pool.cc:227] Handling call
0809 15:46:56.969966 (+ 247us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:57.729120 (+759154us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:58.732345 (+1003225us) spinlock_profiling.cc:243] Waited 715 ms on lock 0x7b54000762d4. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e9220f51 00007f34e921f99e 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772 00007f34e764a94c 00007f34e7649d86
0809 15:46:58.732661 (+ 316us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1644406528}
W20260809 15:46:58.715931 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 176392us
W20260809 15:46:58.724532 27438 log.cc:924] Time spent T f7715d71dbac447b9351fc5eb36c5e80 P 67b3aeadfa7a432abe8af4d292e04ab9: Append to log took a long time: real 1.767s user 0.018s sys 0.017s
I20260809 15:46:58.728349 24452 rpcz_store.cc:276] 0809 15:46:57.685140 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:57.686968 (+ 1828us) service_pool.cc:227] Handling call
0809 15:46:57.741851 (+ 54883us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:58.555210 (+813359us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:58.712078 (+156868us) spinlock_profiling.cc:243] Waited 810 ms on lock 0x7b5400521a54. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e9220f51 00007f34e921f99e 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772 00007f34e764a94c 00007f34e7649d86
0809 15:46:58.712346 (+ 268us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1863806848}
I20260809 15:46:58.741394 23115 catalog_manager.cc:5495] T ae6d2c08a7a94f58ac8dd698e3fd456c P 65c36d164b0540f099d592f38307ee6c reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "65c36d164b0540f099d592f38307ee6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } health_report { overall_health: HEALTHY } } }
I20260809 15:46:58.743361 23115 catalog_manager.cc:5495] T a88421dba80d4d3388aa5eb62e23853b P 65c36d164b0540f099d592f38307ee6c reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "65c36d164b0540f099d592f38307ee6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } health_report { overall_health: HEALTHY } } }
I20260809 15:46:58.745626 23115 catalog_manager.cc:5495] T d2d4f749fcfd4fb3bfb8115af3471598 P 65c36d164b0540f099d592f38307ee6c reported cstate change: term changed from 1 to 4, leader changed from c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66) to 65c36d164b0540f099d592f38307ee6c (127.20.202.67). New cstate: current_term: 4 leader_uuid: "65c36d164b0540f099d592f38307ee6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } health_report { overall_health: HEALTHY } } }
W20260809 15:46:58.717466 23116 rpcz_store.cc:269] 0809 15:46:45.046728 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:45.053580 (+ 6852us) service_pool.cc:227] Handling call
0809 15:46:58.709863 (+13656283us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0809 15:46:49.633413 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0809 15:46:49.634084 (+ 671us) write_op.cc:432] Acquiring schema lock in shared mode
0809 15:46:49.634136 (+ 52us) write_op.cc:435] Acquired schema lock
0809 15:46:49.634148 (+ 12us) tablet.cc:660] Decoding operations
0809 15:46:50.007705 (+373557us) write_op.cc:620] Acquiring the partition lock for write op
0809 15:46:50.007791 (+ 86us) write_op.cc:641] Partition lock acquired for write op
0809 15:46:50.007835 (+ 44us) tablet.cc:683] Acquiring locks for 15 operations
0809 15:46:50.009054 (+ 1219us) tablet.cc:699] Row locks acquired
0809 15:46:50.009088 (+ 34us) write_op.cc:260] PREPARE: finished
0809 15:46:50.009254 (+ 166us) write_op.cc:270] Start()
0809 15:46:50.009391 (+ 137us) write_op.cc:276] Timestamp: P: 1786290410009195 usec, L: 0
0809 15:46:50.009425 (+ 34us) op_driver.cc:331] REPLICATION: starting
0809 15:46:50.014796 (+ 5371us) log.cc:842] Serialized 5895 byte log entry
0809 15:46:50.870590 (+855794us) spinlock_profiling.cc:243] Waited 1.86 ms on lock 0x7b44000232cc. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e613cf11 00007f34e620aa95 00007f34e620b4b2 00007f34e8eb3511 00007f34e8eaee1a 00007f34e8ea2bee 00007f34e8ea1f36 00007f34e8ea1b58 00007f34e8eab551 00007f34e8eab509 00007f34e8eab499
0809 15:46:52.028514 (+1157924us) op_driver.cc:444] REPLICATION: finished
0809 15:46:52.320183 (+291669us) write_op.cc:301] APPLY: starting
0809 15:46:52.320270 (+ 87us) tablet.cc:1365] starting BulkCheckPresence
0809 15:46:52.388769 (+ 68499us) tablet.cc:1368] finished BulkCheckPresence
0809 15:46:52.388794 (+ 25us) tablet.cc:1370] starting ApplyRowOperation cycle
0809 15:46:52.510604 (+121810us) tablet.cc:1381] finished ApplyRowOperation cycle
0809 15:46:52.510865 (+ 261us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=15
0809 15:46:52.510940 (+ 75us) write_op.cc:312] APPLY: finished
0809 15:46:52.511348 (+ 408us) log.cc:842] Serialized 107 byte log entry
0809 15:46:52.513507 (+ 2159us) write_op.cc:489] Releasing partition, row and schema locks
0809 15:46:52.523552 (+ 10045us) write_op.cc:454] Released schema lock
0809 15:46:52.571299 (+ 47747us) write_op.cc:341] FINISH: Updating metrics
0809 15:46:52.640300 (+ 69001us) write_op.cc:489] Releasing partition, row and schema locks
0809 15:46:52.646479 (+ 6179us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":1431906,"reported_tablets":20,"child_traces":[["op",{"apply.queue_time_us":281789,"apply.run_cpu_time_us":4907,"apply.run_wall_time_us":340162,"num_ops":15,"prepare.queue_time_us":158158,"prepare.run_cpu_time_us":14713,"prepare.run_wall_time_us":597156,"raft.queue_time_us":328040,"raft.run_cpu_time_us":9875,"raft.run_wall_time_us":306274,"replication_time_us":2010755,"spinlock_wait_cycles":4285824,"thread_start_us":559533,"threads_started":4,"wal-append.queue_time_us":181696,"wal-append.run_cpu_time_us":89121,"wal-append.run_wall_time_us":3390302}]]}
I20260809 15:46:58.747201 23353 leader_election.cc:302] T ae6d2c08a7a94f58ac8dd698e3fd456c P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
W20260809 15:46:58.748728 23470 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 150061us
I20260809 15:46:58.751786 23518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9e7cadcd24d45fcbfa84209225e8f2d" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:58.752314 23521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2db057fd10e149a9895dff6ef8c9f038" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:58.753770 27496 raft_consensus.cc:2781] T d2d4f749fcfd4fb3bfb8115af3471598 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260809 15:46:58.754215 27496 raft_consensus.cc:482] T d2d4f749fcfd4fb3bfb8115af3471598 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Starting leader election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:46:58.754568 27496 raft_consensus.cc:3037] T d2d4f749fcfd4fb3bfb8115af3471598 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Advancing to term 6
I20260809 15:46:58.752596 23518 raft_consensus.cc:2379] T f9e7cadcd24d45fcbfa84209225e8f2d P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d in current term 6: Already voted for candidate 65c36d164b0540f099d592f38307ee6c in this term.
I20260809 15:46:58.757208 23392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a823d35f4f348d1b52ba9e6d80b00d2" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:58.760036 23392 raft_consensus.cc:2454] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 8.
I20260809 15:46:58.749105 27347 leader_election.cc:288] T 30c1740ef6874df0a332fbd6fd4b03ce P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 6 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:58.763662 24410 leader_election.cc:302] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:46:58.764472 23522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "538a228ddfb54bc6a36528d5ff07597b" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:58.765549 27298 raft_consensus.cc:482] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:46:58.766579 24451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0741ab44d99439abc3aa3bdd27ee160" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f"
I20260809 15:46:58.768246 24471 raft_consensus.cc:1226] T 0a51c0baaa6e432997c9d3552af4f254 P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Rejecting Update request from peer 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 4. Current term is 5. Ops: []
I20260809 15:46:58.773175 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0acc47ce21c3422698d1ef7277c3ed63" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 7 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:58.774397 23526 raft_consensus.cc:3037] T 0acc47ce21c3422698d1ef7277c3ed63 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Advancing to term 7
I20260809 15:46:58.768968 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3296ef50be254fc29cfc77d22a8f7a23" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:58.771618 24475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f9d6565aa4c45a3b3f01210d0eb71f7" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:58.776604 23400 raft_consensus.cc:1226] T 0c29937a80724ddb9d07a27534a146f3 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 5. Current term is 6. Ops: []
W20260809 15:46:58.775504 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 112364us
I20260809 15:46:58.763031 23115 catalog_manager.cc:5495] T a3622f6e3e3641cea1960e3a31498bb7 P 65c36d164b0540f099d592f38307ee6c reported cstate change: term changed from 1 to 3, leader changed from 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65) to 65c36d164b0540f099d592f38307ee6c (127.20.202.67). New cstate: current_term: 3 leader_uuid: "65c36d164b0540f099d592f38307ee6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } health_report { overall_health: HEALTHY } } }
I20260809 15:46:58.778412 23392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0acc47ce21c3422698d1ef7277c3ed63" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 7 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:58.780061 23392 raft_consensus.cc:2379] T 0acc47ce21c3422698d1ef7277c3ed63 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c in current term 7: Already voted for candidate c0f38c1908414566aa9115bb2cbdd85d in this term.
I20260809 15:46:58.780804 24448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3e94b7e50754a81b3d72a07476cb7b9" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f"
I20260809 15:46:58.781600 24455 raft_consensus.cc:1226] T 3d880de657e544468c82157769622a26 P 61e69695258f48baa984c9d395e8ce3f [term 2 FOLLOWER]: Rejecting Update request from peer 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 1. Current term is 2. Ops: []
I20260809 15:46:58.777144 27298 raft_consensus.cc:504] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:58.782090 24452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e14d118b2eaa42aa8a7cd4025aba7128" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f"
I20260809 15:46:58.789186 23351 leader_election.cc:302] T 8e4ffcf83c234ae7910ddef3ec0b4bda P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 61e69695258f48baa984c9d395e8ce3f, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:58.776912 27196 raft_consensus.cc:3037] T d3e94b7e50754a81b3d72a07476cb7b9 P 61e69695258f48baa984c9d395e8ce3f [term 6 FOLLOWER]: Advancing to term 7
I20260809 15:46:58.788681 23518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c29937a80724ddb9d07a27534a146f3" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
W20260809 15:46:58.796020 23472 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 211580us
I20260809 15:46:58.786949 24452 raft_consensus.cc:3037] T e14d118b2eaa42aa8a7cd4025aba7128 P 61e69695258f48baa984c9d395e8ce3f [term 7 FOLLOWER]: Advancing to term 8
I20260809 15:46:58.733004 27428 raft_consensus.cc:504] T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:58.801096 27428 leader_election.cc:288] T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:58.803504 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d880de657e544468c82157769622a26" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f"
I20260809 15:46:58.804215 24468 raft_consensus.cc:3037] T 3d880de657e544468c82157769622a26 P 61e69695258f48baa984c9d395e8ce3f [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:58.808895 27426 raft_consensus.cc:2781] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 61e69695258f48baa984c9d395e8ce3f [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260809 15:46:58.809325 27426 raft_consensus.cc:482] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 61e69695258f48baa984c9d395e8ce3f [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:46:58.809674 27426 raft_consensus.cc:3037] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 61e69695258f48baa984c9d395e8ce3f [term 8 FOLLOWER]: Advancing to term 9
I20260809 15:46:58.810369 27298 leader_election.cc:288] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:58.815137 27530 raft_consensus.cc:482] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:58.815793 23522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "297c82ed906d4b48b184985eed3de601" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:58.817687 24475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bcac4b24e7a4448abdc934bba5a624e" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:58.822275 27588 raft_consensus.cc:2726] T 8e4ffcf83c234ae7910ddef3ec0b4bda P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260809 15:46:58.822700 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3924b648461e463b9d7925e5671d68ae" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:58.831959 24453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e14d118b2eaa42aa8a7cd4025aba7128" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:58.833490 23285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "907bacfa79d04a55a68c8dbe60056462" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:58.834302 23273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0741ab44d99439abc3aa3bdd27ee160" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:58.834656 23277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "294d4bcc1e7a477cb56455581277a60f" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:58.835634 23280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a51c0baaa6e432997c9d3552af4f254" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:58.835811 23286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "563daf42d7784db195ea99c3cb21b370" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:58.836074 23277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a51c0baaa6e432997c9d3552af4f254" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:58.836915 23273 raft_consensus.cc:2379] T c0741ab44d99439abc3aa3bdd27ee160 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 61e69695258f48baa984c9d395e8ce3f in current term 5: Already voted for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in this term.
I20260809 15:46:58.837123 23276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5712f6b6fffc4cc5a8ddfbe63e4097df" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:58.839705 23518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3aa6a9a52804df0ba9b0ca3c82f38c3" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:58.840299 23518 raft_consensus.cc:2454] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 4.
I20260809 15:46:58.841921 24475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3aa6a9a52804df0ba9b0ca3c82f38c3" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:58.842610 24475 raft_consensus.cc:2454] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 4.
I20260809 15:46:58.843326 24453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad8a3b25d7b49e5a2142d5709edd0c7" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
W20260809 15:46:58.843571 23519 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56770 (request call id 1334) took 2118 ms (client timeout 1735 ms). Trace:
W20260809 15:46:58.843935 23519 rpcz_store.cc:269] 0809 15:46:56.724568 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:56.772550 (+ 47982us) service_pool.cc:227] Handling call
0809 15:46:58.843168 (+2070618us) spinlock_profiling.cc:243] Waited 188 ms on lock 0x7b54000f8cd4. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e92242ae 00007f34ee161c27 00007f34e7849c04 00007f34e7849b92 00007f34e7849ac1 00007f34e7849a3c 00007f34e7848cf2 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:46:58.843371 (+ 203us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":433033984}
W20260809 15:46:58.846067 27439 log.cc:924] Time spent T 30c1740ef6874df0a332fbd6fd4b03ce P 67b3aeadfa7a432abe8af4d292e04ab9: Append to log took a long time: real 1.934s user 0.012s sys 0.001s
W20260809 15:46:58.846700 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 70944.9us
I20260809 15:46:58.799201 23115 catalog_manager.cc:5495] T 472816994d814cb198b69e5248697440 P 65c36d164b0540f099d592f38307ee6c reported cstate change: term changed from 1 to 5, leader changed from 65c36d164b0540f099d592f38307ee6c (127.20.202.67) to 61e69695258f48baa984c9d395e8ce3f (127.20.202.68). New cstate: current_term: 5 leader_uuid: "61e69695258f48baa984c9d395e8ce3f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:46:58.850538 24410 leader_election.cc:302] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c; no voters:
I20260809 15:46:58.781975 27531 raft_consensus.cc:504] T 00a1881267a14ba8b78c81040f17a014 P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:58.849750 23353 leader_election.cc:398] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 7 election: Vote denied by peer 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553) with higher term. Message: Invalid argument: T 2a823d35f4f348d1b52ba9e6d80b00d2 P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d for earlier term 7. Current term is 8.
I20260809 15:46:58.853739 23353 leader_election.cc:401] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 7 election: Cancelling election due to peer responding with higher term
I20260809 15:46:58.853828 24452 raft_consensus.cc:2454] T e14d118b2eaa42aa8a7cd4025aba7128 P 61e69695258f48baa984c9d395e8ce3f [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 8.
I20260809 15:46:58.852720 23468 leader_election.cc:302] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c; no voters:
I20260809 15:46:58.845904 27217 consensus_queue.cc:1046] T 0a51c0baaa6e432997c9d3552af4f254 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.009s
I20260809 15:46:58.841862 27053 raft_consensus.cc:3037] T a88421dba80d4d3388aa5eb62e23853b P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:46:58.815755 27530 raft_consensus.cc:504] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:58.857309 23521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cef3e7277d7d4760824a3436ed4a847d" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:58.857438 23223 leader_election.cc:302] T e14d118b2eaa42aa8a7cd4025aba7128 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:46:58.873657 23276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d880de657e544468c82157769622a26" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:58.872321 23285 raft_consensus.cc:3037] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Advancing to term 6
I20260809 15:46:58.861454 23522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a823d35f4f348d1b52ba9e6d80b00d2" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:58.857820 23399 raft_consensus.cc:1261] T 563daf42d7784db195ea99c3cb21b370 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Refusing update from remote peer 61e69695258f48baa984c9d395e8ce3f: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260809 15:46:58.871879 23273 raft_consensus.cc:1226] T df20671a7b874dbb98c0ef28371c1308 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 4. Current term is 5. Ops: []
I20260809 15:46:58.877993 23277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d880de657e544468c82157769622a26" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:58.882206 23351 leader_election.cc:302] T f7715d71dbac447b9351fc5eb36c5e80 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:58.887224 23392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3aa6a9a52804df0ba9b0ca3c82f38c3" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:58.857084 23115 catalog_manager.cc:5495] T f9e7cadcd24d45fcbfa84209225e8f2d P 65c36d164b0540f099d592f38307ee6c reported cstate change: term changed from 3 to 4, leader changed from 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65) to 65c36d164b0540f099d592f38307ee6c (127.20.202.67). New cstate: current_term: 4 leader_uuid: "65c36d164b0540f099d592f38307ee6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } health_report { overall_health: HEALTHY } } }
I20260809 15:46:58.920115 23280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3be1e356b254d22b995be32f1c12f6f" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
W20260809 15:46:58.921797 23468 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.68:34689, user_credentials={real_user=slave}} blocked reactor thread for 75708.2us
I20260809 15:46:58.879470 27547 raft_consensus.cc:2781] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260809 15:46:58.926275 23396 raft_consensus.cc:1226] T 297c82ed906d4b48b184985eed3de601 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 6. Current term is 7. Ops: []
W20260809 15:46:58.927325 23472 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 73922.9us
I20260809 15:46:58.882009 27507 raft_consensus.cc:3037] T 07fcabb569414bddbd39df6cedd24e64 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Advancing to term 6
I20260809 15:46:58.872021 26399 raft_consensus.cc:482] T a3622f6e3e3641cea1960e3a31498bb7 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:58.864182 27217 raft_consensus.cc:3032] T 0a51c0baaa6e432997c9d3552af4f254 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 LEADER]: Stepping down as leader of term 4
I20260809 15:46:58.927081 23399 raft_consensus.cc:1226] T a88421dba80d4d3388aa5eb62e23853b P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 3. Current term is 4. Ops: []
I20260809 15:46:58.870643 27513 consensus_queue.cc:1046] T d1a134ba33ea4076bbaac0ae460c17ba P 65c36d164b0540f099d592f38307ee6c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.121s
I20260809 15:46:58.925232 27547 raft_consensus.cc:482] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Starting leader election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:46:58.944572 27531 leader_election.cc:288] T 00a1881267a14ba8b78c81040f17a014 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 8 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:58.870771 27404 consensus_queue.cc:1046] T 2db057fd10e149a9895dff6ef8c9f038 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.083s
I20260809 15:46:58.935426 23270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0acc47ce21c3422698d1ef7277c3ed63" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 7 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:58.948413 23522 raft_consensus.cc:2454] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 8.
I20260809 15:46:58.944883 27547 raft_consensus.cc:3037] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Advancing to term 5
W20260809 15:46:58.953349 23353 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 139467us
I20260809 15:46:58.931043 27582 raft_consensus.cc:2781] T e14d118b2eaa42aa8a7cd4025aba7128 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Leader election won for term 8
I20260809 15:46:58.956827 27582 raft_consensus.cc:686] T e14d118b2eaa42aa8a7cd4025aba7128 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 LEADER]: Becoming Leader. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Running, Role: LEADER
I20260809 15:46:58.957824 27582 consensus_queue.cc:237] T e14d118b2eaa42aa8a7cd4025aba7128 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:58.866503 27216 raft_consensus.cc:504] T 4564dfc6d4094c1b9240b367ccecf3b1 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:58.929169 27493 raft_consensus.cc:2726] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Leader election lost for term 7. Reason: Vote denied by peer 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553) with higher term. Message: Invalid argument: T 2a823d35f4f348d1b52ba9e6d80b00d2 P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d for earlier term 7. Current term is 8.
I20260809 15:46:58.954073 23472 leader_election.cc:302] T 6ae14a74a26f434db08c652e097385c8 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d
W20260809 15:46:58.955816 23283 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53562 (request call id 1766) took 1911 ms (client timeout 1513 ms). Trace:
I20260809 15:46:58.868889 27394 consensus_queue.cc:1046] T 05ad7d8fdca64e3da527280ca5be9280 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.075s
I20260809 15:46:58.857247 27334 consensus_queue.cc:1046] T c0741ab44d99439abc3aa3bdd27ee160 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.092s
I20260809 15:46:58.953577 23115 catalog_manager.cc:5495] T 213543cdbe814c89a17a963469922c1b P 65c36d164b0540f099d592f38307ee6c reported cstate change: term changed from 1 to 3, leader changed from c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66) to 65c36d164b0540f099d592f38307ee6c (127.20.202.67). New cstate: current_term: 3 leader_uuid: "65c36d164b0540f099d592f38307ee6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } health_report { overall_health: HEALTHY } } }
I20260809 15:46:58.985316 23115 catalog_manager.cc:5495] T ba56725e42f74a1baa4c04d9660727f9 P 65c36d164b0540f099d592f38307ee6c reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "65c36d164b0540f099d592f38307ee6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } health_report { overall_health: HEALTHY } } }
I20260809 15:46:58.931856 27217 raft_consensus.cc:729] T 0a51c0baaa6e432997c9d3552af4f254 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Running, Role: LEADER
W20260809 15:46:58.960445 23283 rpcz_store.cc:269] 0809 15:46:57.013533 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:57.034723 (+ 21190us) service_pool.cc:227] Handling call
0809 15:46:58.889046 (+1854323us) spinlock_profiling.cc:243] Waited 467 ms on lock 0x7b5400044054. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e92242ae 00007f34ee161c27 00007f34e7849c04 00007f34e7849b92 00007f34e7849ac1 00007f34e7849a3c 00007f34e7848cf2 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:46:58.925110 (+ 36064us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1073955200}
I20260809 15:46:58.959342 26399 raft_consensus.cc:504] T a3622f6e3e3641cea1960e3a31498bb7 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:46:58.965269 27595 log.cc:924] Time spent T 05ad7d8fdca64e3da527280ca5be9280 P 67b3aeadfa7a432abe8af4d292e04ab9: Append to log took a long time: real 0.091s user 0.001s sys 0.000s
I20260809 15:46:58.966768 27404 raft_consensus.cc:3032] T 2db057fd10e149a9895dff6ef8c9f038 P 65c36d164b0540f099d592f38307ee6c [term 4 LEADER]: Stepping down as leader of term 4
I20260809 15:46:58.969702 23270 raft_consensus.cc:2454] T 0acc47ce21c3422698d1ef7277c3ed63 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 6.
I20260809 15:46:58.971415 27197 consensus_queue.cc:1035] T 563daf42d7784db195ea99c3cb21b370 P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.012s
I20260809 15:46:58.971592 26966 consensus_queue.cc:1046] T 05ad7d8fdca64e3da527280ca5be9280 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.001s
I20260809 15:46:58.976257 23277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00a1881267a14ba8b78c81040f17a014" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:58.978091 23273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "563daf42d7784db195ea99c3cb21b370" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:58.954777 23280 raft_consensus.cc:2454] T d3be1e356b254d22b995be32f1c12f6f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 6.
I20260809 15:46:58.981241 27530 leader_election.cc:288] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:58.989711 24452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "907bacfa79d04a55a68c8dbe60056462" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:58.986816 23115 catalog_manager.cc:5495] T cef3e7277d7d4760824a3436ed4a847d P 65c36d164b0540f099d592f38307ee6c reported cstate change: term changed from 1 to 6. New cstate: current_term: 6 leader_uuid: "65c36d164b0540f099d592f38307ee6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } health_report { overall_health: HEALTHY } } }
I20260809 15:46:58.993957 23285 raft_consensus.cc:2454] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 6.
W20260809 15:46:59.025571 23468 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.68:34689, user_credentials={real_user=slave}} blocked reactor thread for 96974.2us
I20260809 15:46:59.027709 23526 raft_consensus.cc:2454] T 0acc47ce21c3422698d1ef7277c3ed63 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 7.
I20260809 15:46:59.030622 23286 raft_consensus.cc:2454] T 563daf42d7784db195ea99c3cb21b370 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 5.
I20260809 15:46:59.031713 23402 raft_consensus.cc:1226] T 4564dfc6d4094c1b9240b367ccecf3b1 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 5. Current term is 6. Ops: []
I20260809 15:46:58.861301 27496 raft_consensus.cc:504] T d2d4f749fcfd4fb3bfb8115af3471598 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:58.959962 27267 raft_consensus.cc:2781] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Leader election won for term 8
I20260809 15:46:58.962366 27487 raft_consensus.cc:2781] T ae6d2c08a7a94f58ac8dd698e3fd456c P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260809 15:46:59.049618 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 167586us
I20260809 15:46:59.054553 24452 raft_consensus.cc:2379] T 907bacfa79d04a55a68c8dbe60056462 P 61e69695258f48baa984c9d395e8ce3f [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 in current term 6: Already voted for candidate 61e69695258f48baa984c9d395e8ce3f in this term.
W20260809 15:46:59.055416 24406 leader_election.cc:334] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459): Timed out: RequestConsensusVote RPC to 127.20.202.65:35459 timed out after 1.563s (SENT)
I20260809 15:46:59.055855 24406 leader_election.cc:302] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:59.062553 23115 catalog_manager.cc:5495] T 998c6cee72cf40228c6dba064c0db2da P 65c36d164b0540f099d592f38307ee6c reported cstate change: term changed from 3 to 7. New cstate: current_term: 7 leader_uuid: "65c36d164b0540f099d592f38307ee6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } health_report { overall_health: HEALTHY } } }
I20260809 15:46:59.065109 23277 raft_consensus.cc:3037] T 00a1881267a14ba8b78c81040f17a014 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Advancing to term 8
I20260809 15:46:59.030119 27404 raft_consensus.cc:729] T 2db057fd10e149a9895dff6ef8c9f038 P 65c36d164b0540f099d592f38307ee6c [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:46:59.042830 27217 consensus_queue.cc:260] T 0a51c0baaa6e432997c9d3552af4f254 P 67b3aeadfa7a432abe8af4d292e04ab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:59.067489 23280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a51c0baaa6e432997c9d3552af4f254" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:59.068212 23270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c48a3049897452383257aa757834fb5" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:59.067481 23274 raft_consensus.cc:1261] T 563daf42d7784db195ea99c3cb21b370 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Refusing update from remote peer 61e69695258f48baa984c9d395e8ce3f: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260809 15:46:59.072228 23470 leader_election.cc:302] T 0acc47ce21c3422698d1ef7277c3ed63 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:46:59.055100 27487 raft_consensus.cc:482] T ae6d2c08a7a94f58ac8dd698e3fd456c P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:46:59.083089 23270 raft_consensus.cc:3037] T 0c48a3049897452383257aa757834fb5 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:46:59.082083 23115 catalog_manager.cc:5495] T 538a228ddfb54bc6a36528d5ff07597b P 65c36d164b0540f099d592f38307ee6c reported cstate change: term changed from 2 to 4, leader changed from 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65) to 65c36d164b0540f099d592f38307ee6c (127.20.202.67). New cstate: current_term: 4 leader_uuid: "65c36d164b0540f099d592f38307ee6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } health_report { overall_health: HEALTHY } } }
I20260809 15:46:59.083355 23282 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53554 (request call id 1539) took 2648 ms. Trace:
I20260809 15:46:59.090956 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7baa2492df0340c4a5ac241959ce1ac3" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:59.092422 23351 leader_election.cc:302] T 30c1740ef6874df0a332fbd6fd4b03ce P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
W20260809 15:46:59.093398 24406 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 50817.7us
I20260809 15:46:59.090019 23396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5c2e425a74b4e64ae70b5def750434c" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
W20260809 15:46:59.096319 23472 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 147918us
I20260809 15:46:59.096608 27363 raft_consensus.cc:482] T a88421dba80d4d3388aa5eb62e23853b P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:59.097123 27363 raft_consensus.cc:504] T a88421dba80d4d3388aa5eb62e23853b P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:59.098773 23472 leader_election.cc:302] T 4bcac4b24e7a4448abdc934bba5a624e P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:46:59.102895 24452 raft_consensus.cc:1226] T 907bacfa79d04a55a68c8dbe60056462 P 61e69695258f48baa984c9d395e8ce3f [term 6 FOLLOWER]: Rejecting Update request from peer c0f38c1908414566aa9115bb2cbdd85d for earlier term 5. Current term is 6. Ops: []
W20260809 15:46:59.105842 27597 log.cc:924] Time spent T 00a1881267a14ba8b78c81040f17a014 P c0f38c1908414566aa9115bb2cbdd85d: Append to log took a long time: real 0.180s user 0.001s sys 0.000s
I20260809 15:46:59.106547 27480 raft_consensus.cc:2726] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260809 15:46:59.107589 27512 raft_consensus.cc:2726] T 6ae14a74a26f434db08c652e097385c8 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260809 15:46:59.110112 23270 raft_consensus.cc:2454] T 0c48a3049897452383257aa757834fb5 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 4.
I20260809 15:46:59.111860 26730 raft_consensus.cc:482] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
I20260809 15:46:59.112445 26730 raft_consensus.cc:504] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:59.113720 27081 raft_consensus.cc:3032] T d1a134ba33ea4076bbaac0ae460c17ba P 65c36d164b0540f099d592f38307ee6c [term 4 LEADER]: Stepping down as leader of term 4
I20260809 15:46:59.114108 27081 raft_consensus.cc:729] T d1a134ba33ea4076bbaac0ae460c17ba P 65c36d164b0540f099d592f38307ee6c [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:46:59.114821 27378 raft_consensus.cc:3037] T 3d880de657e544468c82157769622a26 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 1 FOLLOWER]: Advancing to term 2
W20260809 15:46:59.115638 24406 leader_election.cc:334] T 294d4bcc1e7a477cb56455581277a60f P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459): Timed out: RequestConsensusVote RPC to 127.20.202.65:35459 timed out after 1.669s (SENT)
I20260809 15:46:59.116048 24406 leader_election.cc:302] T 294d4bcc1e7a477cb56455581277a60f P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:59.112102 27404 consensus_queue.cc:260] T 2db057fd10e149a9895dff6ef8c9f038 P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:59.118703 27404 raft_consensus.cc:3037] T 2db057fd10e149a9895dff6ef8c9f038 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Advancing to term 5
I20260809 15:46:59.119779 27200 raft_consensus.cc:2726] T 294d4bcc1e7a477cb56455581277a60f P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260809 15:46:59.120571 23522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5c2e425a74b4e64ae70b5def750434c" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:59.121290 23522 raft_consensus.cc:2354] T c5c2e425a74b4e64ae70b5def750434c P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 5. Current term is 6.
I20260809 15:46:59.122047 23273 raft_consensus.cc:1261] T 73f6f44c259547f1b05a3a434616fa54 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
W20260809 15:46:59.125191 23468 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.68:34689, user_credentials={real_user=slave}} blocked reactor thread for 50731.7us
I20260809 15:46:59.126025 23468 leader_election.cc:302] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c; no voters:
W20260809 15:46:59.115092 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 48387.9us
I20260809 15:46:59.112869 27216 leader_election.cc:288] T 4564dfc6d4094c1b9240b367ccecf3b1 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 6 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:59.129876 23270 raft_consensus.cc:1226] T 0c48a3049897452383257aa757834fb5 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 3. Current term is 4. Ops: []
I20260809 15:46:59.081249 27487 raft_consensus.cc:3037] T ae6d2c08a7a94f58ac8dd698e3fd456c P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Advancing to term 7
W20260809 15:46:59.143759 23472 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 45050.9us
W20260809 15:46:59.186129 23468 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.68:34689, user_credentials={real_user=slave}} blocked reactor thread for 60169.5us
W20260809 15:46:59.187620 24410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 46947.5us
I20260809 15:46:59.112068 27032 raft_consensus.cc:2741] T 0acc47ce21c3422698d1ef7277c3ed63 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
I20260809 15:46:59.103034 27588 raft_consensus.cc:2726] T 30c1740ef6874df0a332fbd6fd4b03ce P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260809 15:46:59.199484 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 59815.3us
W20260809 15:46:59.199569 23470 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 72407.9us
I20260809 15:46:59.088727 23282 rpcz_store.cc:276] 0809 15:46:56.423079 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:56.423175 (+ 96us) service_pool.cc:227] Handling call
0809 15:46:56.423484 (+ 309us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 15:46:56.423814 (+ 330us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:46:56.423840 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 15:46:56.427564 (+ 3724us) log.cc:842] Serialized 26 byte log entry
0809 15:46:56.716123 (+288559us) raft_consensus.cc:1631] Marking committed up to 3
0809 15:46:56.716158 (+ 35us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:56.716219 (+ 61us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 15:46:58.845136 (+2128917us) spinlock_profiling.cc:243] Waited 1.5 ms on lock 0x7b44000232cc. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e613cf11 00007f34e620aa95 00007f34e620b4b2 00007f34e8eb3511 00007f34e8eaee1a 00007f34e8ea2bee 00007f34e8ea1f36 00007f34e8ea1b58 00007f34e8eab551 00007f34e8eab509 00007f34e8eab499
0809 15:46:59.036433 (+191297us) raft_consensus.cc:1671] finished
0809 15:46:59.036451 (+ 18us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:59.071460 (+ 35009us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":512,"raft.run_cpu_time_us":323,"raft.run_wall_time_us":27722,"spinlock_wait_cycles":3447552,"thread_start_us":239753,"threads_started":1,"wal-append.queue_time_us":380877}
W20260809 15:46:59.186883 27294 consensus_meta.cc:287] T 2424fd9bd7f14e508fd106cbfda786a4 P 65c36d164b0540f099d592f38307ee6c: Time spent flushing consensus metadata: real 0.565s user 0.007s sys 0.002s
I20260809 15:46:59.130366 27267 raft_consensus.cc:686] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 65c36d164b0540f099d592f38307ee6c [term 8 LEADER]: Becoming Leader. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
W20260809 15:46:59.201941 27599 log.cc:924] Time spent T d3be1e356b254d22b995be32f1c12f6f P 61e69695258f48baa984c9d395e8ce3f: Append to log took a long time: real 0.099s user 0.000s sys 0.006s
I20260809 15:46:59.223536 27032 consensus_queue.cc:1046] T 0c29937a80724ddb9d07a27534a146f3 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: INVALID_TERM, Last received: 5.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 15:46:59.193542 27512 raft_consensus.cc:2781] T 4bcac4b24e7a4448abdc934bba5a624e P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260809 15:46:59.228675 23521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3924b648461e463b9d7925e5671d68ae" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:59.226759 23115 catalog_manager.cc:5495] T 4b30f28311964a0b9149a50495e83d8f P 65c36d164b0540f099d592f38307ee6c reported cstate change: term changed from 1 to 2, leader changed from c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66) to 65c36d164b0540f099d592f38307ee6c (127.20.202.67). New cstate: current_term: 2 leader_uuid: "65c36d164b0540f099d592f38307ee6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } health_report { overall_health: HEALTHY } } }
I20260809 15:46:59.115075 27081 consensus_queue.cc:260] T d1a134ba33ea4076bbaac0ae460c17ba P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:59.261744 23353 leader_election.cc:302] T 0acc47ce21c3422698d1ef7277c3ed63 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:46:59.229303 27512 raft_consensus.cc:482] T 4bcac4b24e7a4448abdc934bba5a624e P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Starting leader election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
I20260809 15:46:59.238047 23115 catalog_manager.cc:5495] T 6ae14a74a26f434db08c652e097385c8 P 65c36d164b0540f099d592f38307ee6c reported cstate change: term changed from 1 to 4, leader changed from 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65) to 65c36d164b0540f099d592f38307ee6c (127.20.202.67). New cstate: current_term: 4 leader_uuid: "65c36d164b0540f099d592f38307ee6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } health_report { overall_health: HEALTHY } } }
I20260809 15:46:59.211344 27428 consensus_queue.cc:1046] T 907bacfa79d04a55a68c8dbe60056462 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.050s
I20260809 15:46:59.256343 27496 leader_election.cc:288] T d2d4f749fcfd4fb3bfb8115af3471598 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 6 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:59.221060 27294 raft_consensus.cc:504] T 2424fd9bd7f14e508fd106cbfda786a4 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:59.279624 23519 raft_consensus.cc:1226] T 953eea268e17491ca76f3a555860a512 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Rejecting Update request from peer 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20260809 15:46:59.256958 24468 raft_consensus.cc:2454] T 3d880de657e544468c82157769622a26 P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 3.
W20260809 15:46:59.277522 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 55752.8us
I20260809 15:46:59.284067 23527 raft_consensus.cc:1226] T d2d4f749fcfd4fb3bfb8115af3471598 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Rejecting Update request from peer c0f38c1908414566aa9115bb2cbdd85d for earlier term 5. Current term is 6. Ops: []
I20260809 15:46:59.202788 25863 consensus_queue.cc:1046] T 0c48a3049897452383257aa757834fb5 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.064s
W20260809 15:46:59.284591 23395 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.505s user 0.010s sys 0.018s
W20260809 15:46:59.308660 23395 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.549s user 0.010s sys 0.019s
I20260809 15:46:59.309243 27497 raft_consensus.cc:2781] T 0acc47ce21c3422698d1ef7277c3ed63 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Leader election won for term 7
I20260809 15:46:59.310285 27428 raft_consensus.cc:3032] T 907bacfa79d04a55a68c8dbe60056462 P c0f38c1908414566aa9115bb2cbdd85d [term 5 LEADER]: Stepping down as leader of term 5
I20260809 15:46:59.310712 27428 raft_consensus.cc:729] T 907bacfa79d04a55a68c8dbe60056462 P c0f38c1908414566aa9115bb2cbdd85d [term 5 LEADER]: Becoming Follower/Learner. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:46:59.310874 27395 raft_consensus.cc:3037] T 28e17dcb5efe45b8b6a8c42da03b4819 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Advancing to term 7
I20260809 15:46:59.311527 27428 consensus_queue.cc:260] T 907bacfa79d04a55a68c8dbe60056462 P c0f38c1908414566aa9115bb2cbdd85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:59.312660 27428 raft_consensus.cc:3037] T 907bacfa79d04a55a68c8dbe60056462 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Advancing to term 6
I20260809 15:46:59.314986 25863 raft_consensus.cc:3032] T 0c48a3049897452383257aa757834fb5 P 65c36d164b0540f099d592f38307ee6c [term 3 LEADER]: Stepping down as leader of term 3
I20260809 15:46:59.315380 25863 raft_consensus.cc:729] T 0c48a3049897452383257aa757834fb5 P 65c36d164b0540f099d592f38307ee6c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:46:59.316211 25863 consensus_queue.cc:260] T 0c48a3049897452383257aa757834fb5 P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:59.317441 25863 raft_consensus.cc:3037] T 0c48a3049897452383257aa757834fb5 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:46:59.319520 23520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4564dfc6d4094c1b9240b367ccecf3b1" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:59.321573 23286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2d4f749fcfd4fb3bfb8115af3471598" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:59.322173 23286 raft_consensus.cc:3037] T d2d4f749fcfd4fb3bfb8115af3471598 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Advancing to term 6
I20260809 15:46:59.326696 24406 leader_election.cc:302] T 0a51c0baaa6e432997c9d3552af4f254 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f; no voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:59.327992 23276 raft_consensus.cc:2454] T 3d880de657e544468c82157769622a26 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 2.
I20260809 15:46:59.285311 27468 consensus_queue.cc:1035] T 73f6f44c259547f1b05a3a434616fa54 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.052s
I20260809 15:46:59.286870 23526 raft_consensus.cc:1226] T 3d880de657e544468c82157769622a26 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Rejecting Update request from peer 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 1. Current term is 3. Ops: []
I20260809 15:46:59.287564 27589 raft_consensus.cc:2781] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260809 15:46:59.288486 23345 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.68:34689, user_credentials={real_user=slave}} blocked reactor thread for 57260.2us
I20260809 15:46:59.290576 27334 consensus_queue.cc:1046] T 297c82ed906d4b48b184985eed3de601 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260809 15:46:59.291221 27053 consensus_queue.cc:1046] T 4564dfc6d4094c1b9240b367ccecf3b1 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20260809 15:46:59.330940 27497 raft_consensus.cc:686] T 0acc47ce21c3422698d1ef7277c3ed63 P c0f38c1908414566aa9115bb2cbdd85d [term 7 LEADER]: Becoming Leader. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:46:59.280994 26611 raft_consensus.cc:3037] T ba56725e42f74a1baa4c04d9660727f9 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:46:59.298662 23115 catalog_manager.cc:5495] T 953eea268e17491ca76f3a555860a512 P 65c36d164b0540f099d592f38307ee6c reported cstate change: term changed from 2 to 4, leader changed from 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65) to 65c36d164b0540f099d592f38307ee6c (127.20.202.67). New cstate: current_term: 4 leader_uuid: "65c36d164b0540f099d592f38307ee6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } health_report { overall_health: HEALTHY } } }
I20260809 15:46:59.339768 24406 leader_election.cc:302] T c0741ab44d99439abc3aa3bdd27ee160 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:59.340375 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2424fd9bd7f14e508fd106cbfda786a4" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 7 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:59.281525 27547 raft_consensus.cc:504] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:59.340646 23115 catalog_manager.cc:5495] T c0741ab44d99439abc3aa3bdd27ee160 P 65c36d164b0540f099d592f38307ee6c reported cstate change: term changed from 2 to 4, leader changed from 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) to 65c36d164b0540f099d592f38307ee6c (127.20.202.67). New cstate: current_term: 4 leader_uuid: "65c36d164b0540f099d592f38307ee6c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260809 15:46:59.341831 23282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3622f6e3e3641cea1960e3a31498bb7" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:59.299775 26730 leader_election.cc:288] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:59.342820 27556 raft_consensus.cc:2726] T 0a51c0baaa6e432997c9d3552af4f254 P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20260809 15:46:59.304970 23353 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 49171.6us
I20260809 15:46:59.343091 23274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae6d2c08a7a94f58ac8dd698e3fd456c" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:59.343500 23519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0741ab44d99439abc3aa3bdd27ee160" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:59.344226 24463 raft_consensus.cc:1226] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Rejecting Update request from peer c0f38c1908414566aa9115bb2cbdd85d for earlier term 4. Current term is 5. Ops: []
W20260809 15:46:59.344784 27426 consensus_meta.cc:287] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 61e69695258f48baa984c9d395e8ce3f: Time spent flushing consensus metadata: real 0.535s user 0.008s sys 0.000s
I20260809 15:46:59.345273 27544 raft_consensus.cc:3037] T c0741ab44d99439abc3aa3bdd27ee160 P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Advancing to term 5
W20260809 15:46:59.331120 27322 consensus_queue.cc:1159] T 0c48a3049897452383257aa757834fb5 P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" responder_term: 4 status { last_received { term: 3 index: 6 } last_committed_idx: 6 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 3. Current term is 4. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260809 15:46:59.319110 23277 raft_consensus.cc:2454] T 00a1881267a14ba8b78c81040f17a014 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 8.
I20260809 15:46:59.269331 26399 leader_election.cc:288] T a3622f6e3e3641cea1960e3a31498bb7 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:59.336143 27589 raft_consensus.cc:482] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Starting leader election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:46:59.347554 27589 raft_consensus.cc:3037] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Advancing to term 5
I20260809 15:46:59.339428 27497 consensus_queue.cc:237] T 0acc47ce21c3422698d1ef7277c3ed63 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 5.2, Last appended by leader: 2, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:59.354054 23274 raft_consensus.cc:2454] T ae6d2c08a7a94f58ac8dd698e3fd456c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 6.
I20260809 15:46:59.355736 23519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae6d2c08a7a94f58ac8dd698e3fd456c" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:46:59.356361 23519 raft_consensus.cc:2454] T ae6d2c08a7a94f58ac8dd698e3fd456c P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 6.
I20260809 15:46:59.344626 23397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3aa6a9a52804df0ba9b0ca3c82f38c3" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:59.346328 23520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e14d118b2eaa42aa8a7cd4025aba7128" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:59.359090 23520 raft_consensus.cc:3037] T e14d118b2eaa42aa8a7cd4025aba7128 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Advancing to term 8
I20260809 15:46:59.359313 23519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3aa6a9a52804df0ba9b0ca3c82f38c3" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:59.342342 27547 leader_election.cc:288] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 5 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:59.360345 23518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30c1740ef6874df0a332fbd6fd4b03ce" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:59.365778 27487 raft_consensus.cc:504] T ae6d2c08a7a94f58ac8dd698e3fd456c P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:59.337167 23283 raft_consensus.cc:1226] T c0741ab44d99439abc3aa3bdd27ee160 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 4. Current term is 5. Ops: []
I20260809 15:46:59.366837 23286 raft_consensus.cc:2454] T d2d4f749fcfd4fb3bfb8115af3471598 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 6.
I20260809 15:46:59.369493 27544 raft_consensus.cc:2726] T c0741ab44d99439abc3aa3bdd27ee160 P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260809 15:46:59.370095 23394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73f6f44c259547f1b05a3a434616fa54" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:59.370277 23277 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "28e17dcb5efe45b8b6a8c42da03b4819"
dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
mode: GRACEFUL
new_leader_uuid: "65c36d164b0540f099d592f38307ee6c"
from {username='slave'} at 127.0.0.1:33468
I20260809 15:46:59.370675 23394 raft_consensus.cc:2454] T 73f6f44c259547f1b05a3a434616fa54 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 6.
I20260809 15:46:59.370986 23277 raft_consensus.cc:595] T 28e17dcb5efe45b8b6a8c42da03b4819 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Received request to transfer leadership to TS 65c36d164b0540f099d592f38307ee6c
I20260809 15:46:59.268169 27512 raft_consensus.cc:3037] T 4bcac4b24e7a4448abdc934bba5a624e P 65c36d164b0540f099d592f38307ee6c [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:59.371356 23277 raft_consensus.cc:603] T 28e17dcb5efe45b8b6a8c42da03b4819 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260809 15:46:59.371742 23400 raft_consensus.cc:1226] T 6ae14a74a26f434db08c652e097385c8 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Rejecting Update request from peer 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 5. Current term is 6. Ops: []
I20260809 15:46:59.372403 23277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73f6f44c259547f1b05a3a434616fa54" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:59.372795 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2424fd9bd7f14e508fd106cbfda786a4" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 7 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:59.373445 23277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "538a228ddfb54bc6a36528d5ff07597b" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:59.373791 23392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0acc47ce21c3422698d1ef7277c3ed63" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:59.373442 23402 raft_consensus.cc:2454] T 2424fd9bd7f14e508fd106cbfda786a4 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 6.
I20260809 15:46:59.375020 23399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3296ef50be254fc29cfc77d22a8f7a23" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:59.378415 23399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae6d2c08a7a94f58ac8dd698e3fd456c" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:59.379164 23399 raft_consensus.cc:2379] T ae6d2c08a7a94f58ac8dd698e3fd456c P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 in current term 7: Already voted for candidate c0f38c1908414566aa9115bb2cbdd85d in this term.
I20260809 15:46:59.379527 27267 consensus_queue.cc:237] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:59.380563 23397 raft_consensus.cc:1261] T e71a05ba1d4d42bd95c3f85f9313399e P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Refusing update from remote peer 67b3aeadfa7a432abe8af4d292e04ab9: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260809 15:46:59.381047 23273 raft_consensus.cc:1226] T d2d4f749fcfd4fb3bfb8115af3471598 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Rejecting Update request from peer c0f38c1908414566aa9115bb2cbdd85d for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20260809 15:46:59.381953 23392 raft_consensus.cc:1226] T f9e7cadcd24d45fcbfa84209225e8f2d P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Rejecting Update request from peer 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 5. Current term is 6. Ops: [5.5-5.5]
I20260809 15:46:59.382146 27394 raft_consensus.cc:3032] T c0741ab44d99439abc3aa3bdd27ee160 P 65c36d164b0540f099d592f38307ee6c [term 4 LEADER]: Stepping down as leader of term 4
I20260809 15:46:59.382560 27394 raft_consensus.cc:729] T c0741ab44d99439abc3aa3bdd27ee160 P 65c36d164b0540f099d592f38307ee6c [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:46:59.383772 23400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30c1740ef6874df0a332fbd6fd4b03ce" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
W20260809 15:46:59.385664 27196 consensus_meta.cc:287] T d3e94b7e50754a81b3d72a07476cb7b9 P 61e69695258f48baa984c9d395e8ce3f: Time spent flushing consensus metadata: real 0.589s user 0.004s sys 0.004s
I20260809 15:46:59.381992 23273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0acc47ce21c3422698d1ef7277c3ed63" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 7 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:59.388175 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f9d6565aa4c45a3b3f01210d0eb71f7" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:59.389942 23274 raft_consensus.cc:1261] T d3be1e356b254d22b995be32f1c12f6f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260809 15:46:59.383430 24463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad8a3b25d7b49e5a2142d5709edd0c7" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:46:59.392557 27294 leader_election.cc:288] T 2424fd9bd7f14e508fd106cbfda786a4 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 7 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:59.393350 23527 raft_consensus.cc:2379] T 2424fd9bd7f14e508fd106cbfda786a4 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 in current term 7: Already voted for candidate 65c36d164b0540f099d592f38307ee6c in this term.
I20260809 15:46:59.392678 23283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ae14a74a26f434db08c652e097385c8" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:59.394559 27378 raft_consensus.cc:482] T cef3e7277d7d4760824a3436ed4a847d P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:59.383437 27394 consensus_queue.cc:260] T c0741ab44d99439abc3aa3bdd27ee160 P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:59.395141 23228 leader_election.cc:302] T d0721a4403a440f383318a3cea9d1442 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:59.395144 27378 raft_consensus.cc:504] T cef3e7277d7d4760824a3436ed4a847d P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:59.395597 27488 consensus_queue.cc:1046] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260809 15:46:59.396425 23228 leader_election.cc:302] T d1a134ba33ea4076bbaac0ae460c17ba P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:59.396807 27263 raft_consensus.cc:3032] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P c0f38c1908414566aa9115bb2cbdd85d [term 4 LEADER]: Stepping down as leader of term 4
I20260809 15:46:59.390179 23277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9e7cadcd24d45fcbfa84209225e8f2d" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:59.387243 23273 raft_consensus.cc:3037] T 0acc47ce21c3422698d1ef7277c3ed63 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Advancing to term 7
I20260809 15:46:59.399911 23470 leader_election.cc:302] T 00a1881267a14ba8b78c81040f17a014 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:46:59.398296 23400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "907bacfa79d04a55a68c8dbe60056462" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:59.341274 24406 leader_election.cc:302] T d3be1e356b254d22b995be32f1c12f6f P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 67b3aeadfa7a432abe8af4d292e04ab9; no voters: c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:59.397234 27378 leader_election.cc:288] T cef3e7277d7d4760824a3436ed4a847d P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:59.417595 23470 leader_election.cc:302] T d2d4f749fcfd4fb3bfb8115af3471598 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
W20260809 15:46:59.378641 27560 log.cc:924] Time spent T e71a05ba1d4d42bd95c3f85f9313399e P 61e69695258f48baa984c9d395e8ce3f: Append to log took a long time: real 0.123s user 0.000s sys 0.000s
I20260809 15:46:59.421890 23520 raft_consensus.cc:2454] T e14d118b2eaa42aa8a7cd4025aba7128 P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 8.
I20260809 15:46:59.423406 24410 leader_election.cc:302] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:59.342450 23282 raft_consensus.cc:2454] T a3622f6e3e3641cea1960e3a31498bb7 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 4.
I20260809 15:46:59.397612 27162 raft_consensus.cc:2726] T d0721a4403a440f383318a3cea9d1442 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260809 15:46:59.342445 23115 catalog_manager.cc:5495] T 2424fd9bd7f14e508fd106cbfda786a4 P 65c36d164b0540f099d592f38307ee6c reported cstate change: term changed from 2 to 5, leader changed from 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65) to 65c36d164b0540f099d592f38307ee6c (127.20.202.67). New cstate: current_term: 5 leader_uuid: "65c36d164b0540f099d592f38307ee6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } health_report { overall_health: HEALTHY } } }
I20260809 15:46:59.449556 23274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2db057fd10e149a9895dff6ef8c9f038" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:59.454628 23285 raft_consensus.cc:1226] T 00a1881267a14ba8b78c81040f17a014 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Rejecting Update request from peer c0f38c1908414566aa9115bb2cbdd85d for earlier term 7. Current term is 8. Ops: [5.4-7.5]
I20260809 15:46:59.397207 27263 raft_consensus.cc:729] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P c0f38c1908414566aa9115bb2cbdd85d [term 4 LEADER]: Becoming Follower/Learner. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:46:59.458066 23115 catalog_manager.cc:5495] T f7715d71dbac447b9351fc5eb36c5e80 P 65c36d164b0540f099d592f38307ee6c reported cstate change: term changed from 2 to 4, leader changed from 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65) to 65c36d164b0540f099d592f38307ee6c (127.20.202.67). New cstate: current_term: 4 leader_uuid: "65c36d164b0540f099d592f38307ee6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } health_report { overall_health: HEALTHY } } }
I20260809 15:46:59.398761 27334 raft_consensus.cc:3032] T 297c82ed906d4b48b184985eed3de601 P 65c36d164b0540f099d592f38307ee6c [term 6 LEADER]: Stepping down as leader of term 6
I20260809 15:46:59.422708 27468 raft_consensus.cc:2781] T 00a1881267a14ba8b78c81040f17a014 P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Leader election won for term 8
I20260809 15:46:59.434082 27196 raft_consensus.cc:2781] T d3be1e356b254d22b995be32f1c12f6f P 61e69695258f48baa984c9d395e8ce3f [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260809 15:46:59.449517 27521 raft_consensus.cc:2726] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260809 15:46:59.463877 23282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c29937a80724ddb9d07a27534a146f3" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:59.460311 23520 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4564dfc6d4094c1b9240b367ccecf3b1"
dest_uuid: "65c36d164b0540f099d592f38307ee6c"
mode: GRACEFUL
new_leader_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
from {username='slave'} at 127.0.0.1:48888
W20260809 15:46:59.465725 23393 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.716s user 0.019s sys 0.048s
I20260809 15:46:59.466568 23276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "297c82ed906d4b48b184985eed3de601" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
W20260809 15:46:59.507812 23393 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.767s user 0.020s sys 0.048s
I20260809 15:46:59.452105 27347 raft_consensus.cc:2726] T d1a134ba33ea4076bbaac0ae460c17ba P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260809 15:46:59.433841 27294 consensus_queue.cc:1046] T c0741ab44d99439abc3aa3bdd27ee160 P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.077s
I20260809 15:46:59.514586 23274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cef3e7277d7d4760824a3436ed4a847d" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:59.518385 23274 raft_consensus.cc:2454] T cef3e7277d7d4760824a3436ed4a847d P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 6.
I20260809 15:46:59.462401 27334 raft_consensus.cc:729] T 297c82ed906d4b48b184985eed3de601 P 65c36d164b0540f099d592f38307ee6c [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
W20260809 15:46:59.517408 24406 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 176198us
W20260809 15:46:59.516620 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 59263.8us
I20260809 15:46:59.418726 27426 raft_consensus.cc:504] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 61e69695258f48baa984c9d395e8ce3f [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:59.523554 27305 raft_consensus.cc:3032] T 05ad7d8fdca64e3da527280ca5be9280 P 65c36d164b0540f099d592f38307ee6c [term 5 LEADER]: Stepping down as leader of term 5
I20260809 15:46:59.523916 27268 raft_consensus.cc:2781] T d2d4f749fcfd4fb3bfb8115af3471598 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Leader election won for term 6
I20260809 15:46:59.529258 27217 raft_consensus.cc:3037] T 0a51c0baaa6e432997c9d3552af4f254 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Advancing to term 5
I20260809 15:46:59.452541 27583 consensus_queue.cc:1046] T f9e7cadcd24d45fcbfa84209225e8f2d P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.060s
I20260809 15:46:59.532021 23228 leader_election.cc:302] T 7baa2492df0340c4a5ac241959ce1ac3 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:59.536172 27334 consensus_queue.cc:260] T 297c82ed906d4b48b184985eed3de601 P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:59.537246 27363 leader_election.cc:288] T a88421dba80d4d3388aa5eb62e23853b P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:59.538512 27263 consensus_queue.cc:260] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P c0f38c1908414566aa9115bb2cbdd85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:59.539605 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0acc47ce21c3422698d1ef7277c3ed63" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:59.521152 27487 leader_election.cc:288] T ae6d2c08a7a94f58ac8dd698e3fd456c P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 7 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:59.541551 27081 raft_consensus.cc:3037] T d1a134ba33ea4076bbaac0ae460c17ba P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Advancing to term 5
I20260809 15:46:59.545073 23285 raft_consensus.cc:1261] T c5c2e425a74b4e64ae70b5def750434c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260809 15:46:59.465780 27162 consensus_queue.cc:1046] T 6ae14a74a26f434db08c652e097385c8 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.081s
I20260809 15:46:59.546988 27491 raft_consensus.cc:3032] T 0c29937a80724ddb9d07a27534a146f3 P 65c36d164b0540f099d592f38307ee6c [term 5 LEADER]: Stepping down as leader of term 5
I20260809 15:46:59.547492 23282 raft_consensus.cc:3032] T 953eea268e17491ca76f3a555860a512 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 LEADER]: Stepping down as leader of term 5
I20260809 15:46:59.582199 23282 raft_consensus.cc:729] T 953eea268e17491ca76f3a555860a512 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Running, Role: LEADER
I20260809 15:46:59.553622 24448 raft_consensus.cc:2454] T d3e94b7e50754a81b3d72a07476cb7b9 P 61e69695258f48baa984c9d395e8ce3f [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 7.
I20260809 15:46:59.466229 27196 raft_consensus.cc:482] T d3be1e356b254d22b995be32f1c12f6f P 61e69695258f48baa984c9d395e8ce3f [term 6 FOLLOWER]: Starting leader election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:46:59.560995 27394 raft_consensus.cc:3037] T c0741ab44d99439abc3aa3bdd27ee160 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Advancing to term 5
I20260809 15:46:59.563445 27589 raft_consensus.cc:504] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:59.503620 27378 raft_consensus.cc:482] T df20671a7b874dbb98c0ef28371c1308 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:59.572199 27583 raft_consensus.cc:3032] T f9e7cadcd24d45fcbfa84209225e8f2d P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 LEADER]: Stepping down as leader of term 5
I20260809 15:46:59.575500 23526 raft_consensus.cc:2354] T 0acc47ce21c3422698d1ef7277c3ed63 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 6. Current term is 7.
W20260809 15:46:59.575196 23472 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 285999us
I20260809 15:46:59.579175 23273 raft_consensus.cc:2454] T 0acc47ce21c3422698d1ef7277c3ed63 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 7.
I20260809 15:46:59.580300 23400 raft_consensus.cc:2396] T 907bacfa79d04a55a68c8dbe60056462 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 for term 6 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260809 15:46:59.581717 23270 raft_consensus.cc:1226] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Rejecting Update request from peer c0f38c1908414566aa9115bb2cbdd85d for earlier term 5. Current term is 6. Ops: []
I20260809 15:46:59.567705 27305 raft_consensus.cc:729] T 05ad7d8fdca64e3da527280ca5be9280 P 65c36d164b0540f099d592f38307ee6c [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
W20260809 15:46:59.584156 24406 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 62003.6us
I20260809 15:46:59.587877 27196 raft_consensus.cc:3037] T d3be1e356b254d22b995be32f1c12f6f P 61e69695258f48baa984c9d395e8ce3f [term 6 FOLLOWER]: Advancing to term 7
I20260809 15:46:59.617185 23285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4564dfc6d4094c1b9240b367ccecf3b1" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:59.618866 27512 raft_consensus.cc:504] T 4bcac4b24e7a4448abdc934bba5a624e P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:59.619931 27305 consensus_queue.cc:260] T 05ad7d8fdca64e3da527280ca5be9280 P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:59.620980 27305 raft_consensus.cc:3037] T 05ad7d8fdca64e3da527280ca5be9280 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Advancing to term 6
I20260809 15:46:59.622566 27316 raft_consensus.cc:2726] T f7715d71dbac447b9351fc5eb36c5e80 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260809 15:46:59.623566 23470 leader_election.cc:302] T a3622f6e3e3641cea1960e3a31498bb7 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:46:59.623368 23282 consensus_queue.cc:260] T 953eea268e17491ca76f3a555860a512 P 67b3aeadfa7a432abe8af4d292e04ab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:59.624066 23276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae6d2c08a7a94f58ac8dd698e3fd456c" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:59.624631 27518 raft_consensus.cc:2726] T 7baa2492df0340c4a5ac241959ce1ac3 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260809 15:46:59.625454 27263 raft_consensus.cc:3037] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Advancing to term 5
I20260809 15:46:59.625747 27426 leader_election.cc:288] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 9 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:59.624656 23276 raft_consensus.cc:3037] T ae6d2c08a7a94f58ac8dd698e3fd456c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Advancing to term 7
I20260809 15:46:59.624341 23282 raft_consensus.cc:3037] T 953eea268e17491ca76f3a555860a512 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Advancing to term 6
I20260809 15:46:59.631646 23527 raft_consensus.cc:1226] T 4bcac4b24e7a4448abdc934bba5a624e P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Rejecting Update request from peer 61e69695258f48baa984c9d395e8ce3f for earlier term 2. Current term is 3. Ops: []
W20260809 15:46:59.631839 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 99882.2us
I20260809 15:46:59.632702 23228 leader_election.cc:398] T c5c2e425a74b4e64ae70b5def750434c P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 election: Vote denied by peer 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553) with higher term. Message: Invalid argument: T c5c2e425a74b4e64ae70b5def750434c P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 5. Current term is 6.
I20260809 15:46:59.632977 23228 leader_election.cc:401] T c5c2e425a74b4e64ae70b5def750434c P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
I20260809 15:46:59.582686 27268 raft_consensus.cc:686] T d2d4f749fcfd4fb3bfb8115af3471598 P 65c36d164b0540f099d592f38307ee6c [term 6 LEADER]: Becoming Leader. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:46:59.644170 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a823d35f4f348d1b52ba9e6d80b00d2" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:46:59.582121 27491 raft_consensus.cc:729] T 0c29937a80724ddb9d07a27534a146f3 P 65c36d164b0540f099d592f38307ee6c [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:46:59.646449 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad8a3b25d7b49e5a2142d5709edd0c7" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:59.647351 23399 raft_consensus.cc:3037] T 73f6f44c259547f1b05a3a434616fa54 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Advancing to term 7
I20260809 15:46:59.643419 23351 leader_election.cc:302] T 73f6f44c259547f1b05a3a434616fa54 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:59.627041 27583 raft_consensus.cc:729] T f9e7cadcd24d45fcbfa84209225e8f2d P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Running, Role: LEADER
I20260809 15:46:59.657812 23397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3aa6a9a52804df0ba9b0ca3c82f38c3" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:59.658561 23394 raft_consensus.cc:1226] T 538a228ddfb54bc6a36528d5ff07597b P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 4. Current term is 5. Ops: []
I20260809 15:46:59.653506 23392 raft_consensus.cc:1226] T ba56725e42f74a1baa4c04d9660727f9 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 3. Current term is 4. Ops: []
I20260809 15:46:59.620925 27512 leader_election.cc:288] T 4bcac4b24e7a4448abdc934bba5a624e P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:59.646687 23395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a823d35f4f348d1b52ba9e6d80b00d2" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:59.662082 23395 raft_consensus.cc:3037] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Advancing to term 9
I20260809 15:46:59.641189 27507 raft_consensus.cc:2781] T a3622f6e3e3641cea1960e3a31498bb7 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260809 15:46:59.650056 27334 raft_consensus.cc:3037] T 297c82ed906d4b48b184985eed3de601 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Advancing to term 7
I20260809 15:46:59.670890 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bcac4b24e7a4448abdc934bba5a624e" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:59.640928 27378 raft_consensus.cc:504] T df20671a7b874dbb98c0ef28371c1308 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:59.664420 27347 raft_consensus.cc:482] T 28e17dcb5efe45b8b6a8c42da03b4819 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:46:59.668228 27507 raft_consensus.cc:482] T a3622f6e3e3641cea1960e3a31498bb7 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:46:59.676753 23394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00a1881267a14ba8b78c81040f17a014" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:59.676116 23392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2424fd9bd7f14e508fd106cbfda786a4" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 7 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:59.681233 23392 raft_consensus.cc:3037] T 2424fd9bd7f14e508fd106cbfda786a4 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Advancing to term 7
I20260809 15:46:59.681535 23228 leader_election.cc:302] T c0741ab44d99439abc3aa3bdd27ee160 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c
I20260809 15:46:59.668700 27512 raft_consensus.cc:482] T 763e58d6ba644eea830a6c1664d39c9b P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:46:59.687055 23520 raft_consensus.cc:595] T 4564dfc6d4094c1b9240b367ccecf3b1 P 65c36d164b0540f099d592f38307ee6c [term 5 LEADER]: Received request to transfer leadership to TS c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:59.687415 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3622f6e3e3641cea1960e3a31498bb7" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:59.699424 27428 consensus_queue.cc:1046] T d2d4f749fcfd4fb3bfb8115af3471598 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 15:46:59.700304 27428 raft_consensus.cc:3032] T d2d4f749fcfd4fb3bfb8115af3471598 P c0f38c1908414566aa9115bb2cbdd85d [term 5 LEADER]: Stepping down as leader of term 5
I20260809 15:46:59.700647 27428 raft_consensus.cc:729] T d2d4f749fcfd4fb3bfb8115af3471598 P c0f38c1908414566aa9115bb2cbdd85d [term 5 LEADER]: Becoming Follower/Learner. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:46:59.702946 27378 leader_election.cc:288] T df20671a7b874dbb98c0ef28371c1308 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:59.703481 27378 raft_consensus.cc:482] T 4b30f28311964a0b9149a50495e83d8f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:46:59.703922 27378 raft_consensus.cc:504] T 4b30f28311964a0b9149a50495e83d8f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:59.705672 27378 leader_election.cc:288] T 4b30f28311964a0b9149a50495e83d8f P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:59.706115 27378 raft_consensus.cc:2726] T c5c2e425a74b4e64ae70b5def750434c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553) with higher term. Message: Invalid argument: T c5c2e425a74b4e64ae70b5def750434c P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 5. Current term is 6.
I20260809 15:46:59.706612 27378 raft_consensus.cc:2726] T c0741ab44d99439abc3aa3bdd27ee160 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260809 15:46:59.708089 26619 raft_consensus.cc:3032] T 4564dfc6d4094c1b9240b367ccecf3b1 P 65c36d164b0540f099d592f38307ee6c [term 5 LEADER]: Stepping down as leader of term 5
I20260809 15:46:59.708521 26619 raft_consensus.cc:729] T 4564dfc6d4094c1b9240b367ccecf3b1 P 65c36d164b0540f099d592f38307ee6c [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:46:59.709614 26619 consensus_queue.cc:260] T 4564dfc6d4094c1b9240b367ccecf3b1 P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:59.682884 27347 raft_consensus.cc:504] T 28e17dcb5efe45b8b6a8c42da03b4819 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:59.688150 23402 raft_consensus.cc:2454] T a3622f6e3e3641cea1960e3a31498bb7 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 4.
I20260809 15:46:59.712080 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a88421dba80d4d3388aa5eb62e23853b" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:59.712445 27347 leader_election.cc:288] T 28e17dcb5efe45b8b6a8c42da03b4819 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:46:59.712648 23402 raft_consensus.cc:2454] T a88421dba80d4d3388aa5eb62e23853b P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 4.
I20260809 15:46:59.713740 23402 raft_consensus.cc:1226] T 07fcabb569414bddbd39df6cedd24e64 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 5. Current term is 6. Ops: []
I20260809 15:46:59.710884 26619 raft_consensus.cc:3037] T 4564dfc6d4094c1b9240b367ccecf3b1 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Advancing to term 6
I20260809 15:46:59.683012 27507 raft_consensus.cc:3037] T a3622f6e3e3641cea1960e3a31498bb7 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Advancing to term 5
I20260809 15:46:59.719880 23402 raft_consensus.cc:1226] T 05ad7d8fdca64e3da527280ca5be9280 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 5. Current term is 6. Ops: []
I20260809 15:46:59.720760 27268 consensus_queue.cc:237] T d2d4f749fcfd4fb3bfb8115af3471598 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:59.727596 27583 consensus_queue.cc:260] T f9e7cadcd24d45fcbfa84209225e8f2d P 67b3aeadfa7a432abe8af4d292e04ab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:59.728840 27583 raft_consensus.cc:3037] T f9e7cadcd24d45fcbfa84209225e8f2d P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Advancing to term 6
I20260809 15:46:59.731916 23282 raft_consensus.cc:1261] T 953eea268e17491ca76f3a555860a512 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 4. (term mismatch)
I20260809 15:46:59.732270 23282 pending_rounds.cc:76] T 953eea268e17491ca76f3a555860a512 P 67b3aeadfa7a432abe8af4d292e04ab9: Aborting all ops after (but not including) 3
I20260809 15:46:59.732496 23282 pending_rounds.cc:97] T 953eea268e17491ca76f3a555860a512 P 67b3aeadfa7a432abe8af4d292e04ab9: Aborting uncommitted NO_OP operation due to leader change: 5.4
I20260809 15:46:59.732731 23282 raft_consensus.cc:2866] T 953eea268e17491ca76f3a555860a512 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 15:46:59.727149 27196 raft_consensus.cc:504] T d3be1e356b254d22b995be32f1c12f6f P 61e69695258f48baa984c9d395e8ce3f [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:59.742211 27196 leader_election.cc:288] T d3be1e356b254d22b995be32f1c12f6f P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 7 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:59.743618 23394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2d4f749fcfd4fb3bfb8115af3471598" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:59.742815 27196 consensus_queue.cc:1046] T 4bcac4b24e7a4448abdc934bba5a624e P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260809 15:46:59.745141 27480 raft_consensus.cc:3032] T 4bcac4b24e7a4448abdc934bba5a624e P 61e69695258f48baa984c9d395e8ce3f [term 2 LEADER]: Stepping down as leader of term 2
I20260809 15:46:59.745513 27480 raft_consensus.cc:729] T 4bcac4b24e7a4448abdc934bba5a624e P 61e69695258f48baa984c9d395e8ce3f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 61e69695258f48baa984c9d395e8ce3f, State: Running, Role: LEADER
I20260809 15:46:59.686760 23351 leader_election.cc:302] T 538a228ddfb54bc6a36528d5ff07597b P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
W20260809 15:46:59.747045 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 69655.1us
I20260809 15:46:59.752210 23392 raft_consensus.cc:2454] T 2424fd9bd7f14e508fd106cbfda786a4 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 7.
W20260809 15:46:59.751936 23472 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 65355.7us
I20260809 15:46:59.753400 23472 leader_election.cc:302] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:59.755537 23472 leader_election.cc:302] T a88421dba80d4d3388aa5eb62e23853b P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:46:59.756719 23395 raft_consensus.cc:2454] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 9.
I20260809 15:46:59.756227 23397 raft_consensus.cc:2454] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 5.
I20260809 15:46:59.765458 24410 leader_election.cc:302] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c
I20260809 15:46:59.683986 27404 consensus_queue.cc:1046] T 538a228ddfb54bc6a36528d5ff07597b P 65c36d164b0540f099d592f38307ee6c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.073s
I20260809 15:46:59.766667 23392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3be1e356b254d22b995be32f1c12f6f" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
W20260809 15:46:59.776849 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 90162.1us
I20260809 15:46:59.777818 23351 leader_election.cc:302] T f9e7cadcd24d45fcbfa84209225e8f2d P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:59.778517 27487 consensus_queue.cc:1035] T d3be1e356b254d22b995be32f1c12f6f P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20260809 15:46:59.778861 23351 leader_election.cc:302] T 6ae14a74a26f434db08c652e097385c8 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:59.779170 26730 raft_consensus.cc:2726] T f9e7cadcd24d45fcbfa84209225e8f2d P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260809 15:46:59.774801 27426 raft_consensus.cc:2781] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 61e69695258f48baa984c9d395e8ce3f [term 9 FOLLOWER]: Leader election won for term 9
I20260809 15:46:59.779747 27480 consensus_queue.cc:260] T 4bcac4b24e7a4448abdc934bba5a624e P 61e69695258f48baa984c9d395e8ce3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:46:59.780911 23399 raft_consensus.cc:1261] T 73f6f44c259547f1b05a3a434616fa54 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260809 15:46:59.768338 27294 raft_consensus.cc:2781] T a88421dba80d4d3388aa5eb62e23853b P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260809 15:46:59.783967 27404 raft_consensus.cc:3032] T 538a228ddfb54bc6a36528d5ff07597b P 65c36d164b0540f099d592f38307ee6c [term 4 LEADER]: Stepping down as leader of term 4
I20260809 15:46:59.784576 23228 leader_election.cc:302] T ae6d2c08a7a94f58ac8dd698e3fd456c P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:46:59.701434 27428 consensus_queue.cc:260] T d2d4f749fcfd4fb3bfb8115af3471598 P c0f38c1908414566aa9115bb2cbdd85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:59.745669 27512 raft_consensus.cc:504] T 763e58d6ba644eea830a6c1664d39c9b P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:59.789003 23276 raft_consensus.cc:2454] T ae6d2c08a7a94f58ac8dd698e3fd456c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 7.
I20260809 15:46:59.762849 27053 raft_consensus.cc:2726] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260809 15:46:59.790841 23472 leader_election.cc:302] T 2424fd9bd7f14e508fd106cbfda786a4 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:46:59.803445 27512 leader_election.cc:288] T 763e58d6ba644eea830a6c1664d39c9b P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:59.798146 27395 raft_consensus.cc:2726] T 73f6f44c259547f1b05a3a434616fa54 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260809 15:46:59.795961 27404 raft_consensus.cc:729] T 538a228ddfb54bc6a36528d5ff07597b P 65c36d164b0540f099d592f38307ee6c [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:46:59.808507 27404 consensus_queue.cc:260] T 538a228ddfb54bc6a36528d5ff07597b P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:59.809779 27404 raft_consensus.cc:3037] T 538a228ddfb54bc6a36528d5ff07597b P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Advancing to term 5
I20260809 15:46:59.790328 27263 raft_consensus.cc:2726] T 538a228ddfb54bc6a36528d5ff07597b P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260809 15:46:59.812690 23228 leader_election.cc:398] T 0acc47ce21c3422698d1ef7277c3ed63 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 6 election: Vote denied by peer 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553) with higher term. Message: Invalid argument: T 0acc47ce21c3422698d1ef7277c3ed63 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 6. Current term is 7.
I20260809 15:46:59.813021 23228 leader_election.cc:401] T 0acc47ce21c3422698d1ef7277c3ed63 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 6 election: Cancelling election due to peer responding with higher term
I20260809 15:46:59.814579 23228 leader_election.cc:302] T 73f6f44c259547f1b05a3a434616fa54 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c
I20260809 15:46:59.815405 27582 raft_consensus.cc:2741] T ae6d2c08a7a94f58ac8dd698e3fd456c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
I20260809 15:46:59.815963 27582 raft_consensus.cc:2726] T 0acc47ce21c3422698d1ef7277c3ed63 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Leader election lost for term 6. Reason: Vote denied by peer 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553) with higher term. Message: Invalid argument: T 0acc47ce21c3422698d1ef7277c3ed63 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 6. Current term is 7.
I20260809 15:46:59.816454 27582 raft_consensus.cc:2741] T 73f6f44c259547f1b05a3a434616fa54 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20260809 15:46:59.821091 23399 raft_consensus.cc:1226] T 4564dfc6d4094c1b9240b367ccecf3b1 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 5. Current term is 6. Ops: []
I20260809 15:46:59.823450 23395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3aa6a9a52804df0ba9b0ca3c82f38c3" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:46:59.796306 27294 raft_consensus.cc:482] T a88421dba80d4d3388aa5eb62e23853b P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260809 15:46:59.830587 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 51813.4us
I20260809 15:46:59.831642 27428 raft_consensus.cc:3037] T d2d4f749fcfd4fb3bfb8115af3471598 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Advancing to term 6
I20260809 15:46:59.795830 27426 raft_consensus.cc:686] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 61e69695258f48baa984c9d395e8ce3f [term 9 LEADER]: Becoming Leader. State: Replica: 61e69695258f48baa984c9d395e8ce3f, State: Running, Role: LEADER
I20260809 15:46:59.834103 27053 consensus_queue.cc:1035] T 73f6f44c259547f1b05a3a434616fa54 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260809 15:46:59.796340 27487 raft_consensus.cc:2726] T 6ae14a74a26f434db08c652e097385c8 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260809 15:46:59.807560 27081 raft_consensus.cc:2781] T 2424fd9bd7f14e508fd106cbfda786a4 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Leader election won for term 7
I20260809 15:46:59.843111 23351 leader_election.cc:302] T 2db057fd10e149a9895dff6ef8c9f038 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:59.829577 27589 leader_election.cc:288] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 5 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:46:59.847497 23395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "763e58d6ba644eea830a6c1664d39c9b" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:46:59.848109 23395 raft_consensus.cc:2454] T 763e58d6ba644eea830a6c1664d39c9b P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 6.
I20260809 15:46:59.850579 23392 raft_consensus.cc:1226] T 0c29937a80724ddb9d07a27534a146f3 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 5. Current term is 6. Ops: []
W20260809 15:46:59.851614 23404 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.658s user 0.051s sys 0.064s
I20260809 15:46:59.840924 27294 raft_consensus.cc:3037] T a88421dba80d4d3388aa5eb62e23853b P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Advancing to term 5
I20260809 15:46:59.862758 23472 leader_election.cc:302] T 763e58d6ba644eea830a6c1664d39c9b P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:46:59.854929 23228 leader_election.cc:302] T 2424fd9bd7f14e508fd106cbfda786a4 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c
I20260809 15:46:59.886343 23228 leader_election.cc:302] T 3296ef50be254fc29cfc77d22a8f7a23 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d
W20260809 15:46:59.853935 23404 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.661s user 0.051s sys 0.064s
W20260809 15:46:59.889328 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 46317us
I20260809 15:46:59.889990 23351 leader_election.cc:302] T 297c82ed906d4b48b184985eed3de601 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:59.857041 27487 raft_consensus.cc:2726] T 2db057fd10e149a9895dff6ef8c9f038 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20260809 15:46:59.857003 27497 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.501s user 0.013s sys 0.025s
W20260809 15:46:59.881642 27496 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.549s user 0.015s sys 0.025s
W20260809 15:46:59.868016 23468 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.592s user 0.018s sys 0.020s
I20260809 15:46:59.855417 27480 raft_consensus.cc:3037] T 4bcac4b24e7a4448abdc934bba5a624e P 61e69695258f48baa984c9d395e8ce3f [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:46:59.902879 23404 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55320 (request call id 1423) took 1263 ms. Trace:
W20260809 15:46:59.904036 23468 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.632s user 0.019s sys 0.020s
W20260809 15:46:59.905977 23468 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.68:34689, user_credentials={real_user=slave}} blocked reactor thread for 645844us
W20260809 15:46:59.881724 27531 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.559s user 0.013s sys 0.031s
I20260809 15:46:59.939248 27395 raft_consensus.cc:2726] T 297c82ed906d4b48b184985eed3de601 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260809 15:46:59.964318 27316 raft_consensus.cc:482] T 998c6cee72cf40228c6dba064c0db2da P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:46:59.853694 27263 consensus_queue.cc:1046] T 00a1881267a14ba8b78c81040f17a014 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.040s
I20260809 15:46:59.942322 23393 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55320 (request call id 1418) took 1353 ms. Trace:
I20260809 15:46:59.956035 23280 raft_consensus.cc:2454] T 0a51c0baaa6e432997c9d3552af4f254 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 5.
I20260809 15:46:59.931671 23283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "763e58d6ba644eea830a6c1664d39c9b" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:59.887641 27582 raft_consensus.cc:2781] T 2424fd9bd7f14e508fd106cbfda786a4 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260809 15:46:59.920182 23468 leader_election.cc:302] T 3d880de657e544468c82157769622a26 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c; no voters: 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:59.921380 27426 consensus_queue.cc:237] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
W20260809 15:46:59.903391 27497 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.547s user 0.013s sys 0.025s
I20260809 15:46:59.887754 26619 raft_consensus.cc:2781] T 763e58d6ba644eea830a6c1664d39c9b P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260809 15:46:59.903626 27496 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.584s user 0.015s sys 0.025s
W20260809 15:46:59.953533 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 46616.8us
I20260809 15:46:59.939653 23274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2424fd9bd7f14e508fd106cbfda786a4" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 7 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:46:59.908763 23351 leader_election.cc:302] T 0c29937a80724ddb9d07a27534a146f3 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:46:59.942173 23282 raft_consensus.cc:1226] T 05ad7d8fdca64e3da527280ca5be9280 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 LEADER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 5. Current term is 6. Ops: []
I20260809 15:46:59.865875 27081 raft_consensus.cc:686] T 2424fd9bd7f14e508fd106cbfda786a4 P 65c36d164b0540f099d592f38307ee6c [term 7 LEADER]: Becoming Leader. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:46:59.856812 27491 consensus_queue.cc:260] T 0c29937a80724ddb9d07a27534a146f3 P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.2, Last appended by leader: 2, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:46:59.919902 23285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a88421dba80d4d3388aa5eb62e23853b" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:46:59.906566 23404 rpcz_store.cc:276] 0809 15:46:58.639037 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:58.639146 (+ 109us) service_pool.cc:227] Handling call
0809 15:46:59.035618 (+396472us) spinlock_profiling.cc:243] Waited 17.7 ms on lock 0x7b58001d7c98. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e613cf11 00007f34e62c0851 00007f34e62c0702 00007f34ef1b0b09 00007f34ee0ec730 00007f34ee1b279b 00007f34ee1b2821 00007f34ee1762b0 00007f34ee1617f9 00007f34ee1612a2 00007f34e7847684
0809 15:46:59.036207 (+ 589us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:59.065379 (+ 29172us) raft_consensus.cc:1520] Early marking committed up to index 4
0809 15:46:59.102805 (+ 37426us) log.cc:842] Serialized 14 byte log entry
0809 15:46:59.902592 (+799787us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 15:46:59.902638 (+ 46us) raft_consensus.cc:1631] Marking committed up to 4
0809 15:46:59.902678 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:59.902733 (+ 55us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:59.902829 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6435,"prepare.queue_time_us":35611,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":54,"raft.queue_time_us":20920,"raft.run_cpu_time_us":109,"raft.run_wall_time_us":16384,"spinlock_wait_cycles":54120576,"thread_start_us":695958,"threads_started":1,"wal-append.queue_time_us":801404}
I20260809 15:46:59.961727 27217 raft_consensus.cc:2726] T 3296ef50be254fc29cfc77d22a8f7a23 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 15:46:59.996826 23283 raft_consensus.cc:2454] T 763e58d6ba644eea830a6c1664d39c9b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 5.
I20260809 15:46:59.995342 23228 leader_election.cc:302] T 30c1740ef6874df0a332fbd6fd4b03ce P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:46:59.982947 23393 rpcz_store.cc:276] 0809 15:46:58.588635 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:58.588759 (+ 124us) service_pool.cc:227] Handling call
0809 15:46:58.592241 (+ 3482us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:46:58.592634 (+ 393us) raft_consensus.cc:1520] Early marking committed up to index 3
0809 15:46:59.508190 (+915556us) log.cc:842] Serialized 14 byte log entry
0809 15:46:59.942020 (+433830us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 15:46:59.942070 (+ 50us) raft_consensus.cc:1631] Marking committed up to 3
0809 15:46:59.942112 (+ 42us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:59.942171 (+ 59us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:46:59.942267 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":35769,"prepare.queue_time_us":195344,"prepare.run_cpu_time_us":63,"prepare.run_wall_time_us":5248,"raft.queue_time_us":45140,"raft.run_cpu_time_us":105,"raft.run_wall_time_us":5041,"thread_start_us":1160896,"threads_started":2}
I20260809 15:46:59.986934 27582 raft_consensus.cc:482] T 2424fd9bd7f14e508fd106cbfda786a4 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:47:00.006786 23285 raft_consensus.cc:2454] T a88421dba80d4d3388aa5eb62e23853b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 4.
I20260809 15:46:59.994184 26619 raft_consensus.cc:482] T 763e58d6ba644eea830a6c1664d39c9b P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260809 15:46:59.978127 27531 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.656s user 0.013s sys 0.031s
W20260809 15:47:00.000499 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 91733.6us
I20260809 15:46:59.995949 27394 raft_consensus.cc:482] T 953eea268e17491ca76f3a555860a512 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:00.036891 23351 leader_election.cc:302] T cef3e7277d7d4760824a3436ed4a847d P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c
I20260809 15:47:00.010489 23276 raft_consensus.cc:1261] T cee02fff2c9c48a19f43eb358a89e7b0 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
W20260809 15:47:00.038578 23468 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.68:34689, user_credentials={real_user=slave}} blocked reactor thread for 118458us
I20260809 15:47:00.040539 27081 consensus_queue.cc:237] T 2424fd9bd7f14e508fd106cbfda786a4 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.2, Last appended by leader: 2, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:00.003528 27468 raft_consensus.cc:686] T 00a1881267a14ba8b78c81040f17a014 P 65c36d164b0540f099d592f38307ee6c [term 8 LEADER]: Becoming Leader. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:46:59.988921 27316 raft_consensus.cc:504] T 998c6cee72cf40228c6dba064c0db2da P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:00.044373 27394 raft_consensus.cc:504] T 953eea268e17491ca76f3a555860a512 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:00.045560 27217 raft_consensus.cc:482] T cee02fff2c9c48a19f43eb358a89e7b0 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:00.046116 27217 raft_consensus.cc:504] T cee02fff2c9c48a19f43eb358a89e7b0 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:00.046496 27316 leader_election.cc:288] T 998c6cee72cf40228c6dba064c0db2da P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
W20260809 15:47:00.003136 27589 consensus_queue.cc:1159] T 0c29937a80724ddb9d07a27534a146f3 P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "c0f38c1908414566aa9115bb2cbdd85d" responder_term: 6 status { last_received { term: 5 index: 2 } last_committed_idx: 2 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 5. Current term is 6. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260809 15:47:00.005143 27491 raft_consensus.cc:3037] T 0c29937a80724ddb9d07a27534a146f3 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Advancing to term 6
I20260809 15:47:00.036188 26619 raft_consensus.cc:3037] T 763e58d6ba644eea830a6c1664d39c9b P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Advancing to term 7
I20260809 15:47:00.049252 23468 leader_election.cc:302] T d3e94b7e50754a81b3d72a07476cb7b9 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c; no voters: c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:00.046408 27394 leader_election.cc:288] T 953eea268e17491ca76f3a555860a512 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:46:59.998250 26966 raft_consensus.cc:482] T 07fcabb569414bddbd39df6cedd24e64 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:00.036561 24468 raft_consensus.cc:1226] T d3be1e356b254d22b995be32f1c12f6f P 61e69695258f48baa984c9d395e8ce3f [term 7 FOLLOWER]: Rejecting Update request from peer c0f38c1908414566aa9115bb2cbdd85d for earlier term 6. Current term is 7. Ops: []
I20260809 15:47:00.054188 27267 raft_consensus.cc:482] T df20671a7b874dbb98c0ef28371c1308 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:00.054661 27267 raft_consensus.cc:504] T df20671a7b874dbb98c0ef28371c1308 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:00.056517 27267 leader_election.cc:288] T df20671a7b874dbb98c0ef28371c1308 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:47:00.053323 27487 raft_consensus.cc:2781] T cef3e7277d7d4760824a3436ed4a847d P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260809 15:47:00.057443 27487 raft_consensus.cc:482] T cef3e7277d7d4760824a3436ed4a847d P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Starting leader election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:47:00.057772 27487 raft_consensus.cc:3037] T cef3e7277d7d4760824a3436ed4a847d P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Advancing to term 7
I20260809 15:47:00.053488 27488 consensus_queue.cc:1035] T c5c2e425a74b4e64ae70b5def750434c P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.006s
I20260809 15:47:00.059374 27507 raft_consensus.cc:504] T a3622f6e3e3641cea1960e3a31498bb7 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:00.060634 23280 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55434 (request call id 241) took 1018 ms. Trace:
W20260809 15:47:00.053557 27363 consensus_queue.cc:1159] T 0c29937a80724ddb9d07a27534a146f3 P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" responder_term: 5 status { last_received { term: 5 index: 2 } last_committed_idx: 2 last_received_current_leader { term: 5 index: 2 } }
I20260809 15:47:00.049288 27497 raft_consensus.cc:2726] T 0c29937a80724ddb9d07a27534a146f3 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260809 15:47:00.060964 23280 rpcz_store.cc:276] 0809 15:46:59.042191 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:59.054753 (+ 12562us) service_pool.cc:227] Handling call
0809 15:47:00.060471 (+1005718us) spinlock_profiling.cc:243] Waited 884 ms on lock 0x7b54000bfc54. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e92242ae 00007f34ee161c27 00007f34e7849c04 00007f34e7849b92 00007f34e7849ac1 00007f34e7849a3c 00007f34e7848cf2 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:47:00.060570 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2032421632}
I20260809 15:47:00.079545 27488 consensus_queue.cc:1046] T d3be1e356b254d22b995be32f1c12f6f P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260809 15:47:00.080363 27488 raft_consensus.cc:482] T 28e17dcb5efe45b8b6a8c42da03b4819 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:00.080773 27488 raft_consensus.cc:504] T 28e17dcb5efe45b8b6a8c42da03b4819 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:00.082612 27488 leader_election.cc:288] T 28e17dcb5efe45b8b6a8c42da03b4819 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:00.083043 27488 raft_consensus.cc:3032] T d3be1e356b254d22b995be32f1c12f6f P c0f38c1908414566aa9115bb2cbdd85d [term 6 LEADER]: Stepping down as leader of term 6
I20260809 15:47:00.062891 23397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "953eea268e17491ca76f3a555860a512" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:00.003649 27582 raft_consensus.cc:3037] T 2424fd9bd7f14e508fd106cbfda786a4 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Advancing to term 7
I20260809 15:47:00.073983 23351 leader_election.cc:302] T ae6d2c08a7a94f58ac8dd698e3fd456c P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:47:00.074862 23396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3622f6e3e3641cea1960e3a31498bb7" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:47:00.087419 23396 raft_consensus.cc:3037] T a3622f6e3e3641cea1960e3a31498bb7 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Advancing to term 5
I20260809 15:47:00.075641 23393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df20671a7b874dbb98c0ef28371c1308" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:00.088105 23393 raft_consensus.cc:2454] T df20671a7b874dbb98c0ef28371c1308 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 5.
I20260809 15:47:00.090512 23394 raft_consensus.cc:2454] T d2d4f749fcfd4fb3bfb8115af3471598 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 6.
I20260809 15:47:00.061266 27507 leader_election.cc:288] T a3622f6e3e3641cea1960e3a31498bb7 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 5 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:47:00.075060 27497 consensus_queue.cc:1035] T 953eea268e17491ca76f3a555860a512 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.016s
I20260809 15:47:00.095656 27659 consensus_queue.cc:1035] T e71a05ba1d4d42bd95c3f85f9313399e P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20260809 15:47:00.097466 27659 raft_consensus.cc:2726] T 30c1740ef6874df0a332fbd6fd4b03ce P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260809 15:47:00.098362 27633 raft_consensus.cc:2781] T d3e94b7e50754a81b3d72a07476cb7b9 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Leader election won for term 7
I20260809 15:47:00.098933 27633 raft_consensus.cc:686] T d3e94b7e50754a81b3d72a07476cb7b9 P 65c36d164b0540f099d592f38307ee6c [term 7 LEADER]: Becoming Leader. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:47:00.099888 27633 consensus_queue.cc:237] T d3e94b7e50754a81b3d72a07476cb7b9 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:00.076764 27639 raft_consensus.cc:2781] T 3d880de657e544468c82157769622a26 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Leader election won for term 3
I20260809 15:47:00.101478 27639 raft_consensus.cc:686] T 3d880de657e544468c82157769622a26 P 65c36d164b0540f099d592f38307ee6c [term 3 LEADER]: Becoming Leader. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:47:00.101684 27217 leader_election.cc:288] T cee02fff2c9c48a19f43eb358a89e7b0 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:00.069698 27395 raft_consensus.cc:3032] T 00a1881267a14ba8b78c81040f17a014 P c0f38c1908414566aa9115bb2cbdd85d [term 7 LEADER]: Stepping down as leader of term 7
I20260809 15:47:00.102224 27395 raft_consensus.cc:729] T 00a1881267a14ba8b78c81040f17a014 P c0f38c1908414566aa9115bb2cbdd85d [term 7 LEADER]: Becoming Follower/Learner. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:47:00.102758 23285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df20671a7b874dbb98c0ef28371c1308" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:00.101044 27322 consensus_queue.cc:1035] T cee02fff2c9c48a19f43eb358a89e7b0 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.003s
I20260809 15:47:00.103057 27395 consensus_queue.cc:260] T 00a1881267a14ba8b78c81040f17a014 P c0f38c1908414566aa9115bb2cbdd85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:00.104411 27395 raft_consensus.cc:3037] T 00a1881267a14ba8b78c81040f17a014 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Advancing to term 8
I20260809 15:47:00.105448 23280 raft_consensus.cc:1226] T 30c1740ef6874df0a332fbd6fd4b03ce P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 5. Current term is 6. Ops: []
I20260809 15:47:00.089442 23472 leader_election.cc:302] T df20671a7b874dbb98c0ef28371c1308 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:47:00.108551 23285 raft_consensus.cc:2454] T df20671a7b874dbb98c0ef28371c1308 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 5.
I20260809 15:47:00.051167 23400 consensus_queue.cc:237] T 4b30f28311964a0b9149a50495e83d8f P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } attrs { replace: true } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } }
I20260809 15:47:00.110248 27497 raft_consensus.cc:2781] T ae6d2c08a7a94f58ac8dd698e3fd456c P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Leader election won for term 7
W20260809 15:47:00.070886 27634 log.cc:924] Time spent T 4564dfc6d4094c1b9240b367ccecf3b1 P 65c36d164b0540f099d592f38307ee6c: Append to log took a long time: real 0.109s user 0.002s sys 0.000s
I20260809 15:47:00.110026 23283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3622f6e3e3641cea1960e3a31498bb7" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:47:00.113197 23277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998c6cee72cf40228c6dba064c0db2da" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:00.117331 23276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "953eea268e17491ca76f3a555860a512" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:00.117930 23283 raft_consensus.cc:3037] T a3622f6e3e3641cea1960e3a31498bb7 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Advancing to term 5
I20260809 15:47:00.118060 23276 raft_consensus.cc:2454] T 953eea268e17491ca76f3a555860a512 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 6.
W20260809 15:47:00.119678 23223 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.68:34689, user_credentials={real_user=slave}} blocked reactor thread for 420653us
I20260809 15:47:00.086489 27488 raft_consensus.cc:729] T d3be1e356b254d22b995be32f1c12f6f P c0f38c1908414566aa9115bb2cbdd85d [term 6 LEADER]: Becoming Follower/Learner. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:47:00.067014 27583 raft_consensus.cc:482] T 07fcabb569414bddbd39df6cedd24e64 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:00.145819 23277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e17dcb5efe45b8b6a8c42da03b4819" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
W20260809 15:47:00.147395 23468 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.68:34689, user_credentials={real_user=slave}} blocked reactor thread for 98215.7us
I20260809 15:47:00.154379 23277 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7baa2492df0340c4a5ac241959ce1ac3"
dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
mode: GRACEFUL
new_leader_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
from {username='slave'} at 127.0.0.1:33468
I20260809 15:47:00.154927 23277 raft_consensus.cc:595] T 7baa2492df0340c4a5ac241959ce1ac3 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Received request to transfer leadership to TS c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:00.155226 23277 raft_consensus.cc:603] T 7baa2492df0340c4a5ac241959ce1ac3 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260809 15:47:00.152237 23353 leader_election.cc:302] T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c
I20260809 15:47:00.114113 27428 raft_consensus.cc:482] T 6ced718c1a5c4da79cb8c1ebd1435b87 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:47:00.129956 27497 raft_consensus.cc:686] T ae6d2c08a7a94f58ac8dd698e3fd456c P c0f38c1908414566aa9115bb2cbdd85d [term 7 LEADER]: Becoming Leader. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:47:00.175486 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee02fff2c9c48a19f43eb358a89e7b0" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:47:00.176134 24468 raft_consensus.cc:2454] T cee02fff2c9c48a19f43eb358a89e7b0 P 61e69695258f48baa984c9d395e8ce3f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 6.
I20260809 15:47:00.175918 27497 consensus_queue.cc:237] T ae6d2c08a7a94f58ac8dd698e3fd456c P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:00.059449 26966 raft_consensus.cc:504] T 07fcabb569414bddbd39df6cedd24e64 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:00.134143 27404 raft_consensus.cc:2781] T df20671a7b874dbb98c0ef28371c1308 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260809 15:47:00.183837 23470 leader_election.cc:302] T 953eea268e17491ca76f3a555860a512 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters: c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:00.112332 27294 raft_consensus.cc:504] T a88421dba80d4d3388aa5eb62e23853b P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:00.187958 26730 raft_consensus.cc:482] T 05ad7d8fdca64e3da527280ca5be9280 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:00.201086 27588 raft_consensus.cc:482] T d1a134ba33ea4076bbaac0ae460c17ba P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:47:00.202436 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae6d2c08a7a94f58ac8dd698e3fd456c" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:47:00.203004 27395 raft_consensus.cc:2726] T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260809 15:47:00.203018 23522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998c6cee72cf40228c6dba064c0db2da" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:00.204432 23518 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "df20671a7b874dbb98c0ef28371c1308"
dest_uuid: "65c36d164b0540f099d592f38307ee6c"
mode: GRACEFUL
new_leader_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
from {username='slave'} at 127.0.0.1:48888
I20260809 15:47:00.220358 23526 raft_consensus.cc:3037] T ae6d2c08a7a94f58ac8dd698e3fd456c P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Advancing to term 7
I20260809 15:47:00.222112 23520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad8a3b25d7b49e5a2142d5709edd0c7" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:00.144248 27347 consensus_queue.cc:1046] T 6ae14a74a26f434db08c652e097385c8 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.067s
I20260809 15:47:00.226545 23277 raft_consensus.cc:1261] T d2d4f749fcfd4fb3bfb8115af3471598 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260809 15:47:00.218956 26730 raft_consensus.cc:504] T 05ad7d8fdca64e3da527280ca5be9280 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:00.253093 26730 leader_election.cc:288] T 05ad7d8fdca64e3da527280ca5be9280 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:00.254223 27518 raft_consensus.cc:3032] T 6ae14a74a26f434db08c652e097385c8 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 LEADER]: Stepping down as leader of term 5
I20260809 15:47:00.254599 27518 raft_consensus.cc:729] T 6ae14a74a26f434db08c652e097385c8 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Running, Role: LEADER
I20260809 15:47:00.255434 27518 consensus_queue.cc:260] T 6ae14a74a26f434db08c652e097385c8 P 67b3aeadfa7a432abe8af4d292e04ab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:00.256834 27518 raft_consensus.cc:3037] T 6ae14a74a26f434db08c652e097385c8 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Advancing to term 6
I20260809 15:47:00.240671 26966 leader_election.cc:288] T 07fcabb569414bddbd39df6cedd24e64 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:47:00.178874 27263 raft_consensus.cc:482] T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
I20260809 15:47:00.221114 27363 raft_consensus.cc:482] T ba56725e42f74a1baa4c04d9660727f9 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:00.247301 27492 consensus_queue.cc:1046] T 30c1740ef6874df0a332fbd6fd4b03ce P 65c36d164b0540f099d592f38307ee6c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.103s
W20260809 15:47:00.248185 27645 log.cc:924] Time spent T 0acc47ce21c3422698d1ef7277c3ed63 P c0f38c1908414566aa9115bb2cbdd85d: Append to log took a long time: real 0.249s user 0.000s sys 0.000s
I20260809 15:47:00.111618 27468 consensus_queue.cc:237] T 00a1881267a14ba8b78c81040f17a014 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:00.248946 23528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e17dcb5efe45b8b6a8c42da03b4819" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:00.249542 27659 raft_consensus.cc:482] T ba56725e42f74a1baa4c04d9660727f9 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:00.195225 27404 raft_consensus.cc:482] T df20671a7b874dbb98c0ef28371c1308 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260809 15:47:00.219875 27648 log.cc:924] Time spent T 73f6f44c259547f1b05a3a434616fa54 P 67b3aeadfa7a432abe8af4d292e04ab9: Append to log took a long time: real 0.260s user 0.001s sys 0.000s
I20260809 15:47:00.290705 23276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07fcabb569414bddbd39df6cedd24e64" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:00.153880 27583 raft_consensus.cc:504] T 07fcabb569414bddbd39df6cedd24e64 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:00.222617 27268 raft_consensus.cc:2781] T 953eea268e17491ca76f3a555860a512 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260809 15:47:00.326414 23394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07fcabb569414bddbd39df6cedd24e64" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:00.358135 23394 raft_consensus.cc:2454] T 07fcabb569414bddbd39df6cedd24e64 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 6.
I20260809 15:47:00.347606 23397 raft_consensus.cc:1261] T d2d4f749fcfd4fb3bfb8115af3471598 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260809 15:47:00.351444 23528 raft_consensus.cc:2454] T 28e17dcb5efe45b8b6a8c42da03b4819 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 7.
I20260809 15:47:00.221074 27588 raft_consensus.cc:504] T d1a134ba33ea4076bbaac0ae460c17ba P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:00.383404 23276 raft_consensus.cc:2454] T 07fcabb569414bddbd39df6cedd24e64 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 6.
I20260809 15:47:00.349408 27404 raft_consensus.cc:3037] T df20671a7b874dbb98c0ef28371c1308 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Advancing to term 6
W20260809 15:47:00.409255 23470 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 45107.2us
W20260809 15:47:00.405966 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 410694us
I20260809 15:47:00.408686 23397 raft_consensus.cc:1226] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d [term 9 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 8. Current term is 9. Ops: []
I20260809 15:47:00.416159 23228 leader_election.cc:302] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:00.361732 27268 raft_consensus.cc:482] T 953eea268e17491ca76f3a555860a512 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:47:00.320125 27263 raft_consensus.cc:504] T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:00.324820 27363 raft_consensus.cc:504] T ba56725e42f74a1baa4c04d9660727f9 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:00.190114 27428 raft_consensus.cc:504] T 6ced718c1a5c4da79cb8c1ebd1435b87 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:00.382539 27519 raft_consensus.cc:482] T 213543cdbe814c89a17a963469922c1b P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:47:00.427815 23472 leader_election.cc:302] T 07fcabb569414bddbd39df6cedd24e64 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:47:00.430639 23228 leader_election.cc:302] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:00.348394 27659 raft_consensus.cc:504] T ba56725e42f74a1baa4c04d9660727f9 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:00.424431 27267 raft_consensus.cc:482] T d1a134ba33ea4076bbaac0ae460c17ba P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 15:47:00.438792 23353 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 286623us
I20260809 15:47:00.438978 23286 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53554 (request call id 1601) took 1069 ms. Trace:
I20260809 15:47:00.432973 23277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "907bacfa79d04a55a68c8dbe60056462" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
W20260809 15:47:00.431452 24406 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 309562us
I20260809 15:47:00.434398 23270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3be1e356b254d22b995be32f1c12f6f" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:47:00.441150 23223 leader_election.cc:302] T cee02fff2c9c48a19f43eb358a89e7b0 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:47:00.324370 27488 consensus_queue.cc:260] T d3be1e356b254d22b995be32f1c12f6f P c0f38c1908414566aa9115bb2cbdd85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:00.421165 27268 raft_consensus.cc:3037] T 953eea268e17491ca76f3a555860a512 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Advancing to term 7
I20260809 15:47:00.444662 24406 leader_election.cc:302] T 0c48a3049897452383257aa757834fb5 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c
I20260809 15:47:00.325500 27639 consensus_queue.cc:237] T 3d880de657e544468c82157769622a26 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:00.439400 23286 rpcz_store.cc:276] 0809 15:46:59.369173 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:59.370377 (+ 1204us) service_pool.cc:227] Handling call
0809 15:46:59.370835 (+ 458us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 15:46:59.371327 (+ 492us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:46:59.371375 (+ 48us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 15:46:59.372110 (+ 735us) log.cc:842] Serialized 26 byte log entry
0809 15:46:59.581461 (+209351us) raft_consensus.cc:1631] Marking committed up to 3
0809 15:46:59.581497 (+ 36us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:46:59.582032 (+ 535us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 15:47:00.438796 (+856764us) raft_consensus.cc:1671] finished
0809 15:47:00.438813 (+ 17us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:47:00.438927 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":3001,"raft.run_cpu_time_us":82,"raft.run_wall_time_us":83,"thread_start_us":208889,"threads_started":1,"wal-append.queue_time_us":521893}
W20260809 15:47:00.430877 27480 consensus_meta.cc:287] T 4bcac4b24e7a4448abdc934bba5a624e P 61e69695258f48baa984c9d395e8ce3f: Time spent flushing consensus metadata: real 0.514s user 0.012s sys 0.002s
I20260809 15:47:00.447562 23353 leader_election.cc:302] T 4564dfc6d4094c1b9240b367ccecf3b1 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:47:00.453128 23394 raft_consensus.cc:1261] T 05ad7d8fdca64e3da527280ca5be9280 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Refusing update from remote peer 67b3aeadfa7a432abe8af4d292e04ab9: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260809 15:47:00.453171 23395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df20671a7b874dbb98c0ef28371c1308" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:00.418538 26611 consensus_queue.cc:1035] T d2d4f749fcfd4fb3bfb8115af3471598 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.020s
I20260809 15:47:00.453990 23395 raft_consensus.cc:2454] T df20671a7b874dbb98c0ef28371c1308 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 5.
I20260809 15:47:00.451792 23399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b30f28311964a0b9149a50495e83d8f" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:00.454273 23270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a88421dba80d4d3388aa5eb62e23853b" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:47:00.457211 23270 raft_consensus.cc:3037] T a88421dba80d4d3388aa5eb62e23853b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Advancing to term 5
I20260809 15:47:00.457636 23286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05ad7d8fdca64e3da527280ca5be9280" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:00.457218 23397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cef3e7277d7d4760824a3436ed4a847d" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:00.460109 23228 leader_election.cc:302] T df20671a7b874dbb98c0ef28371c1308 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:47:00.431916 27669 raft_consensus.cc:2726] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260809 15:47:00.431463 27658 raft_consensus.cc:2726] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260809 15:47:00.468583 23394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e17dcb5efe45b8b6a8c42da03b4819" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:00.469147 23394 raft_consensus.cc:2454] T 28e17dcb5efe45b8b6a8c42da03b4819 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 7.
I20260809 15:47:00.471607 23228 leader_election.cc:302] T 28e17dcb5efe45b8b6a8c42da03b4819 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:47:00.472443 27378 raft_consensus.cc:2781] T 28e17dcb5efe45b8b6a8c42da03b4819 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260809 15:47:00.472914 27378 raft_consensus.cc:482] T 28e17dcb5efe45b8b6a8c42da03b4819 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Starting leader election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:47:00.473398 27378 raft_consensus.cc:3037] T 28e17dcb5efe45b8b6a8c42da03b4819 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Advancing to term 7
I20260809 15:47:00.478991 23393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07fcabb569414bddbd39df6cedd24e64" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:00.479586 23393 raft_consensus.cc:2454] T 07fcabb569414bddbd39df6cedd24e64 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 6.
I20260809 15:47:00.480667 27531 consensus_queue.cc:1035] T d2d4f749fcfd4fb3bfb8115af3471598 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260809 15:47:00.481655 27263 leader_election.cc:288] T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:00.482481 27081 raft_consensus.cc:482] T c5c2e425a74b4e64ae70b5def750434c P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:00.482963 27081 raft_consensus.cc:504] T c5c2e425a74b4e64ae70b5def750434c P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:00.484830 27081 leader_election.cc:288] T c5c2e425a74b4e64ae70b5def750434c P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:47:00.482698 23520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cef3e7277d7d4760824a3436ed4a847d" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:00.487493 23277 raft_consensus.cc:2362] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 6. Re-sending same reply.
I20260809 15:47:00.486227 23228 leader_election.cc:302] T 07fcabb569414bddbd39df6cedd24e64 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:47:00.461010 27294 leader_election.cc:288] T a88421dba80d4d3388aa5eb62e23853b P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 5 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:47:00.499142 23526 raft_consensus.cc:2454] T ae6d2c08a7a94f58ac8dd698e3fd456c P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 7.
I20260809 15:47:00.500607 23270 raft_consensus.cc:2454] T a88421dba80d4d3388aa5eb62e23853b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 5.
I20260809 15:47:00.503031 23470 leader_election.cc:302] T a88421dba80d4d3388aa5eb62e23853b P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:47:00.503425 27661 raft_consensus.cc:2781] T cee02fff2c9c48a19f43eb358a89e7b0 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260809 15:47:00.503965 27633 raft_consensus.cc:2781] T a88421dba80d4d3388aa5eb62e23853b P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Leader election won for term 5
I20260809 15:47:00.504520 27633 raft_consensus.cc:686] T a88421dba80d4d3388aa5eb62e23853b P 65c36d164b0540f099d592f38307ee6c [term 5 LEADER]: Becoming Leader. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:47:00.505564 27633 consensus_queue.cc:237] T a88421dba80d4d3388aa5eb62e23853b P 65c36d164b0540f099d592f38307ee6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:00.468478 26399 raft_consensus.cc:2781] T 07fcabb569414bddbd39df6cedd24e64 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260809 15:47:00.449422 27267 raft_consensus.cc:504] T d1a134ba33ea4076bbaac0ae460c17ba P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:00.435084 27519 raft_consensus.cc:504] T 213543cdbe814c89a17a963469922c1b P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:00.466841 27480 raft_consensus.cc:2781] T 0c48a3049897452383257aa757834fb5 P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Leader election won for term 4
I20260809 15:47:00.468340 27217 raft_consensus.cc:2781] T df20671a7b874dbb98c0ef28371c1308 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260809 15:47:00.509754 23528 raft_consensus.cc:1261] T 05ad7d8fdca64e3da527280ca5be9280 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Refusing update from remote peer 67b3aeadfa7a432abe8af4d292e04ab9: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260809 15:47:00.509835 23396 raft_consensus.cc:2454] T a3622f6e3e3641cea1960e3a31498bb7 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 5.
W20260809 15:47:00.511046 23353 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 54594.4us
W20260809 15:47:00.503822 27666 log.cc:924] Time spent T 73f6f44c259547f1b05a3a434616fa54 P c0f38c1908414566aa9115bb2cbdd85d: Append to log took a long time: real 0.059s user 0.000s sys 0.000s
I20260809 15:47:00.521068 23520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df20671a7b874dbb98c0ef28371c1308" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:00.523758 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b30f28311964a0b9149a50495e83d8f" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:00.524731 23286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5c2e425a74b4e64ae70b5def750434c" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
W20260809 15:47:00.527567 24406 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 82979.5us
I20260809 15:47:00.503894 27661 raft_consensus.cc:482] T cee02fff2c9c48a19f43eb358a89e7b0 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:47:00.511500 27658 raft_consensus.cc:2781] T 07fcabb569414bddbd39df6cedd24e64 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260809 15:47:00.533727 23286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1a134ba33ea4076bbaac0ae460c17ba" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:00.532482 24463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3e94b7e50754a81b3d72a07476cb7b9" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:47:00.540103 24463 raft_consensus.cc:2379] T d3e94b7e50754a81b3d72a07476cb7b9 P 61e69695258f48baa984c9d395e8ce3f [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d in current term 7: Already voted for candidate 65c36d164b0540f099d592f38307ee6c in this term.
I20260809 15:47:00.537750 27658 raft_consensus.cc:482] T 07fcabb569414bddbd39df6cedd24e64 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:47:00.539412 27589 raft_consensus.cc:3032] T 30c1740ef6874df0a332fbd6fd4b03ce P 65c36d164b0540f099d592f38307ee6c [term 5 LEADER]: Stepping down as leader of term 5
I20260809 15:47:00.513088 26399 raft_consensus.cc:482] T 07fcabb569414bddbd39df6cedd24e64 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:47:00.513314 27294 raft_consensus.cc:482] T 2db057fd10e149a9895dff6ef8c9f038 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:00.498391 27677 raft_consensus.cc:2726] T 4564dfc6d4094c1b9240b367ccecf3b1 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260809 15:47:00.544457 23286 raft_consensus.cc:2379] T d1a134ba33ea4076bbaac0ae460c17ba P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d in current term 5: Already voted for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in this term.
I20260809 15:47:00.545490 23353 leader_election.cc:302] T 998c6cee72cf40228c6dba064c0db2da P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:47:00.476661 27660 consensus_queue.cc:1035] T 05ad7d8fdca64e3da527280ca5be9280 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.068s
I20260809 15:47:00.530933 27217 raft_consensus.cc:482] T df20671a7b874dbb98c0ef28371c1308 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:47:00.567651 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e17dcb5efe45b8b6a8c42da03b4819" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:00.569422 23526 raft_consensus.cc:2379] T 28e17dcb5efe45b8b6a8c42da03b4819 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 in current term 7: Already voted for candidate 65c36d164b0540f099d592f38307ee6c in this term.
W20260809 15:47:00.565065 23472 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 85564.7us
I20260809 15:47:00.533389 27661 raft_consensus.cc:3037] T cee02fff2c9c48a19f43eb358a89e7b0 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Advancing to term 7
I20260809 15:47:00.573143 23285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ced718c1a5c4da79cb8c1ebd1435b87" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:00.564029 27294 raft_consensus.cc:504] T 2db057fd10e149a9895dff6ef8c9f038 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:00.580313 27267 leader_election.cc:288] T d1a134ba33ea4076bbaac0ae460c17ba P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:47:00.581287 27294 leader_election.cc:288] T 2db057fd10e149a9895dff6ef8c9f038 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:47:00.583611 27556 consensus_queue.cc:1035] T 563daf42d7784db195ea99c3cb21b370 P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260809 15:47:00.590334 27487 consensus_meta.cc:287] T cef3e7277d7d4760824a3436ed4a847d P c0f38c1908414566aa9115bb2cbdd85d: Time spent flushing consensus metadata: real 0.532s user 0.007s sys 0.005s
I20260809 15:47:00.591953 23528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee02fff2c9c48a19f43eb358a89e7b0" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:00.541052 27583 leader_election.cc:288] T 07fcabb569414bddbd39df6cedd24e64 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:00.600128 23277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1a134ba33ea4076bbaac0ae460c17ba" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:00.599948 23286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba56725e42f74a1baa4c04d9660727f9" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
W20260809 15:47:00.612963 23353 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 67557.6us
I20260809 15:47:00.611202 23286 raft_consensus.cc:2454] T ba56725e42f74a1baa4c04d9660727f9 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 4.
W20260809 15:47:00.579172 27671 log.cc:924] Time spent T d3e94b7e50754a81b3d72a07476cb7b9 P 65c36d164b0540f099d592f38307ee6c: Append to log took a long time: real 0.067s user 0.000s sys 0.001s
I20260809 15:47:00.553593 27658 raft_consensus.cc:3037] T 07fcabb569414bddbd39df6cedd24e64 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Advancing to term 7
I20260809 15:47:00.553370 27589 raft_consensus.cc:729] T 30c1740ef6874df0a332fbd6fd4b03ce P 65c36d164b0540f099d592f38307ee6c [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:47:00.564348 26399 raft_consensus.cc:3037] T 07fcabb569414bddbd39df6cedd24e64 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Advancing to term 7
I20260809 15:47:00.614164 23470 leader_election.cc:302] T ba56725e42f74a1baa4c04d9660727f9 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
W20260809 15:47:00.615703 23472 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 45105.6us
I20260809 15:47:00.611068 23277 raft_consensus.cc:2454] T d1a134ba33ea4076bbaac0ae460c17ba P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 5.
I20260809 15:47:00.568044 27217 raft_consensus.cc:3037] T df20671a7b874dbb98c0ef28371c1308 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Advancing to term 6
I20260809 15:47:00.627991 23277 raft_consensus.cc:1226] T 30c1740ef6874df0a332fbd6fd4b03ce P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 5. Current term is 6. Ops: []
I20260809 15:47:00.564678 27588 leader_election.cc:288] T d1a134ba33ea4076bbaac0ae460c17ba P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:00.591148 27487 raft_consensus.cc:504] T cef3e7277d7d4760824a3436ed4a847d P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:00.626421 23400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba56725e42f74a1baa4c04d9660727f9" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:00.629559 23353 leader_election.cc:302] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c
I20260809 15:47:00.653692 23472 leader_election.cc:302] T a3622f6e3e3641cea1960e3a31498bb7 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:47:00.654028 23228 leader_election.cc:302] T cef3e7277d7d4760824a3436ed4a847d P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
W20260809 15:47:00.655709 24406 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 122864us
I20260809 15:47:00.656100 23228 leader_election.cc:302] T 4b30f28311964a0b9149a50495e83d8f P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
W20260809 15:47:00.656654 23283 consensus_meta.cc:287] T a3622f6e3e3641cea1960e3a31498bb7 P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent flushing consensus metadata: real 0.527s user 0.008s sys 0.000s
I20260809 15:47:00.656967 23283 raft_consensus.cc:2454] T a3622f6e3e3641cea1960e3a31498bb7 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 5.
I20260809 15:47:00.657845 23527 raft_consensus.cc:1226] T 00a1881267a14ba8b78c81040f17a014 P 65c36d164b0540f099d592f38307ee6c [term 8 LEADER]: Rejecting Update request from peer c0f38c1908414566aa9115bb2cbdd85d for earlier term 7. Current term is 8. Ops: []
I20260809 15:47:00.649219 23286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2db057fd10e149a9895dff6ef8c9f038" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:00.658725 23286 raft_consensus.cc:2454] T 2db057fd10e149a9895dff6ef8c9f038 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 4.
I20260809 15:47:00.608939 27363 leader_election.cc:288] T ba56725e42f74a1baa4c04d9660727f9 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:47:00.661573 23283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "213543cdbe814c89a17a963469922c1b" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:00.614598 27428 leader_election.cc:288] T 6ced718c1a5c4da79cb8c1ebd1435b87 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:00.610811 27480 raft_consensus.cc:686] T 0c48a3049897452383257aa757834fb5 P 61e69695258f48baa984c9d395e8ce3f [term 4 LEADER]: Becoming Leader. State: Replica: 61e69695258f48baa984c9d395e8ce3f, State: Running, Role: LEADER
W20260809 15:47:00.619683 26619 consensus_meta.cc:287] T 763e58d6ba644eea830a6c1664d39c9b P 65c36d164b0540f099d592f38307ee6c: Time spent flushing consensus metadata: real 0.571s user 0.010s sys 0.003s
I20260809 15:47:00.663781 23528 raft_consensus.cc:1261] T e14d118b2eaa42aa8a7cd4025aba7128 P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Refusing update from remote peer 67b3aeadfa7a432abe8af4d292e04ab9: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260809 15:47:00.631876 23470 leader_election.cc:302] T d1a134ba33ea4076bbaac0ae460c17ba P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:47:00.679752 23404 raft_consensus.cc:1226] T cef3e7277d7d4760824a3436ed4a847d P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 6. Current term is 7. Ops: []
I20260809 15:47:00.663882 23400 raft_consensus.cc:2454] T ba56725e42f74a1baa4c04d9660727f9 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 4.
I20260809 15:47:00.662607 27322 raft_consensus.cc:2781] T a3622f6e3e3641cea1960e3a31498bb7 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Leader election won for term 5
I20260809 15:47:00.681747 24406 leader_election.cc:302] T d3be1e356b254d22b995be32f1c12f6f P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f; no voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d
W20260809 15:47:00.685457 23470 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 53638.4us
I20260809 15:47:00.621057 27395 raft_consensus.cc:482] T 0a51c0baaa6e432997c9d3552af4f254 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
I20260809 15:47:00.687162 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07fcabb569414bddbd39df6cedd24e64" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:00.688234 23522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba56725e42f74a1baa4c04d9660727f9" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:00.630548 27518 raft_consensus.cc:482] T 4564dfc6d4094c1b9240b367ccecf3b1 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:47:00.621186 27682 raft_consensus.cc:2726] T 998c6cee72cf40228c6dba064c0db2da P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260809 15:47:00.665592 27659 leader_election.cc:288] T ba56725e42f74a1baa4c04d9660727f9 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:00.686380 26619 raft_consensus.cc:504] T 763e58d6ba644eea830a6c1664d39c9b P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:00.694600 27589 consensus_queue.cc:260] T 30c1740ef6874df0a332fbd6fd4b03ce P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:00.695477 27053 raft_consensus.cc:2781] T d1a134ba33ea4076bbaac0ae460c17ba P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260809 15:47:00.651607 27689 raft_consensus.cc:2781] T ba56725e42f74a1baa4c04d9660727f9 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260809 15:47:00.698154 27695 raft_consensus.cc:2726] T d3be1e356b254d22b995be32f1c12f6f P 61e69695258f48baa984c9d395e8ce3f [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260809 15:47:00.652348 27519 leader_election.cc:288] T 213543cdbe814c89a17a963469922c1b P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:00.660032 27660 raft_consensus.cc:2726] T cef3e7277d7d4760824a3436ed4a847d P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260809 15:47:00.705160 23470 leader_election.cc:302] T 2db057fd10e149a9895dff6ef8c9f038 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:47:00.665290 27583 raft_consensus.cc:2726] T 4b30f28311964a0b9149a50495e83d8f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260809 15:47:00.628531 27267 consensus_queue.cc:1046] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.065s
I20260809 15:47:00.684563 27657 raft_consensus.cc:482] T 297c82ed906d4b48b184985eed3de601 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:00.740932 23393 raft_consensus.cc:1261] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d [term 9 FOLLOWER]: Refusing update from remote peer 61e69695258f48baa984c9d395e8ce3f: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260809 15:47:00.662545 27669 consensus_queue.cc:1035] T 05ad7d8fdca64e3da527280ca5be9280 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.087s
I20260809 15:47:00.751628 23286 raft_consensus.cc:1261] T 4b30f28311964a0b9149a50495e83d8f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260809 15:47:00.708105 27053 raft_consensus.cc:482] T d1a134ba33ea4076bbaac0ae460c17ba P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:47:00.756258 27053 raft_consensus.cc:3037] T d1a134ba33ea4076bbaac0ae460c17ba P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Advancing to term 6
I20260809 15:47:00.756621 27487 leader_election.cc:288] T cef3e7277d7d4760824a3436ed4a847d P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 7 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:00.757608 27426 raft_consensus.cc:482] T 16ca412a0dae4feebe85dd01bf1a9173 P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:47:00.758087 27426 raft_consensus.cc:504] T 16ca412a0dae4feebe85dd01bf1a9173 P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:00.694600 27322 raft_consensus.cc:686] T a3622f6e3e3641cea1960e3a31498bb7 P 65c36d164b0540f099d592f38307ee6c [term 5 LEADER]: Becoming Leader. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:47:00.735985 27689 raft_consensus.cc:482] T ba56725e42f74a1baa4c04d9660727f9 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:47:00.739708 27196 raft_consensus.cc:482] T 294d4bcc1e7a477cb56455581277a60f P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:47:00.740163 27512 raft_consensus.cc:482] T f9e7cadcd24d45fcbfa84209225e8f2d P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:00.738837 27659 raft_consensus.cc:482] T 563daf42d7784db195ea99c3cb21b370 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:00.759703 27657 raft_consensus.cc:504] T 297c82ed906d4b48b184985eed3de601 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:47:00.791543 24406 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 109875us
I20260809 15:47:00.704701 27347 consensus_queue.cc:1035] T e14d118b2eaa42aa8a7cd4025aba7128 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.055s
I20260809 15:47:00.693547 27378 raft_consensus.cc:504] T 28e17dcb5efe45b8b6a8c42da03b4819 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:00.796232 23519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16ca412a0dae4feebe85dd01bf1a9173" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:00.798745 26619 leader_election.cc:288] T 763e58d6ba644eea830a6c1664d39c9b P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 7 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:47:00.800511 27268 raft_consensus.cc:504] T 953eea268e17491ca76f3a555860a512 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:00.802129 23283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "763e58d6ba644eea830a6c1664d39c9b" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:47:00.802516 27268 leader_election.cc:288] T 953eea268e17491ca76f3a555860a512 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 7 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:47:00.802678 23283 raft_consensus.cc:3037] T 763e58d6ba644eea830a6c1664d39c9b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Advancing to term 7
I20260809 15:47:00.804852 27322 consensus_queue.cc:237] T a3622f6e3e3641cea1960e3a31498bb7 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:00.747357 27081 raft_consensus.cc:2781] T 2db057fd10e149a9895dff6ef8c9f038 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260809 15:47:00.735983 27492 consensus_queue.cc:1046] T 30c1740ef6874df0a332fbd6fd4b03ce P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.032s
I20260809 15:47:00.792907 23280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cef3e7277d7d4760824a3436ed4a847d" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
W20260809 15:47:00.790398 27582 consensus_meta.cc:287] T 2424fd9bd7f14e508fd106cbfda786a4 P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent flushing consensus metadata: real 0.675s user 0.006s sys 0.003s
W20260809 15:47:00.806039 23345 consensus_peers.cc:487] T 4b30f28311964a0b9149a50495e83d8f P c0f38c1908414566aa9115bb2cbdd85d -> Peer 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689): Couldn't send request to peer 61e69695258f48baa984c9d395e8ce3f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4b30f28311964a0b9149a50495e83d8f. This is attempt 1: this message will repeat every 5th retry.
I20260809 15:47:00.738402 27395 raft_consensus.cc:504] T 0a51c0baaa6e432997c9d3552af4f254 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:00.765945 27294 raft_consensus.cc:482] T c0741ab44d99439abc3aa3bdd27ee160 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:00.808662 23228 leader_election.cc:302] T ba56725e42f74a1baa4c04d9660727f9 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters:
W20260809 15:47:00.808907 23353 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 179420us
I20260809 15:47:00.835826 23353 leader_election.cc:302] T 28e17dcb5efe45b8b6a8c42da03b4819 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters: 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:47:00.810555 24406 leader_election.cc:302] T 907bacfa79d04a55a68c8dbe60056462 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 67b3aeadfa7a432abe8af4d292e04ab9; no voters: c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:00.813294 23283 raft_consensus.cc:2454] T 763e58d6ba644eea830a6c1664d39c9b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 7.
I20260809 15:47:00.840575 23283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "953eea268e17491ca76f3a555860a512" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:47:00.788461 27689 raft_consensus.cc:3037] T ba56725e42f74a1baa4c04d9660727f9 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Advancing to term 5
I20260809 15:47:00.813951 27513 raft_consensus.cc:482] T 4b30f28311964a0b9149a50495e83d8f P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:47:00.814280 27544 consensus_queue.cc:1035] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260809 15:47:00.766888 27426 leader_election.cc:288] T 16ca412a0dae4feebe85dd01bf1a9173 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:00.706622 27518 raft_consensus.cc:504] T 4564dfc6d4094c1b9240b367ccecf3b1 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:00.810070 27677 raft_consensus.cc:2726] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260809 15:47:00.808537 27665 log.cc:924] Time spent T 2a823d35f4f348d1b52ba9e6d80b00d2 P 61e69695258f48baa984c9d395e8ce3f: Append to log took a long time: real 0.297s user 0.000s sys 0.001s
W20260809 15:47:00.863020 24406 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 52529.2us
I20260809 15:47:00.793264 27196 raft_consensus.cc:504] T 294d4bcc1e7a477cb56455581277a60f P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:00.829772 27217 raft_consensus.cc:504] T df20671a7b874dbb98c0ef28371c1308 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:00.841450 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1a134ba33ea4076bbaac0ae460c17ba" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:00.805956 27488 raft_consensus.cc:3037] T d3be1e356b254d22b995be32f1c12f6f P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Advancing to term 7
I20260809 15:47:00.841102 23396 raft_consensus.cc:1261] T 2424fd9bd7f14e508fd106cbfda786a4 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 5 index: 2. Preceding OpId from leader: term: 7 index: 3. (index mismatch)
I20260809 15:47:00.816636 27081 raft_consensus.cc:482] T 2db057fd10e149a9895dff6ef8c9f038 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:47:00.844030 23470 leader_election.cc:302] T 763e58d6ba644eea830a6c1664d39c9b P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:47:00.844709 23394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2db057fd10e149a9895dff6ef8c9f038" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:00.847618 23400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5c2e425a74b4e64ae70b5def750434c" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:00.851646 23283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "297c82ed906d4b48b184985eed3de601" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:00.845515 23395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a88421dba80d4d3388aa5eb62e23853b" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:47:00.832496 27378 leader_election.cc:288] T 28e17dcb5efe45b8b6a8c42da03b4819 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 7 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:00.866828 23394 raft_consensus.cc:2454] T 2db057fd10e149a9895dff6ef8c9f038 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 5.
I20260809 15:47:00.861569 23519 raft_consensus.cc:1261] T c5c2e425a74b4e64ae70b5def750434c P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260809 15:47:00.794427 27659 raft_consensus.cc:504] T 563daf42d7784db195ea99c3cb21b370 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:00.812593 26399 raft_consensus.cc:504] T 07fcabb569414bddbd39df6cedd24e64 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:00.867136 23395 raft_consensus.cc:3037] T a88421dba80d4d3388aa5eb62e23853b P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Advancing to term 5
I20260809 15:47:00.864550 23402 raft_consensus.cc:2454] T d1a134ba33ea4076bbaac0ae460c17ba P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 4.
I20260809 15:47:00.867918 23283 raft_consensus.cc:2454] T 297c82ed906d4b48b184985eed3de601 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 6.
I20260809 15:47:00.843328 27347 raft_consensus.cc:2781] T ba56725e42f74a1baa4c04d9660727f9 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260809 15:47:00.842351 27294 raft_consensus.cc:504] T c0741ab44d99439abc3aa3bdd27ee160 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:00.870244 23519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1a134ba33ea4076bbaac0ae460c17ba" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:00.795452 27512 raft_consensus.cc:504] T f9e7cadcd24d45fcbfa84209225e8f2d P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:00.848953 27428 raft_consensus.cc:2781] T 28e17dcb5efe45b8b6a8c42da03b4819 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260809 15:47:00.808279 27404 raft_consensus.cc:504] T df20671a7b874dbb98c0ef28371c1308 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:00.874428 23397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba56725e42f74a1baa4c04d9660727f9" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:01.092788 23395 raft_consensus.cc:2454] T a88421dba80d4d3388aa5eb62e23853b P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 5.
I20260809 15:47:01.100781 23397 raft_consensus.cc:2454] T ba56725e42f74a1baa4c04d9660727f9 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 4.
I20260809 15:47:00.935624 23529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e17dcb5efe45b8b6a8c42da03b4819" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:47:00.857722 27513 raft_consensus.cc:504] T 4b30f28311964a0b9149a50495e83d8f P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:00.874230 27519 consensus_queue.cc:1035] T 4b30f28311964a0b9149a50495e83d8f P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.078s
I20260809 15:47:00.870788 27081 raft_consensus.cc:3037] T 2db057fd10e149a9895dff6ef8c9f038 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Advancing to term 6
I20260809 15:47:00.871716 27426 raft_consensus.cc:2781] T 907bacfa79d04a55a68c8dbe60056462 P 61e69695258f48baa984c9d395e8ce3f [term 6 FOLLOWER]: Leader election won for term 6
I20260809 15:47:00.874900 27347 raft_consensus.cc:482] T ba56725e42f74a1baa4c04d9660727f9 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:47:00.879019 27657 leader_election.cc:288] T 297c82ed906d4b48b184985eed3de601 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:47:00.807289 27480 consensus_queue.cc:237] T 0c48a3049897452383257aa757834fb5 P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
W20260809 15:47:01.096484 23470 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 221128us
I20260809 15:47:01.093958 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "953eea268e17491ca76f3a555860a512" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:47:01.092475 23526 raft_consensus.cc:2379] T 07fcabb569414bddbd39df6cedd24e64 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 in current term 7: Already voted for candidate 65c36d164b0540f099d592f38307ee6c in this term.
I20260809 15:47:01.118572 23470 leader_election.cc:302] T 297c82ed906d4b48b184985eed3de601 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:47:01.090603 23400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "763e58d6ba644eea830a6c1664d39c9b" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:47:00.833204 27582 raft_consensus.cc:504] T 2424fd9bd7f14e508fd106cbfda786a4 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:01.111129 23529 raft_consensus.cc:2379] T 28e17dcb5efe45b8b6a8c42da03b4819 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 in current term 7: Already voted for candidate 65c36d164b0540f099d592f38307ee6c in this term.
I20260809 15:47:01.119287 23400 raft_consensus.cc:3037] T 763e58d6ba644eea830a6c1664d39c9b P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Advancing to term 7
I20260809 15:47:01.118041 23520 raft_consensus.cc:2379] T df20671a7b874dbb98c0ef28371c1308 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 in current term 6: Already voted for candidate 65c36d164b0540f099d592f38307ee6c in this term.
I20260809 15:47:01.120301 23518 raft_consensus.cc:595] T df20671a7b874dbb98c0ef28371c1308 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Received request to transfer leadership to TS c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:01.120760 23518 raft_consensus.cc:603] T df20671a7b874dbb98c0ef28371c1308 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260809 15:47:01.108503 23517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05ad7d8fdca64e3da527280ca5be9280" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:01.124022 27497 consensus_queue.cc:1035] T c5c2e425a74b4e64ae70b5def750434c P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260809 15:47:01.124408 24448 raft_consensus.cc:1261] T cee02fff2c9c48a19f43eb358a89e7b0 P 61e69695258f48baa984c9d395e8ce3f [term 6 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260809 15:47:01.124608 23529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "213543cdbe814c89a17a963469922c1b" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:01.103477 27363 raft_consensus.cc:2781] T 763e58d6ba644eea830a6c1664d39c9b P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Leader election won for term 7
I20260809 15:47:01.101576 27428 raft_consensus.cc:482] T 28e17dcb5efe45b8b6a8c42da03b4819 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:47:01.111840 27688 raft_consensus.cc:482] T 0a51c0baaa6e432997c9d3552af4f254 P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:01.114585 27347 raft_consensus.cc:3037] T ba56725e42f74a1baa4c04d9660727f9 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Advancing to term 5
I20260809 15:47:01.135754 23274 raft_consensus.cc:2379] T 2424fd9bd7f14e508fd106cbfda786a4 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c in current term 7: Already voted for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in this term.
I20260809 15:47:01.136034 23394 raft_consensus.cc:3037] T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Advancing to term 7
I20260809 15:47:01.122336 27394 raft_consensus.cc:482] T 0c29937a80724ddb9d07a27534a146f3 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:01.156181 23529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3e94b7e50754a81b3d72a07476cb7b9" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:01.124367 24453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bcac4b24e7a4448abdc934bba5a624e" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f"
I20260809 15:47:01.158452 27394 raft_consensus.cc:504] T 0c29937a80724ddb9d07a27534a146f3 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:01.158965 27196 leader_election.cc:288] T 294d4bcc1e7a477cb56455581277a60f P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:01.159538 24448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3aa6a9a52804df0ba9b0ca3c82f38c3" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f"
I20260809 15:47:01.115031 27639 raft_consensus.cc:482] T 7660dfa2ef694927b8eda4100a2bcd9b P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:47:01.158522 23529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cef3e7277d7d4760824a3436ed4a847d" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:47:01.136114 27428 raft_consensus.cc:3037] T 28e17dcb5efe45b8b6a8c42da03b4819 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Advancing to term 8
I20260809 15:47:01.160756 27518 leader_election.cc:288] T 4564dfc6d4094c1b9240b367ccecf3b1 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:01.162375 27294 leader_election.cc:288] T c0741ab44d99439abc3aa3bdd27ee160 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:01.129208 27426 raft_consensus.cc:686] T 907bacfa79d04a55a68c8dbe60056462 P 61e69695258f48baa984c9d395e8ce3f [term 6 LEADER]: Becoming Leader. State: Replica: 61e69695258f48baa984c9d395e8ce3f, State: Running, Role: LEADER
I20260809 15:47:01.164497 23517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ced718c1a5c4da79cb8c1ebd1435b87" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:01.109539 27053 raft_consensus.cc:504] T d1a134ba33ea4076bbaac0ae460c17ba P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:47:01.168565 23470 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 50090.6us
W20260809 15:47:01.133188 27699 log.cc:924] Time spent T d3be1e356b254d22b995be32f1c12f6f P 67b3aeadfa7a432abe8af4d292e04ab9: Append to log took a long time: real 0.262s user 0.000s sys 0.005s
I20260809 15:47:01.171520 24452 raft_consensus.cc:1226] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 61e69695258f48baa984c9d395e8ce3f [term 9 LEADER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 8. Current term is 9. Ops: []
I20260809 15:47:01.173345 24448 raft_consensus.cc:2379] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c in current term 5: Already voted for candidate 61e69695258f48baa984c9d395e8ce3f in this term.
I20260809 15:47:01.179121 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "297c82ed906d4b48b184985eed3de601" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:01.180595 23353 leader_election.cc:302] T 213543cdbe814c89a17a963469922c1b P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:47:01.183960 23402 raft_consensus.cc:2454] T 297c82ed906d4b48b184985eed3de601 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 7.
I20260809 15:47:01.157871 27704 raft_consensus.cc:482] T 907bacfa79d04a55a68c8dbe60056462 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:01.165249 27217 leader_election.cc:288] T df20671a7b874dbb98c0ef28371c1308 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 6 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:01.184412 23353 leader_election.cc:302] T 05ad7d8fdca64e3da527280ca5be9280 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:47:01.169414 26399 leader_election.cc:288] T 07fcabb569414bddbd39df6cedd24e64 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 7 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:47:01.205593 23529 raft_consensus.cc:1226] T 953eea268e17491ca76f3a555860a512 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Rejecting Update request from peer c0f38c1908414566aa9115bb2cbdd85d for earlier term 6. Current term is 7. Ops: []
I20260809 15:47:01.205508 27704 raft_consensus.cc:504] T 907bacfa79d04a55a68c8dbe60056462 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:01.221120 23397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e17dcb5efe45b8b6a8c42da03b4819" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
W20260809 15:47:01.225644 27703 log.cc:924] Time spent T 953eea268e17491ca76f3a555860a512 P 67b3aeadfa7a432abe8af4d292e04ab9: Append to log took a long time: real 0.062s user 0.000s sys 0.000s
I20260809 15:47:01.227123 27582 leader_election.cc:288] T 2424fd9bd7f14e508fd106cbfda786a4 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 7 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:01.229604 27689 raft_consensus.cc:504] T ba56725e42f74a1baa4c04d9660727f9 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:01.230265 27395 leader_election.cc:288] T 0a51c0baaa6e432997c9d3552af4f254 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:01.173651 27322 raft_consensus.cc:2781] T 297c82ed906d4b48b184985eed3de601 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260809 15:47:01.247713 27322 raft_consensus.cc:482] T 297c82ed906d4b48b184985eed3de601 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:47:01.248059 27322 raft_consensus.cc:3037] T 297c82ed906d4b48b184985eed3de601 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Advancing to term 8
W20260809 15:47:01.205094 27658 consensus_meta.cc:287] T 07fcabb569414bddbd39df6cedd24e64 P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent flushing consensus metadata: real 0.573s user 0.006s sys 0.010s
I20260809 15:47:01.242172 27518 raft_consensus.cc:482] T 3d880de657e544468c82157769622a26 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:01.251638 27658 raft_consensus.cc:504] T 07fcabb569414bddbd39df6cedd24e64 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:01.252168 27518 raft_consensus.cc:504] T 3d880de657e544468c82157769622a26 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:01.253655 27658 leader_election.cc:288] T 07fcabb569414bddbd39df6cedd24e64 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 7 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:01.254016 27518 leader_election.cc:288] T 3d880de657e544468c82157769622a26 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:01.257090 23273 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53562 (request call id 1808) took 1074 ms. Trace:
I20260809 15:47:01.258077 23472 leader_election.cc:302] T c5c2e425a74b4e64ae70b5def750434c P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:01.259338 27513 leader_election.cc:288] T 4b30f28311964a0b9149a50495e83d8f P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:47:01.260324 27513 consensus_queue.cc:1035] T 2424fd9bd7f14e508fd106cbfda786a4 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 15:47:01.260637 27496 raft_consensus.cc:2726] T c5c2e425a74b4e64ae70b5def750434c P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260809 15:47:01.230913 23396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "563daf42d7784db195ea99c3cb21b370" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:01.186280 27682 raft_consensus.cc:2726] T 213543cdbe814c89a17a963469922c1b P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260809 15:47:01.175662 27639 raft_consensus.cc:504] T 7660dfa2ef694927b8eda4100a2bcd9b P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:01.231280 23519 raft_consensus.cc:2354] T d1a134ba33ea4076bbaac0ae460c17ba P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d for earlier term 5. Current term is 6.
I20260809 15:47:01.254980 26399 raft_consensus.cc:482] T 6ae14a74a26f434db08c652e097385c8 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:01.222270 27426 consensus_queue.cc:237] T 907bacfa79d04a55a68c8dbe60056462 P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:01.187361 23276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0741ab44d99439abc3aa3bdd27ee160" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:01.265765 23276 raft_consensus.cc:2454] T c0741ab44d99439abc3aa3bdd27ee160 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 5.
I20260809 15:47:01.165481 27363 raft_consensus.cc:686] T 763e58d6ba644eea830a6c1664d39c9b P 65c36d164b0540f099d592f38307ee6c [term 7 LEADER]: Becoming Leader. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:47:01.220129 27053 leader_election.cc:288] T d1a134ba33ea4076bbaac0ae460c17ba P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 6 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:47:01.237609 27488 raft_consensus.cc:2726] T 05ad7d8fdca64e3da527280ca5be9280 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260809 15:47:01.176920 27688 raft_consensus.cc:504] T 0a51c0baaa6e432997c9d3552af4f254 P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
W20260809 15:47:01.236555 23353 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 52222us
I20260809 15:47:01.273273 23353 leader_election.cc:302] T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c
I20260809 15:47:01.274135 23353 leader_election.cc:302] T cef3e7277d7d4760824a3436ed4a847d P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:47:01.237257 23282 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53562 (request call id 1802) took 1126 ms. Trace:
I20260809 15:47:01.238456 23394 raft_consensus.cc:1261] T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260809 15:47:01.228236 27394 leader_election.cc:288] T 0c29937a80724ddb9d07a27534a146f3 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:47:01.241518 23517 raft_consensus.cc:1261] T 4b30f28311964a0b9149a50495e83d8f P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260809 15:47:01.243242 23529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4564dfc6d4094c1b9240b367ccecf3b1" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:01.243819 24452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a51c0baaa6e432997c9d3552af4f254" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:47:01.240566 27659 leader_election.cc:288] T 563daf42d7784db195ea99c3cb21b370 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:01.246022 24453 raft_consensus.cc:2454] T 4bcac4b24e7a4448abdc934bba5a624e P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 3.
I20260809 15:47:01.246774 24448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "907bacfa79d04a55a68c8dbe60056462" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:47:01.264248 23400 raft_consensus.cc:2454] T 763e58d6ba644eea830a6c1664d39c9b P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 7.
I20260809 15:47:01.218623 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4564dfc6d4094c1b9240b367ccecf3b1" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:01.267750 23472 leader_election.cc:302] T 953eea268e17491ca76f3a555860a512 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:01.257472 23273 rpcz_store.cc:276] 0809 15:47:00.182681 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:00.182791 (+ 110us) service_pool.cc:227] Handling call
0809 15:47:00.219598 (+ 36807us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 15:47:00.249654 (+ 30056us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:47:00.249684 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 15:47:00.596944 (+347260us) spinlock_profiling.cc:243] Waited 23.5 ms on lock 0x7b2c0000f228. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e7c03e11 00007f34e7c13442 00007f34e925bc11 00007f34e922058a 00007f34e921f99e 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772
0809 15:47:00.597260 (+ 316us) log.cc:842] Serialized 26 byte log entry
0809 15:47:00.707800 (+110540us) raft_consensus.cc:1631] Marking committed up to 3
0809 15:47:00.707832 (+ 32us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:47:00.707893 (+ 61us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 15:47:01.256769 (+548876us) raft_consensus.cc:1671] finished
0809 15:47:01.256796 (+ 27us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:47:01.257026 (+ 230us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":29727,"raft.queue_time_us":30208,"raft.run_cpu_time_us":72,"raft.run_wall_time_us":72,"spinlock_wait_cycles":53994240,"thread_start_us":110220,"threads_started":1,"wal-append.queue_time_us":277406}
I20260809 15:47:01.234649 23286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07fcabb569414bddbd39df6cedd24e64" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:47:01.282052 23286 raft_consensus.cc:2379] T 07fcabb569414bddbd39df6cedd24e64 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c in current term 7: Already voted for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in this term.
I20260809 15:47:01.282685 23402 raft_consensus.cc:2379] T 4564dfc6d4094c1b9240b367ccecf3b1 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 in current term 6: Already voted for candidate c0f38c1908414566aa9115bb2cbdd85d in this term.
W20260809 15:47:01.257615 27661 consensus_meta.cc:287] T cee02fff2c9c48a19f43eb358a89e7b0 P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent flushing consensus metadata: real 0.645s user 0.010s sys 0.000s
I20260809 15:47:01.272173 23522 raft_consensus.cc:2379] T ba56725e42f74a1baa4c04d9660727f9 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 in current term 5: Already voted for candidate 65c36d164b0540f099d592f38307ee6c in this term.
I20260809 15:47:01.274534 23282 rpcz_store.cc:276] 0809 15:47:00.110968 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:00.111998 (+ 1030us) service_pool.cc:227] Handling call
0809 15:47:00.130306 (+ 18308us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 15:47:00.167051 (+ 36745us) raft_consensus.cc:1520] Early marking committed up to index 5
0809 15:47:00.462770 (+295719us) log.cc:842] Serialized 14 byte log entry
0809 15:47:00.664661 (+201891us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 15:47:00.687086 (+ 22425us) log.cc:842] Serialized 26 byte log entry
0809 15:47:00.687398 (+ 312us) raft_consensus.cc:1631] Marking committed up to 6
0809 15:47:00.689674 (+ 2276us) log.cc:842] Serialized 14 byte log entry
0809 15:47:00.692832 (+ 3158us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:47:00.692972 (+ 140us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 15:47:01.237081 (+544109us) raft_consensus.cc:1671] finished
0809 15:47:01.237100 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:47:01.237207 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":16841,"prepare.queue_time_us":384669,"prepare.run_cpu_time_us":83,"prepare.run_wall_time_us":2435,"raft.queue_time_us":26190,"raft.run_cpu_time_us":303,"raft.run_wall_time_us":27504,"thread_start_us":387612,"threads_started":2,"wal-append.queue_time_us":330875}
I20260809 15:47:01.275316 23394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c29937a80724ddb9d07a27534a146f3" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 7 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:01.277073 23396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1a134ba33ea4076bbaac0ae460c17ba" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:47:01.277119 23353 leader_election.cc:302] T 6ced718c1a5c4da79cb8c1ebd1435b87 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:47:01.286013 23396 raft_consensus.cc:3037] T d1a134ba33ea4076bbaac0ae460c17ba P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Advancing to term 6
I20260809 15:47:01.245553 27704 leader_election.cc:288] T 907bacfa79d04a55a68c8dbe60056462 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:01.278129 23529 raft_consensus.cc:2454] T 4564dfc6d4094c1b9240b367ccecf3b1 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 6.
I20260809 15:47:01.284938 23400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07fcabb569414bddbd39df6cedd24e64" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:47:01.288295 23400 raft_consensus.cc:3037] T 07fcabb569414bddbd39df6cedd24e64 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Advancing to term 7
I20260809 15:47:01.287365 24452 raft_consensus.cc:2379] T 0a51c0baaa6e432997c9d3552af4f254 P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d in current term 5: Already voted for candidate 61e69695258f48baa984c9d395e8ce3f in this term.
I20260809 15:47:01.289413 23517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "294d4bcc1e7a477cb56455581277a60f" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:01.289588 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df20671a7b874dbb98c0ef28371c1308" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:47:01.289989 24453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0741ab44d99439abc3aa3bdd27ee160" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:47:01.290455 23517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df20671a7b874dbb98c0ef28371c1308" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:47:01.290628 24453 raft_consensus.cc:2454] T c0741ab44d99439abc3aa3bdd27ee160 P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 5.
I20260809 15:47:01.291486 23517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2424fd9bd7f14e508fd106cbfda786a4" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 7 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:47:01.291643 23529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d880de657e544468c82157769622a26" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:01.292469 23517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07fcabb569414bddbd39df6cedd24e64" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:47:01.293272 27497 raft_consensus.cc:2726] T 6ced718c1a5c4da79cb8c1ebd1435b87 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260809 15:47:01.294314 23286 raft_consensus.cc:1261] T 00a1881267a14ba8b78c81040f17a014 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260809 15:47:01.290145 23402 raft_consensus.cc:3037] T df20671a7b874dbb98c0ef28371c1308 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Advancing to term 6
I20260809 15:47:01.295444 26730 consensus_queue.cc:1035] T 4b30f28311964a0b9149a50495e83d8f P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260809 15:47:01.291437 23404 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55320 (request call id 1453) took 1230 ms. Trace:
I20260809 15:47:01.298355 23404 rpcz_store.cc:276] 0809 15:47:00.061265 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:00.061375 (+ 110us) service_pool.cc:227] Handling call
0809 15:47:00.061662 (+ 287us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:47:00.686037 (+624375us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:47:01.291188 (+605151us) spinlock_profiling.cc:243] Waited 618 ms on lock 0x7b54000a7dd4. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e9220f51 00007f34e921f99e 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772 00007f34e764a94c 00007f34e7649d86
0809 15:47:01.291376 (+ 188us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1421079424}
I20260809 15:47:01.288740 23228 leader_election.cc:302] T 4564dfc6d4094c1b9240b367ccecf3b1 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters: c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:01.295058 23400 raft_consensus.cc:2454] T 07fcabb569414bddbd39df6cedd24e64 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 7.
I20260809 15:47:01.300251 23400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07fcabb569414bddbd39df6cedd24e64" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:47:01.300886 23400 raft_consensus.cc:2379] T 07fcabb569414bddbd39df6cedd24e64 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 in current term 7: Already voted for candidate 65c36d164b0540f099d592f38307ee6c in this term.
I20260809 15:47:01.301669 23228 leader_election.cc:302] T 07fcabb569414bddbd39df6cedd24e64 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:01.279646 27682 raft_consensus.cc:2726] T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260809 15:47:01.297601 23394 raft_consensus.cc:2454] T 0c29937a80724ddb9d07a27534a146f3 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 6.
I20260809 15:47:01.303311 23517 raft_consensus.cc:1261] T ae6d2c08a7a94f58ac8dd698e3fd456c P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260809 15:47:01.305310 23276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16ca412a0dae4feebe85dd01bf1a9173" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:01.280925 27395 raft_consensus.cc:2726] T cef3e7277d7d4760824a3436ed4a847d P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260809 15:47:01.309979 23472 leader_election.cc:302] T 07fcabb569414bddbd39df6cedd24e64 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:47:01.310328 23468 leader_election.cc:302] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:01.309433 23470 leader_election.cc:302] T c0741ab44d99439abc3aa3bdd27ee160 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:47:01.284014 26966 raft_consensus.cc:482] T 0c48a3049897452383257aa757834fb5 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:01.316193 23276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "907bacfa79d04a55a68c8dbe60056462" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:01.316774 23276 raft_consensus.cc:2454] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 6.
I20260809 15:47:01.318260 27639 leader_election.cc:288] T 7660dfa2ef694927b8eda4100a2bcd9b P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:01.318248 27363 consensus_queue.cc:237] T 763e58d6ba644eea830a6c1664d39c9b P 65c36d164b0540f099d592f38307ee6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:01.318872 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d880de657e544468c82157769622a26" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:47:01.319311 23404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2424fd9bd7f14e508fd106cbfda786a4" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 7 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:47:01.320180 23404 raft_consensus.cc:2379] T 2424fd9bd7f14e508fd106cbfda786a4 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 in current term 7: Already voted for candidate 65c36d164b0540f099d592f38307ee6c in this term.
I20260809 15:47:01.321244 23404 raft_consensus.cc:1261] T 00a1881267a14ba8b78c81040f17a014 P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260809 15:47:01.321780 27688 leader_election.cc:288] T 0a51c0baaa6e432997c9d3552af4f254 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:47:01.324501 23280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a51c0baaa6e432997c9d3552af4f254" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:01.325069 23280 raft_consensus.cc:2454] T 0a51c0baaa6e432997c9d3552af4f254 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 5.
I20260809 15:47:01.327337 23273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d880de657e544468c82157769622a26" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:01.327862 23286 raft_consensus.cc:1261] T a88421dba80d4d3388aa5eb62e23853b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260809 15:47:01.327903 23273 raft_consensus.cc:2454] T 3d880de657e544468c82157769622a26 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 2.
I20260809 15:47:01.307713 27659 raft_consensus.cc:2781] T 4564dfc6d4094c1b9240b367ccecf3b1 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260809 15:47:01.329649 27659 raft_consensus.cc:482] T 4564dfc6d4094c1b9240b367ccecf3b1 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:47:01.330011 27659 raft_consensus.cc:3037] T 4564dfc6d4094c1b9240b367ccecf3b1 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Advancing to term 6
W20260809 15:47:01.302771 27715 log.cc:924] Time spent T 05ad7d8fdca64e3da527280ca5be9280 P c0f38c1908414566aa9115bb2cbdd85d: Append to log took a long time: real 0.203s user 0.000s sys 0.001s
I20260809 15:47:01.317399 27305 consensus_queue.cc:1035] T d3e94b7e50754a81b3d72a07476cb7b9 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.015s
I20260809 15:47:01.333050 23518 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:48888 (request call id 6) took 1129 ms. Trace:
I20260809 15:47:01.333377 23518 rpcz_store.cc:276] 0809 15:47:00.203785 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:00.204308 (+ 523us) service_pool.cc:227] Handling call
0809 15:47:01.332863 (+1128555us) spinlock_profiling.cc:243] Waited 905 ms on lock 0x7b54001fc554. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e921abc3 00007f34ee16481a 00007f34e7857d04 00007f34e7857c92 00007f34e7857bc1 00007f34e7857b3c 00007f34e7856df2 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:47:01.332962 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2082327936}
I20260809 15:47:01.334815 27081 raft_consensus.cc:504] T 2db057fd10e149a9895dff6ef8c9f038 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:01.336786 27081 leader_election.cc:288] T 2db057fd10e149a9895dff6ef8c9f038 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 6 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:47:01.312439 27518 raft_consensus.cc:2726] T 07fcabb569414bddbd39df6cedd24e64 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260809 15:47:01.336907 27659 raft_consensus.cc:504] T 4564dfc6d4094c1b9240b367ccecf3b1 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:01.285145 27394 raft_consensus.cc:2726] T 953eea268e17491ca76f3a555860a512 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260809 15:47:01.302330 27704 raft_consensus.cc:482] T 4bcac4b24e7a4448abdc934bba5a624e P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
I20260809 15:47:01.342355 27322 raft_consensus.cc:504] T 297c82ed906d4b48b184985eed3de601 P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:47:01.343779 27716 log.cc:924] Time spent T e14d118b2eaa42aa8a7cd4025aba7128 P 65c36d164b0540f099d592f38307ee6c: Append to log took a long time: real 0.118s user 0.002s sys 0.000s
I20260809 15:47:01.359445 23273 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f7715d71dbac447b9351fc5eb36c5e80"
dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
mode: GRACEFUL
new_leader_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
from {username='slave'} at 127.0.0.1:33468
I20260809 15:47:01.359964 23273 raft_consensus.cc:595] T f7715d71dbac447b9351fc5eb36c5e80 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Received request to transfer leadership to TS c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:01.302402 27488 consensus_queue.cc:1046] T 953eea268e17491ca76f3a555860a512 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.049s
I20260809 15:47:01.360567 27639 raft_consensus.cc:2781] T 07fcabb569414bddbd39df6cedd24e64 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Leader election won for term 7
I20260809 15:47:01.272884 26399 raft_consensus.cc:504] T 6ae14a74a26f434db08c652e097385c8 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:01.326844 27428 raft_consensus.cc:504] T 28e17dcb5efe45b8b6a8c42da03b4819 P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:47:01.361549 27721 log.cc:924] Time spent T d2d4f749fcfd4fb3bfb8115af3471598 P c0f38c1908414566aa9115bb2cbdd85d: Append to log took a long time: real 0.138s user 0.001s sys 0.000s
I20260809 15:47:01.363617 23280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "294d4bcc1e7a477cb56455581277a60f" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:01.361739 23402 raft_consensus.cc:2454] T df20671a7b874dbb98c0ef28371c1308 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 6.
I20260809 15:47:01.368767 23400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a51c0baaa6e432997c9d3552af4f254" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:01.369901 23400 raft_consensus.cc:2454] T 0a51c0baaa6e432997c9d3552af4f254 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 4.
I20260809 15:47:01.355012 23396 raft_consensus.cc:2454] T d1a134ba33ea4076bbaac0ae460c17ba P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 6.
I20260809 15:47:01.351188 26966 raft_consensus.cc:504] T 0c48a3049897452383257aa757834fb5 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:01.357898 27704 raft_consensus.cc:504] T 4bcac4b24e7a4448abdc934bba5a624e P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
W20260809 15:47:01.358667 23353 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 43683.4us
I20260809 15:47:01.360260 23273 raft_consensus.cc:603] T f7715d71dbac447b9351fc5eb36c5e80 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260809 15:47:01.372074 23282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1a134ba33ea4076bbaac0ae460c17ba" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:47:01.379333 23282 raft_consensus.cc:3037] T d1a134ba33ea4076bbaac0ae460c17ba P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Advancing to term 6
I20260809 15:47:01.372484 24410 leader_election.cc:302] T 0a51c0baaa6e432997c9d3552af4f254 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:47:01.362511 24468 raft_consensus.cc:2379] T 3d880de657e544468c82157769622a26 P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 in current term 3: Already voted for candidate 65c36d164b0540f099d592f38307ee6c in this term.
I20260809 15:47:01.385103 23286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c29937a80724ddb9d07a27534a146f3" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 7 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:01.418514 23397 raft_consensus.cc:2354] T 28e17dcb5efe45b8b6a8c42da03b4819 P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 7. Current term is 8.
W20260809 15:47:01.417300 24410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 44896us
I20260809 15:47:01.287738 27661 raft_consensus.cc:504] T cee02fff2c9c48a19f43eb358a89e7b0 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:01.360801 27488 raft_consensus.cc:3032] T 953eea268e17491ca76f3a555860a512 P c0f38c1908414566aa9115bb2cbdd85d [term 6 LEADER]: Stepping down as leader of term 6
I20260809 15:47:01.421391 23280 raft_consensus.cc:1261] T a3622f6e3e3641cea1960e3a31498bb7 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260809 15:47:01.364269 27513 raft_consensus.cc:2726] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260809 15:47:01.362200 24453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "563daf42d7784db195ea99c3cb21b370" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:47:01.362618 26399 leader_election.cc:288] T 6ae14a74a26f434db08c652e097385c8 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:47:01.429066 24406 leader_election.cc:302] T 16ca412a0dae4feebe85dd01bf1a9173 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:47:01.314941 27468 consensus_queue.cc:1035] T cee02fff2c9c48a19f43eb358a89e7b0 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.105s
I20260809 15:47:01.350201 27496 consensus_queue.cc:1046] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.070s
I20260809 15:47:01.423501 23280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b30f28311964a0b9149a50495e83d8f" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:01.348692 27032 consensus_queue.cc:1046] T cef3e7277d7d4760824a3436ed4a847d P 65c36d164b0540f099d592f38307ee6c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: INVALID_TERM, Last received: 6.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.069s
I20260809 15:47:01.436123 23270 raft_consensus.cc:1226] T cee02fff2c9c48a19f43eb358a89e7b0 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 6. Current term is 7. Ops: []
I20260809 15:47:01.435575 23280 raft_consensus.cc:2454] T 4b30f28311964a0b9149a50495e83d8f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 3.
I20260809 15:47:01.426772 23228 leader_election.cc:302] T 2424fd9bd7f14e508fd106cbfda786a4 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:01.368412 27657 raft_consensus.cc:2781] T c0741ab44d99439abc3aa3bdd27ee160 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260809 15:47:01.431198 23276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a51c0baaa6e432997c9d3552af4f254" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:01.435860 23528 raft_consensus.cc:3032] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 65c36d164b0540f099d592f38307ee6c [term 8 LEADER]: Stepping down as leader of term 8
I20260809 15:47:01.437903 24406 leader_election.cc:302] T 3d880de657e544468c82157769622a26 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c
I20260809 15:47:01.438359 23282 raft_consensus.cc:2454] T d1a134ba33ea4076bbaac0ae460c17ba P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 6.
W20260809 15:47:01.437708 23472 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 127786us
I20260809 15:47:01.440062 23472 leader_election.cc:302] T 0c29937a80724ddb9d07a27534a146f3 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
W20260809 15:47:01.436738 23468 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.68:34689, user_credentials={real_user=slave}} blocked reactor thread for 120687us
I20260809 15:47:01.438629 23276 raft_consensus.cc:2454] T 0a51c0baaa6e432997c9d3552af4f254 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 5.
W20260809 15:47:01.440667 23470 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 131323us
I20260809 15:47:01.440166 23228 leader_election.cc:302] T df20671a7b874dbb98c0ef28371c1308 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c
I20260809 15:47:01.442554 23468 leader_election.cc:302] T 4bcac4b24e7a4448abdc934bba5a624e P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c; no voters: c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:01.440584 23528 raft_consensus.cc:729] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 65c36d164b0540f099d592f38307ee6c [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:47:01.423725 27639 raft_consensus.cc:686] T 07fcabb569414bddbd39df6cedd24e64 P 65c36d164b0540f099d592f38307ee6c [term 7 LEADER]: Becoming Leader. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:47:01.423548 27488 raft_consensus.cc:729] T 953eea268e17491ca76f3a555860a512 P c0f38c1908414566aa9115bb2cbdd85d [term 6 LEADER]: Becoming Follower/Learner. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:47:01.430531 27556 raft_consensus.cc:2781] T 0a51c0baaa6e432997c9d3552af4f254 P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260809 15:47:01.447067 24406 leader_election.cc:302] T 294d4bcc1e7a477cb56455581277a60f P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:47:01.435882 27695 raft_consensus.cc:2726] T 16ca412a0dae4feebe85dd01bf1a9173 P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260809 15:47:01.439877 27657 raft_consensus.cc:482] T c0741ab44d99439abc3aa3bdd27ee160 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:47:01.452909 23472 leader_election.cc:302] T d1a134ba33ea4076bbaac0ae460c17ba P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:47:01.455998 23393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b30f28311964a0b9149a50495e83d8f" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:01.457391 23396 raft_consensus.cc:1261] T a88421dba80d4d3388aa5eb62e23853b P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260809 15:47:01.458675 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7660dfa2ef694927b8eda4100a2bcd9b" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:01.440383 27496 consensus_queue.cc:1225] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Peer 61e69695258f48baa984c9d395e8ce3f log is divergent from this leader: its last log entry 9.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20260809 15:47:01.443431 27518 raft_consensus.cc:2726] T 2424fd9bd7f14e508fd106cbfda786a4 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260809 15:47:01.444944 27669 raft_consensus.cc:2781] T df20671a7b874dbb98c0ef28371c1308 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Leader election won for term 6
I20260809 15:47:01.521883 23276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba56725e42f74a1baa4c04d9660727f9" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
W20260809 15:47:01.517937 23468 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.68:34689, user_credentials={real_user=slave}} blocked reactor thread for 55445.6us
I20260809 15:47:01.448539 27556 raft_consensus.cc:482] T 0a51c0baaa6e432997c9d3552af4f254 P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:47:01.524315 23396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2db057fd10e149a9895dff6ef8c9f038" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:47:01.524880 23396 raft_consensus.cc:3037] T 2db057fd10e149a9895dff6ef8c9f038 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Advancing to term 6
I20260809 15:47:01.525544 23353 leader_election.cc:398] T d1a134ba33ea4076bbaac0ae460c17ba P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 pre-election: Vote denied by peer 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553) with higher term. Message: Invalid argument: T d1a134ba33ea4076bbaac0ae460c17ba P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d for earlier term 5. Current term is 6.
I20260809 15:47:01.525839 23353 leader_election.cc:401] T d1a134ba33ea4076bbaac0ae460c17ba P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 pre-election: Cancelling election due to peer responding with higher term
I20260809 15:47:01.448882 27546 raft_consensus.cc:2741] T 3d880de657e544468c82157769622a26 P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260809 15:47:01.526620 23280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2db057fd10e149a9895dff6ef8c9f038" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:47:01.527217 23280 raft_consensus.cc:3037] T 2db057fd10e149a9895dff6ef8c9f038 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Advancing to term 6
I20260809 15:47:01.450160 26399 raft_consensus.cc:2781] T 0c29937a80724ddb9d07a27534a146f3 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260809 15:47:01.525883 23404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ae14a74a26f434db08c652e097385c8" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:01.529656 23404 raft_consensus.cc:2454] T 6ae14a74a26f434db08c652e097385c8 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 6.
I20260809 15:47:01.450325 27689 leader_election.cc:288] T ba56725e42f74a1baa4c04d9660727f9 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 5 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:47:01.461678 23223 leader_election.cc:302] T 3d880de657e544468c82157769622a26 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c
W20260809 15:47:01.563741 23472 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 96257.6us
I20260809 15:47:01.531955 23395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba56725e42f74a1baa4c04d9660727f9" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:47:01.566817 23394 raft_consensus.cc:1226] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d [term 9 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 8. Current term is 9. Ops: [8.8-8.8]
I20260809 15:47:01.567514 23395 raft_consensus.cc:3037] T ba56725e42f74a1baa4c04d9660727f9 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Advancing to term 5
I20260809 15:47:01.459903 27513 raft_consensus.cc:2781] T 4bcac4b24e7a4448abdc934bba5a624e P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Leader election won for term 3
I20260809 15:47:01.574990 23286 raft_consensus.cc:3037] T 3d880de657e544468c82157769622a26 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:47:01.459009 27657 raft_consensus.cc:3037] T c0741ab44d99439abc3aa3bdd27ee160 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Advancing to term 6
I20260809 15:47:01.468765 27491 raft_consensus.cc:2781] T d1a134ba33ea4076bbaac0ae460c17ba P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Leader election won for term 6
I20260809 15:47:01.580843 23395 raft_consensus.cc:2454] T ba56725e42f74a1baa4c04d9660727f9 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 5.
I20260809 15:47:01.587342 23472 leader_election.cc:302] T 6ae14a74a26f434db08c652e097385c8 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:47:01.587554 23282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ae14a74a26f434db08c652e097385c8" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:01.590548 23282 raft_consensus.cc:2396] T 6ae14a74a26f434db08c652e097385c8 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c for term 7 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 2.
I20260809 15:47:01.531278 27556 raft_consensus.cc:3037] T 0a51c0baaa6e432997c9d3552af4f254 P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Advancing to term 6
W20260809 15:47:01.533027 27662 log.cc:924] Time spent T d2d4f749fcfd4fb3bfb8115af3471598 P 65c36d164b0540f099d592f38307ee6c: Append to log took a long time: real 0.082s user 0.000s sys 0.000s
W20260809 15:47:01.589092 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 291824us
I20260809 15:47:01.595506 23351 leader_election.cc:302] T 907bacfa79d04a55a68c8dbe60056462 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters: 61e69695258f48baa984c9d395e8ce3f
I20260809 15:47:01.461284 27426 raft_consensus.cc:2726] T 294d4bcc1e7a477cb56455581277a60f P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260809 15:47:01.533655 26399 raft_consensus.cc:482] T 0c29937a80724ddb9d07a27534a146f3 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:47:01.600052 23351 leader_election.cc:302] T 0a51c0baaa6e432997c9d3552af4f254 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters: 61e69695258f48baa984c9d395e8ce3f
I20260809 15:47:01.601549 23518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bcac4b24e7a4448abdc934bba5a624e" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
W20260809 15:47:01.657836 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 57948.1us
I20260809 15:47:01.452351 27363 consensus_queue.cc:1035] T 00a1881267a14ba8b78c81040f17a014 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.138s
W20260809 15:47:01.634290 23470 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 168220us
I20260809 15:47:01.575899 27316 raft_consensus.cc:2726] T d1a134ba33ea4076bbaac0ae460c17ba P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: Vote denied by peer 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553) with higher term. Message: Invalid argument: T d1a134ba33ea4076bbaac0ae460c17ba P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d for earlier term 5. Current term is 6.
I20260809 15:47:01.668367 23286 raft_consensus.cc:1261] T 3d880de657e544468c82157769622a26 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260809 15:47:01.587859 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7660dfa2ef694927b8eda4100a2bcd9b" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
W20260809 15:47:01.589104 27723 log.cc:924] Time spent T e14d118b2eaa42aa8a7cd4025aba7128 P 61e69695258f48baa984c9d395e8ce3f: Append to log took a long time: real 0.058s user 0.000s sys 0.000s
I20260809 15:47:01.572443 27677 consensus_queue.cc:1035] T ae6d2c08a7a94f58ac8dd698e3fd456c P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.094s
I20260809 15:47:01.676185 23472 leader_election.cc:302] T ba56725e42f74a1baa4c04d9660727f9 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:47:01.579499 26619 consensus_queue.cc:1035] T a88421dba80d4d3388aa5eb62e23853b P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.096s
I20260809 15:47:01.598687 27513 raft_consensus.cc:686] T 4bcac4b24e7a4448abdc934bba5a624e P 65c36d164b0540f099d592f38307ee6c [term 3 LEADER]: Becoming Leader. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:47:01.665934 27395 raft_consensus.cc:2781] T 907bacfa79d04a55a68c8dbe60056462 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260809 15:47:01.638427 27480 raft_consensus.cc:482] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:01.593366 27659 leader_election.cc:288] T 4564dfc6d4094c1b9240b367ccecf3b1 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 6 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:01.669939 27682 raft_consensus.cc:2781] T 0a51c0baaa6e432997c9d3552af4f254 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260809 15:47:01.693102 23396 raft_consensus.cc:2454] T 2db057fd10e149a9895dff6ef8c9f038 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 6.
I20260809 15:47:01.667739 27394 raft_consensus.cc:2781] T 6ae14a74a26f434db08c652e097385c8 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260809 15:47:01.692312 27268 consensus_queue.cc:1035] T 00a1881267a14ba8b78c81040f17a014 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.005s
I20260809 15:47:01.698674 23394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4564dfc6d4094c1b9240b367ccecf3b1" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:47:01.692001 27316 raft_consensus.cc:482] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:01.664954 27267 raft_consensus.cc:482] T 538a228ddfb54bc6a36528d5ff07597b P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:01.690254 27695 raft_consensus.cc:482] T d3e94b7e50754a81b3d72a07476cb7b9 P 61e69695258f48baa984c9d395e8ce3f [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:01.670130 26399 raft_consensus.cc:3037] T 0c29937a80724ddb9d07a27534a146f3 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Advancing to term 7
I20260809 15:47:01.684865 21289 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260809 15:47:01.673867 27428 leader_election.cc:288] T 28e17dcb5efe45b8b6a8c42da03b4819 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 8 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:01.573251 27488 consensus_queue.cc:260] T 953eea268e17491ca76f3a555860a512 P c0f38c1908414566aa9115bb2cbdd85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:01.705186 27669 raft_consensus.cc:686] T df20671a7b874dbb98c0ef28371c1308 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 LEADER]: Becoming Leader. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Running, Role: LEADER
I20260809 15:47:01.674129 27704 leader_election.cc:288] T 4bcac4b24e7a4448abdc934bba5a624e P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:01.729714 23394 raft_consensus.cc:2379] T 4564dfc6d4094c1b9240b367ccecf3b1 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 in current term 6: Already voted for candidate c0f38c1908414566aa9115bb2cbdd85d in this term.
I20260809 15:47:01.655911 27491 raft_consensus.cc:686] T d1a134ba33ea4076bbaac0ae460c17ba P 65c36d164b0540f099d592f38307ee6c [term 6 LEADER]: Becoming Leader. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:47:01.685434 27322 leader_election.cc:288] T 297c82ed906d4b48b184985eed3de601 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 8 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:47:01.699558 24453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bcac4b24e7a4448abdc934bba5a624e" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:47:01.700023 27395 raft_consensus.cc:482] T 907bacfa79d04a55a68c8dbe60056462 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:47:01.700701 26966 leader_election.cc:288] T 0c48a3049897452383257aa757834fb5 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:01.744681 27267 raft_consensus.cc:504] T 538a228ddfb54bc6a36528d5ff07597b P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:01.744719 21289 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 15:47:01.751446 27695 raft_consensus.cc:504] T d3e94b7e50754a81b3d72a07476cb7b9 P 61e69695258f48baa984c9d395e8ce3f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:01.740156 27497 raft_consensus.cc:482] T 538a228ddfb54bc6a36528d5ff07597b P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:01.753016 26399 raft_consensus.cc:504] T 0c29937a80724ddb9d07a27534a146f3 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:47:01.729135 27734 log.cc:924] Time spent T 0acc47ce21c3422698d1ef7277c3ed63 P 65c36d164b0540f099d592f38307ee6c: Append to log took a long time: real 0.164s user 0.000s sys 0.001s
I20260809 15:47:01.717422 27731 raft_consensus.cc:2781] T ba56725e42f74a1baa4c04d9660727f9 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Leader election won for term 5
W20260809 15:47:01.732137 27738 log.cc:924] Time spent T 907bacfa79d04a55a68c8dbe60056462 P 61e69695258f48baa984c9d395e8ce3f: Append to log took a long time: real 0.075s user 0.001s sys 0.000s
I20260809 15:47:01.717541 27682 raft_consensus.cc:482] T 0a51c0baaa6e432997c9d3552af4f254 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Starting leader election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
I20260809 15:47:01.726672 27394 raft_consensus.cc:482] T 6ae14a74a26f434db08c652e097385c8 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:47:01.692948 27081 consensus_queue.cc:1046] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.050s
I20260809 15:47:01.777868 27512 leader_election.cc:288] T f9e7cadcd24d45fcbfa84209225e8f2d P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:47:01.818910 21289 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 15:47:01.804811 27659 raft_consensus.cc:2726] T 3d880de657e544468c82157769622a26 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 15:47:01.742995 27316 raft_consensus.cc:504] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:01.781391 27404 leader_election.cc:288] T df20671a7b874dbb98c0ef28371c1308 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 6 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:47:01.596983 27639 consensus_queue.cc:237] T 07fcabb569414bddbd39df6cedd24e64 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:01.729867 27661 leader_election.cc:288] T cee02fff2c9c48a19f43eb358a89e7b0 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 7 election: Requested vote from peers 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:01.717648 27480 raft_consensus.cc:504] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
W20260809 15:47:01.777887 27347 consensus_meta.cc:287] T ba56725e42f74a1baa4c04d9660727f9 P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent flushing consensus metadata: real 0.620s user 0.007s sys 0.002s
W20260809 15:47:01.809995 24410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 107630us
I20260809 15:47:01.818043 27761 raft_consensus.cc:482] T 538a228ddfb54bc6a36528d5ff07597b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
W20260809 15:47:01.779430 27744 log.cc:924] Time spent T 763e58d6ba644eea830a6c1664d39c9b P 65c36d164b0540f099d592f38307ee6c: Append to log took a long time: real 0.083s user 0.002s sys 0.000s
I20260809 15:47:01.818259 27682 raft_consensus.cc:3037] T 0a51c0baaa6e432997c9d3552af4f254 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Advancing to term 5
I20260809 15:47:01.815824 24453 raft_consensus.cc:2379] T 4bcac4b24e7a4448abdc934bba5a624e P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d in current term 3: Already voted for candidate 65c36d164b0540f099d592f38307ee6c in this term.
I20260809 15:47:01.811465 27488 raft_consensus.cc:3037] T 953eea268e17491ca76f3a555860a512 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Advancing to term 7
I20260809 15:47:01.805395 27518 raft_consensus.cc:482] T f9e7cadcd24d45fcbfa84209225e8f2d P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:01.860790 23394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "297c82ed906d4b48b184985eed3de601" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:47:01.817844 27513 consensus_queue.cc:237] T 4bcac4b24e7a4448abdc934bba5a624e P 65c36d164b0540f099d592f38307ee6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:01.910849 23394 raft_consensus.cc:3037] T 297c82ed906d4b48b184985eed3de601 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Advancing to term 8
I20260809 15:47:01.793942 27556 raft_consensus.cc:504] T 0a51c0baaa6e432997c9d3552af4f254 P 61e69695258f48baa984c9d395e8ce3f [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:01.817649 27394 raft_consensus.cc:3037] T 6ae14a74a26f434db08c652e097385c8 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Advancing to term 7
I20260809 15:47:01.809065 27497 raft_consensus.cc:504] T 538a228ddfb54bc6a36528d5ff07597b P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:01.869968 23393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df20671a7b874dbb98c0ef28371c1308" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:47:01.786711 27395 raft_consensus.cc:3037] T 907bacfa79d04a55a68c8dbe60056462 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Advancing to term 7
W20260809 15:47:01.872453 27740 log.cc:924] Time spent T 563daf42d7784db195ea99c3cb21b370 P 67b3aeadfa7a432abe8af4d292e04ab9: Append to log took a long time: real 0.199s user 0.001s sys 0.001s
I20260809 15:47:01.877005 23399 raft_consensus.cc:1261] T a3622f6e3e3641cea1960e3a31498bb7 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260809 15:47:01.880342 23404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9e7cadcd24d45fcbfa84209225e8f2d" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:01.863044 27669 consensus_queue.cc:237] T df20671a7b874dbb98c0ef28371c1308 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:01.886250 27792 ts_tablet_manager.cc:927] T 4b30f28311964a0b9149a50495e83d8f P 61e69695258f48baa984c9d395e8ce3f: Initiating tablet copy from peer c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:47:01.886422 23529 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56766 (request call id 1993) took 1001 ms. Trace:
I20260809 15:47:01.886766 27267 leader_election.cc:288] T 538a228ddfb54bc6a36528d5ff07597b P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:47:01.861331 27491 consensus_queue.cc:237] T d1a134ba33ea4076bbaac0ae460c17ba P 65c36d164b0540f099d592f38307ee6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:01.888258 24452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c48a3049897452383257aa757834fb5" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:47:01.888608 27316 leader_election.cc:288] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:01.890053 27480 leader_election.cc:288] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:01.890497 24453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3aa6a9a52804df0ba9b0ca3c82f38c3" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:47:01.891948 26399 leader_election.cc:288] T 0c29937a80724ddb9d07a27534a146f3 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 7 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
W20260809 15:47:01.900305 27755 log.cc:924] Time spent T e14d118b2eaa42aa8a7cd4025aba7128 P 67b3aeadfa7a432abe8af4d292e04ab9: Append to log took a long time: real 0.209s user 0.000s sys 0.001s
I20260809 15:47:01.900642 23400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3aa6a9a52804df0ba9b0ca3c82f38c3" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:01.928772 23400 raft_consensus.cc:2454] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 5.
I20260809 15:47:01.894639 27761 raft_consensus.cc:504] T 538a228ddfb54bc6a36528d5ff07597b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:01.912668 24468 raft_consensus.cc:1261] T 3d880de657e544468c82157769622a26 P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260809 15:47:01.915292 23393 raft_consensus.cc:2379] T df20671a7b874dbb98c0ef28371c1308 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c in current term 6: Already voted for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in this term.
I20260809 15:47:01.918536 23399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3e94b7e50754a81b3d72a07476cb7b9" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:01.931823 23280 raft_consensus.cc:2454] T 2db057fd10e149a9895dff6ef8c9f038 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 6.
I20260809 15:47:01.917083 23404 raft_consensus.cc:2454] T f9e7cadcd24d45fcbfa84209225e8f2d P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 6.
I20260809 15:47:01.877846 27347 raft_consensus.cc:504] T ba56725e42f74a1baa4c04d9660727f9 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:01.931842 24410 leader_election.cc:302] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:47:01.916721 27589 raft_consensus.cc:3037] T 30c1740ef6874df0a332fbd6fd4b03ce P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Advancing to term 6
I20260809 15:47:01.919941 23529 rpcz_store.cc:276] 0809 15:47:00.884970 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:01.292401 (+407431us) service_pool.cc:227] Handling call
0809 15:47:01.292658 (+ 257us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 15:47:01.292858 (+ 200us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:47:01.292880 (+ 22us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 15:47:01.293272 (+ 392us) log.cc:842] Serialized 26 byte log entry
0809 15:47:01.351888 (+ 58616us) raft_consensus.cc:1631] Marking committed up to 1
0809 15:47:01.351935 (+ 47us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:47:01.351991 (+ 56us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 15:47:01.886236 (+534245us) raft_consensus.cc:1671] finished
0809 15:47:01.886255 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:47:01.886370 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":57248,"threads_started":1,"wal-append.queue_time_us":153947}
I20260809 15:47:01.936393 23394 raft_consensus.cc:2454] T 297c82ed906d4b48b184985eed3de601 P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 8.
I20260809 15:47:01.945326 23276 raft_consensus.cc:2379] T ba56725e42f74a1baa4c04d9660727f9 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c in current term 5: Already voted for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in this term.
I20260809 15:47:01.938089 24453 raft_consensus.cc:2454] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 5.
W20260809 15:47:01.947229 27743 log.cc:924] Time spent T 0c48a3049897452383257aa757834fb5 P 61e69695258f48baa984c9d395e8ce3f: Append to log took a long time: real 0.067s user 0.000s sys 0.001s
I20260809 15:47:01.950407 23280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "297c82ed906d4b48b184985eed3de601" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:47:01.951030 23280 raft_consensus.cc:3037] T 297c82ed906d4b48b184985eed3de601 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Advancing to term 8
I20260809 15:47:01.952090 23345 leader_election.cc:302] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:47:01.945017 27677 raft_consensus.cc:482] T 00a1881267a14ba8b78c81040f17a014 P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:01.954376 27677 raft_consensus.cc:504] T 00a1881267a14ba8b78c81040f17a014 P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:01.956434 27677 leader_election.cc:288] T 00a1881267a14ba8b78c81040f17a014 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:01.956996 27677 raft_consensus.cc:2781] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260809 15:47:01.957322 27677 raft_consensus.cc:482] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:47:01.961265 27497 leader_election.cc:288] T 538a228ddfb54bc6a36528d5ff07597b P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:01.962037 23273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c48a3049897452383257aa757834fb5" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:01.962703 23273 raft_consensus.cc:2454] T 0c48a3049897452383257aa757834fb5 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 4.
I20260809 15:47:01.951306 27480 raft_consensus.cc:2781] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260809 15:47:01.964030 23282 raft_consensus.cc:1261] T ae6d2c08a7a94f58ac8dd698e3fd456c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260809 15:47:01.965551 27497 raft_consensus.cc:482] T d3be1e356b254d22b995be32f1c12f6f P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:01.966013 27497 raft_consensus.cc:504] T d3be1e356b254d22b995be32f1c12f6f P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:01.968443 27657 raft_consensus.cc:504] T c0741ab44d99439abc3aa3bdd27ee160 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:01.968889 24463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3be1e356b254d22b995be32f1c12f6f" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:47:01.969568 24463 raft_consensus.cc:2454] T d3be1e356b254d22b995be32f1c12f6f P 61e69695258f48baa984c9d395e8ce3f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 7.
I20260809 15:47:01.970458 27657 leader_election.cc:288] T c0741ab44d99439abc3aa3bdd27ee160 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 6 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:01.970633 23345 leader_election.cc:302] T d3be1e356b254d22b995be32f1c12f6f P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:47:01.945461 27695 leader_election.cc:288] T d3e94b7e50754a81b3d72a07476cb7b9 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:01.943758 27316 raft_consensus.cc:482] T 472816994d814cb198b69e5248697440 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
I20260809 15:47:01.971812 27316 raft_consensus.cc:504] T 472816994d814cb198b69e5248697440 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:01.971464 27493 raft_consensus.cc:2781] T d3be1e356b254d22b995be32f1c12f6f P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260809 15:47:01.973809 27316 leader_election.cc:288] T 472816994d814cb198b69e5248697440 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:01.974092 27493 raft_consensus.cc:482] T d3be1e356b254d22b995be32f1c12f6f P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:47:01.974473 27493 raft_consensus.cc:3037] T d3be1e356b254d22b995be32f1c12f6f P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Advancing to term 8
I20260809 15:47:01.981825 23280 raft_consensus.cc:2454] T 297c82ed906d4b48b184985eed3de601 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 8.
I20260809 15:47:01.989744 23283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00a1881267a14ba8b78c81040f17a014" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:01.990403 23283 raft_consensus.cc:2454] T 00a1881267a14ba8b78c81040f17a014 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 8.
I20260809 15:47:01.991089 27659 raft_consensus.cc:482] T 0c29937a80724ddb9d07a27534a146f3 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:47:01.991781 27659 raft_consensus.cc:504] T 0c29937a80724ddb9d07a27534a146f3 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:01.992650 27792 tablet_copy_client.cc:316] T 4b30f28311964a0b9149a50495e83d8f P 61e69695258f48baa984c9d395e8ce3f: tablet copy: Beginning tablet copy session from remote peer at address 127.20.202.66:36263
I20260809 15:47:01.993685 27659 leader_election.cc:288] T 0c29937a80724ddb9d07a27534a146f3 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:01.994550 27796 raft_consensus.cc:482] T 6ae14a74a26f434db08c652e097385c8 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:01.995082 27796 raft_consensus.cc:504] T 6ae14a74a26f434db08c652e097385c8 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:01.995724 23424 tablet_copy_service.cc:140] P c0f38c1908414566aa9115bb2cbdd85d: Received BeginTabletCopySession request for tablet 4b30f28311964a0b9149a50495e83d8f from peer 61e69695258f48baa984c9d395e8ce3f ({username='slave'} at 127.0.0.1:52856)
I20260809 15:47:01.996351 23424 tablet_copy_service.cc:161] P c0f38c1908414566aa9115bb2cbdd85d: Beginning new tablet copy session on tablet 4b30f28311964a0b9149a50495e83d8f from peer 61e69695258f48baa984c9d395e8ce3f at {username='slave'} at 127.0.0.1:52856: session id = 61e69695258f48baa984c9d395e8ce3f-4b30f28311964a0b9149a50495e83d8f
I20260809 15:47:01.999684 24463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472816994d814cb198b69e5248697440" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:47:01.999919 27556 leader_election.cc:288] T 0a51c0baaa6e432997c9d3552af4f254 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 6 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:47:01.984175 27497 leader_election.cc:288] T d3be1e356b254d22b995be32f1c12f6f P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:02.001634 23392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a51c0baaa6e432997c9d3552af4f254" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:47:02.002094 27682 raft_consensus.cc:504] T 0a51c0baaa6e432997c9d3552af4f254 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:02.003085 23392 raft_consensus.cc:3037] T 0a51c0baaa6e432997c9d3552af4f254 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Advancing to term 6
I20260809 15:47:02.007364 27796 leader_election.cc:288] T 6ae14a74a26f434db08c652e097385c8 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:02.009037 23273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e17dcb5efe45b8b6a8c42da03b4819" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:47:02.009586 23273 raft_consensus.cc:3037] T 28e17dcb5efe45b8b6a8c42da03b4819 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Advancing to term 8
I20260809 15:47:02.010250 23283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a51c0baaa6e432997c9d3552af4f254" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:47:02.010772 23283 raft_consensus.cc:3037] T 0a51c0baaa6e432997c9d3552af4f254 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Advancing to term 6
I20260809 15:47:02.010705 24463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a51c0baaa6e432997c9d3552af4f254" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f"
I20260809 15:47:02.009584 23392 raft_consensus.cc:2454] T 0a51c0baaa6e432997c9d3552af4f254 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 6.
I20260809 15:47:02.013711 24410 leader_election.cc:302] T 0a51c0baaa6e432997c9d3552af4f254 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:47:02.009582 23282 raft_consensus.cc:1261] T 0c48a3049897452383257aa757834fb5 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Refusing update from remote peer 61e69695258f48baa984c9d395e8ce3f: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260809 15:47:02.011670 24463 raft_consensus.cc:2354] T 0a51c0baaa6e432997c9d3552af4f254 P 61e69695258f48baa984c9d395e8ce3f [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d for earlier term 5. Current term is 6.
I20260809 15:47:02.016381 27394 raft_consensus.cc:504] T 6ae14a74a26f434db08c652e097385c8 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:02.018272 27394 leader_election.cc:288] T 6ae14a74a26f434db08c652e097385c8 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 7 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:47:02.020083 23345 leader_election.cc:398] T 0a51c0baaa6e432997c9d3552af4f254 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 election: Vote denied by peer 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689) with higher term. Message: Invalid argument: T 0a51c0baaa6e432997c9d3552af4f254 P 61e69695258f48baa984c9d395e8ce3f [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d for earlier term 5. Current term is 6.
I20260809 15:47:02.020398 23345 leader_election.cc:401] T 0a51c0baaa6e432997c9d3552af4f254 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
I20260809 15:47:01.957634 27677 raft_consensus.cc:3037] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Advancing to term 6
I20260809 15:47:01.914645 27518 raft_consensus.cc:504] T f9e7cadcd24d45fcbfa84209225e8f2d P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:02.025776 23351 leader_election.cc:302] T 00a1881267a14ba8b78c81040f17a014 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:47:02.027107 27797 raft_consensus.cc:2726] T 0a51c0baaa6e432997c9d3552af4f254 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689) with higher term. Message: Invalid argument: T 0a51c0baaa6e432997c9d3552af4f254 P 61e69695258f48baa984c9d395e8ce3f [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d for earlier term 5. Current term is 6.
I20260809 15:47:02.029850 27797 raft_consensus.cc:2781] T 00a1881267a14ba8b78c81040f17a014 P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260809 15:47:02.030112 23424 tablet_copy_source_session.cc:205] T 4b30f28311964a0b9149a50495e83d8f P c0f38c1908414566aa9115bb2cbdd85d: Tablet Copy: opened 0 blocks and 1 log segments
W20260809 15:47:02.030859 23223 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.68:34689, user_credentials={real_user=slave}} blocked reactor thread for 569220us
I20260809 15:47:02.028502 23286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "538a228ddfb54bc6a36528d5ff07597b" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:02.032572 23223 leader_election.cc:302] T 563daf42d7784db195ea99c3cb21b370 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:02.028738 23400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ae14a74a26f434db08c652e097385c8" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:02.033324 23400 raft_consensus.cc:2454] T 6ae14a74a26f434db08c652e097385c8 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 6.
I20260809 15:47:02.036558 23283 raft_consensus.cc:2454] T 0a51c0baaa6e432997c9d3552af4f254 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 6.
I20260809 15:47:02.037482 23283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a51c0baaa6e432997c9d3552af4f254" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:47:02.038144 23283 raft_consensus.cc:2354] T 0a51c0baaa6e432997c9d3552af4f254 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d for earlier term 5. Current term is 6.
I20260809 15:47:02.023677 23273 raft_consensus.cc:2454] T 28e17dcb5efe45b8b6a8c42da03b4819 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 8.
I20260809 15:47:02.040158 23282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3be1e356b254d22b995be32f1c12f6f" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:02.040364 23351 leader_election.cc:398] T 0a51c0baaa6e432997c9d3552af4f254 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 election: Vote denied by peer 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459) with higher term. Message: Invalid argument: T 0a51c0baaa6e432997c9d3552af4f254 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d for earlier term 5. Current term is 6.
I20260809 15:47:02.040853 23228 leader_election.cc:302] T 6ae14a74a26f434db08c652e097385c8 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:47:02.040944 23351 leader_election.cc:302] T 28e17dcb5efe45b8b6a8c42da03b4819 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:47:02.030187 27797 raft_consensus.cc:482] T 00a1881267a14ba8b78c81040f17a014 P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:47:02.042023 27797 raft_consensus.cc:3037] T 00a1881267a14ba8b78c81040f17a014 P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Advancing to term 9
I20260809 15:47:02.032728 23286 raft_consensus.cc:2454] T 538a228ddfb54bc6a36528d5ff07597b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 4.
I20260809 15:47:02.042804 23392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c29937a80724ddb9d07a27534a146f3" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:02.043536 23392 raft_consensus.cc:2379] T 0c29937a80724ddb9d07a27534a146f3 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 in current term 6: Already voted for candidate c0f38c1908414566aa9115bb2cbdd85d in this term.
I20260809 15:47:02.009752 27682 leader_election.cc:288] T 0a51c0baaa6e432997c9d3552af4f254 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:02.045658 23351 leader_election.cc:302] T 538a228ddfb54bc6a36528d5ff07597b P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:47:02.024745 27217 raft_consensus.cc:482] T 30c1740ef6874df0a332fbd6fd4b03ce P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:02.050292 27781 raft_consensus.cc:2781] T 6ae14a74a26f434db08c652e097385c8 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260809 15:47:02.050707 27781 raft_consensus.cc:482] T 6ae14a74a26f434db08c652e097385c8 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:47:02.051012 27781 raft_consensus.cc:3037] T 6ae14a74a26f434db08c652e097385c8 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Advancing to term 7
I20260809 15:47:02.055389 27217 raft_consensus.cc:504] T 30c1740ef6874df0a332fbd6fd4b03ce P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:02.057616 27217 leader_election.cc:288] T 30c1740ef6874df0a332fbd6fd4b03ce P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:02.058643 27686 raft_consensus.cc:2781] T 28e17dcb5efe45b8b6a8c42da03b4819 P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Leader election won for term 8
I20260809 15:47:02.059160 27686 raft_consensus.cc:686] T 28e17dcb5efe45b8b6a8c42da03b4819 P c0f38c1908414566aa9115bb2cbdd85d [term 8 LEADER]: Becoming Leader. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:47:02.059955 27686 consensus_queue.cc:237] T 28e17dcb5efe45b8b6a8c42da03b4819 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:02.060436 27493 raft_consensus.cc:504] T d3be1e356b254d22b995be32f1c12f6f P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:02.039723 27659 raft_consensus.cc:2726] T 563daf42d7784db195ea99c3cb21b370 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260809 15:47:02.072178 23392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30c1740ef6874df0a332fbd6fd4b03ce" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:02.027487 27497 consensus_queue.cc:1035] T ae6d2c08a7a94f58ac8dd698e3fd456c P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.042s
I20260809 15:47:02.047326 27658 raft_consensus.cc:482] T 73f6f44c259547f1b05a3a434616fa54 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:47:02.078329 27787 raft_consensus.cc:2781] T 0a51c0baaa6e432997c9d3552af4f254 P 61e69695258f48baa984c9d395e8ce3f [term 6 FOLLOWER]: Leader election won for term 6
I20260809 15:47:02.069947 27428 raft_consensus.cc:2781] T 538a228ddfb54bc6a36528d5ff07597b P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260809 15:47:02.057464 27790 consensus_queue.cc:1035] T 0c48a3049897452383257aa757834fb5 P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.034s
I20260809 15:47:02.081363 23286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3be1e356b254d22b995be32f1c12f6f" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:47:02.070433 24463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee02fff2c9c48a19f43eb358a89e7b0" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f"
I20260809 15:47:02.085041 23282 raft_consensus.cc:1261] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Refusing update from remote peer 61e69695258f48baa984c9d395e8ce3f: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260809 15:47:02.087821 27633 raft_consensus.cc:3032] T cef3e7277d7d4760824a3436ed4a847d P 65c36d164b0540f099d592f38307ee6c [term 6 LEADER]: Stepping down as leader of term 6
I20260809 15:47:02.088197 27633 raft_consensus.cc:729] T cef3e7277d7d4760824a3436ed4a847d P 65c36d164b0540f099d592f38307ee6c [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
W20260809 15:47:02.090108 23274 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53554 (request call id 1602) took 2191 ms (client timeout 1701 ms). Trace:
W20260809 15:47:02.090509 23274 rpcz_store.cc:269] 0809 15:46:59.898519 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:59.898642 (+ 123us) service_pool.cc:227] Handling call
0809 15:47:02.089877 (+2191235us) spinlock_profiling.cc:243] Waited 1.14 s on lock 0x7b54003a93d4. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e92242ae 00007f34ee161c27 00007f34e7849c04 00007f34e7849b92 00007f34e7849ac1 00007f34e7849a3c 00007f34e7848cf2 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:47:02.090014 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2621814016}
I20260809 15:47:02.091118 23270 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53554 (request call id 1635) took 1580 ms. Trace:
I20260809 15:47:02.091476 23270 rpcz_store.cc:276] 0809 15:47:00.510412 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:00.522580 (+ 12168us) service_pool.cc:227] Handling call
0809 15:47:00.522821 (+ 241us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:47:01.437082 (+914261us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:47:02.090890 (+653808us) spinlock_profiling.cc:243] Waited 913 ms on lock 0x7b54002f58d4. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e9220f51 00007f34e921f99e 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772 00007f34e764a94c 00007f34e7649d86
0809 15:47:02.091061 (+ 171us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2100065536}
I20260809 15:47:02.093021 23277 raft_consensus.cc:1261] T 2424fd9bd7f14e508fd106cbfda786a4 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 5 index: 2. Preceding OpId from leader: term: 7 index: 3. (index mismatch)
I20260809 15:47:02.093902 23277 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53554 (request call id 1640) took 1461 ms. Trace:
I20260809 15:47:02.094102 23277 rpcz_store.cc:276] 0809 15:47:00.632256 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:00.632544 (+ 288us) service_pool.cc:227] Handling call
0809 15:47:02.092806 (+1460262us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:47:02.093290 (+ 484us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:47:02.093777 (+ 487us) spinlock_profiling.cc:243] Waited 1.46 s on lock 0x7b54003a93d0. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e921fa09 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:47:02.093861 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3357742848}
I20260809 15:47:02.081734 27428 raft_consensus.cc:482] T 538a228ddfb54bc6a36528d5ff07597b P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:47:02.095629 27428 raft_consensus.cc:3037] T 538a228ddfb54bc6a36528d5ff07597b P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Advancing to term 6
W20260809 15:47:02.096324 27707 log.cc:924] Time spent T 0acc47ce21c3422698d1ef7277c3ed63 P 67b3aeadfa7a432abe8af4d292e04ab9: Append to log took a long time: real 1.227s user 0.008s sys 0.000s
I20260809 15:47:02.098161 23285 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53562 (request call id 1811) took 1563 ms. Trace:
I20260809 15:47:02.098466 23285 rpcz_store.cc:276] 0809 15:47:00.534544 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:00.574061 (+ 39517us) service_pool.cc:227] Handling call
0809 15:47:00.574352 (+ 291us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 15:47:00.574565 (+ 213us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:47:00.574589 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 15:47:00.574980 (+ 391us) log.cc:842] Serialized 26 byte log entry
0809 15:47:00.705307 (+130327us) raft_consensus.cc:1631] Marking committed up to 1
0809 15:47:00.705340 (+ 33us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:47:00.705451 (+ 111us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 15:47:02.095689 (+1390238us) spinlock_profiling.cc:243] Waited 2.27 ms on lock 0x7b44000232cc. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e613cf11 00007f34e620aa95 00007f34e620b4b2 00007f34e8eb3511 00007f34e8eaee1a 00007f34e8ea2bee 00007f34e8ea1f36 00007f34e8ea1b58 00007f34e8eab551 00007f34e8eab509 00007f34e8eab499
0809 15:47:02.097891 (+ 2202us) raft_consensus.cc:1671] finished
0809 15:47:02.097909 (+ 18us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:47:02.098107 (+ 198us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5231488,"thread_start_us":129936,"threads_started":1,"wal-append.queue_time_us":275383}
W20260809 15:47:02.100721 27710 log.cc:924] Time spent T cee02fff2c9c48a19f43eb358a89e7b0 P 65c36d164b0540f099d592f38307ee6c: Append to log took a long time: real 0.987s user 0.001s sys 0.001s
I20260809 15:47:02.080868 27658 raft_consensus.cc:504] T 73f6f44c259547f1b05a3a434616fa54 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:02.105389 27658 leader_election.cc:288] T 73f6f44c259547f1b05a3a434616fa54 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:02.105507 27428 raft_consensus.cc:504] T 538a228ddfb54bc6a36528d5ff07597b P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:02.110387 27347 leader_election.cc:288] T ba56725e42f74a1baa4c04d9660727f9 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:02.114497 23285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "538a228ddfb54bc6a36528d5ff07597b" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:47:02.115093 23285 raft_consensus.cc:3037] T 538a228ddfb54bc6a36528d5ff07597b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Advancing to term 6
I20260809 15:47:02.078887 27787 raft_consensus.cc:686] T 0a51c0baaa6e432997c9d3552af4f254 P 61e69695258f48baa984c9d395e8ce3f [term 6 LEADER]: Becoming Leader. State: Replica: 61e69695258f48baa984c9d395e8ce3f, State: Running, Role: LEADER
I20260809 15:47:02.119378 27787 consensus_queue.cc:237] T 0a51c0baaa6e432997c9d3552af4f254 P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
W20260809 15:47:02.120376 27720 log.cc:924] Time spent T a88421dba80d4d3388aa5eb62e23853b P 65c36d164b0540f099d592f38307ee6c: Append to log took a long time: real 0.897s user 0.003s sys 0.000s
I20260809 15:47:02.120638 27790 consensus_queue.cc:1035] T 907bacfa79d04a55a68c8dbe60056462 P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260809 15:47:02.125756 23228 leader_election.cc:398] T 28e17dcb5efe45b8b6a8c42da03b4819 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 7 election: Vote denied by peer c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263) with higher term. Message: Invalid argument: T 28e17dcb5efe45b8b6a8c42da03b4819 P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 7. Current term is 8.
I20260809 15:47:02.126067 23228 leader_election.cc:401] T 28e17dcb5efe45b8b6a8c42da03b4819 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 7 election: Cancelling election due to peer responding with higher term
I20260809 15:47:02.127207 23394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73f6f44c259547f1b05a3a434616fa54" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
W20260809 15:47:02.127806 27719 log.cc:924] Time spent T 05ad7d8fdca64e3da527280ca5be9280 P 65c36d164b0540f099d592f38307ee6c: Append to log took a long time: real 0.908s user 0.000s sys 0.002s
I20260809 15:47:02.129611 23392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba56725e42f74a1baa4c04d9660727f9" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:47:02.130148 23392 raft_consensus.cc:2379] T ba56725e42f74a1baa4c04d9660727f9 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 in current term 5: Already voted for candidate 65c36d164b0540f099d592f38307ee6c in this term.
I20260809 15:47:02.131116 23521 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56770 (request call id 1369) took 1334 ms. Trace:
I20260809 15:47:02.131505 23521 rpcz_store.cc:276] 0809 15:47:00.796799 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:00.797749 (+ 950us) service_pool.cc:227] Handling call
0809 15:47:00.798050 (+ 301us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 15:47:00.817783 (+ 19733us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:47:00.817811 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 15:47:00.830862 (+ 13051us) log.cc:842] Serialized 26 byte log entry
0809 15:47:01.113113 (+282251us) raft_consensus.cc:1631] Marking committed up to 3
0809 15:47:01.113813 (+ 700us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:47:01.113874 (+ 61us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 15:47:02.127176 (+1013302us) spinlock_profiling.cc:243] Waited 13.7 ms on lock 0x7b44000232cc. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e613cf11 00007f34e620aa95 00007f34e620b4b2 00007f34e8eb3511 00007f34e8eaee1a 00007f34e8ea2bee 00007f34e8ea1f36 00007f34e8ea1b58 00007f34e8eab551 00007f34e8eab509 00007f34e8eab499
0809 15:47:02.128851 (+ 1675us) raft_consensus.cc:1671] finished
0809 15:47:02.128867 (+ 16us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:47:02.131057 (+ 2190us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":35011,"raft.queue_time_us":20019,"raft.run_cpu_time_us":90,"raft.run_wall_time_us":6667,"spinlock_wait_cycles":31481600,"thread_start_us":281941,"threads_started":1,"wal-append.queue_time_us":374094}
I20260809 15:47:02.127972 27792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b30f28311964a0b9149a50495e83d8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:47:02.133685 27797 raft_consensus.cc:504] T 00a1881267a14ba8b78c81040f17a014 P c0f38c1908414566aa9115bb2cbdd85d [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:47:02.151237 27807 log.cc:924] Time spent T 4bcac4b24e7a4448abdc934bba5a624e P 65c36d164b0540f099d592f38307ee6c: Append to log took a long time: real 0.070s user 0.001s sys 0.000s
I20260809 15:47:02.075788 27395 raft_consensus.cc:504] T 907bacfa79d04a55a68c8dbe60056462 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:02.163014 23396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9e7cadcd24d45fcbfa84209225e8f2d" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:02.170392 23396 raft_consensus.cc:2454] T f9e7cadcd24d45fcbfa84209225e8f2d P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 6.
I20260809 15:47:02.132187 27347 raft_consensus.cc:2726] T 28e17dcb5efe45b8b6a8c42da03b4819 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Leader election lost for term 7. Reason: Vote denied by peer c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263) with higher term. Message: Invalid argument: T 28e17dcb5efe45b8b6a8c42da03b4819 P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 7. Current term is 8.
I20260809 15:47:02.174005 23472 leader_election.cc:302] T 2db057fd10e149a9895dff6ef8c9f038 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:47:02.165406 27518 leader_election.cc:288] T f9e7cadcd24d45fcbfa84209225e8f2d P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:02.172576 23228 leader_election.cc:302] T f9e7cadcd24d45fcbfa84209225e8f2d P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:47:02.177224 23285 raft_consensus.cc:2454] T 538a228ddfb54bc6a36528d5ff07597b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 6.
I20260809 15:47:02.178745 23351 leader_election.cc:302] T 538a228ddfb54bc6a36528d5ff07597b P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:47:02.155388 27493 leader_election.cc:288] T d3be1e356b254d22b995be32f1c12f6f P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 8 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:02.181576 23395 raft_consensus.cc:1226] T 907bacfa79d04a55a68c8dbe60056462 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Rejecting Update request from peer 61e69695258f48baa984c9d395e8ce3f for earlier term 6. Current term is 7. Ops: []
I20260809 15:47:02.170228 27731 raft_consensus.cc:686] T ba56725e42f74a1baa4c04d9660727f9 P 65c36d164b0540f099d592f38307ee6c [term 5 LEADER]: Becoming Leader. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:47:02.198263 24463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3be1e356b254d22b995be32f1c12f6f" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f"
I20260809 15:47:02.180253 27796 raft_consensus.cc:2781] T f9e7cadcd24d45fcbfa84209225e8f2d P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260809 15:47:02.216459 27704 raft_consensus.cc:2932] T 4b30f28311964a0b9149a50495e83d8f P c0f38c1908414566aa9115bb2cbdd85d [term 3 LEADER]: Committing config change with OpId 3.4: config changed from index -1 to 4, NON_VOTER 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } attrs { replace: true } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } } }
I20260809 15:47:02.219776 23521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e17dcb5efe45b8b6a8c42da03b4819" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:47:02.220335 23521 raft_consensus.cc:3037] T 28e17dcb5efe45b8b6a8c42da03b4819 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Advancing to term 8
I20260809 15:47:02.220630 23520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "538a228ddfb54bc6a36528d5ff07597b" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:47:02.221565 23519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "538a228ddfb54bc6a36528d5ff07597b" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:02.222141 24448 raft_consensus.cc:1261] T d3e94b7e50754a81b3d72a07476cb7b9 P 61e69695258f48baa984c9d395e8ce3f [term 7 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260809 15:47:02.223052 24453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0741ab44d99439abc3aa3bdd27ee160" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f"
I20260809 15:47:02.223291 27395 leader_election.cc:288] T 907bacfa79d04a55a68c8dbe60056462 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 7 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:02.223686 24453 raft_consensus.cc:3037] T c0741ab44d99439abc3aa3bdd27ee160 P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Advancing to term 6
I20260809 15:47:02.224308 27780 raft_consensus.cc:482] T a88421dba80d4d3388aa5eb62e23853b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:02.224828 27780 raft_consensus.cc:504] T a88421dba80d4d3388aa5eb62e23853b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:02.224426 23353 leader_election.cc:302] T 4bcac4b24e7a4448abdc934bba5a624e P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c
I20260809 15:47:02.226377 27497 raft_consensus.cc:3037] T 4bcac4b24e7a4448abdc934bba5a624e P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:47:02.221251 23520 raft_consensus.cc:3037] T 538a228ddfb54bc6a36528d5ff07597b P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Advancing to term 6
I20260809 15:47:02.228500 27480 raft_consensus.cc:482] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:47:02.228896 27480 raft_consensus.cc:3037] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Advancing to term 6
I20260809 15:47:02.229663 23276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "907bacfa79d04a55a68c8dbe60056462" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:47:02.224833 27633 consensus_queue.cc:260] T cef3e7277d7d4760824a3436ed4a847d P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:02.235200 27633 raft_consensus.cc:3037] T cef3e7277d7d4760824a3436ed4a847d P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Advancing to term 7
I20260809 15:47:02.211896 27796 raft_consensus.cc:482] T f9e7cadcd24d45fcbfa84209225e8f2d P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:47:02.235816 27796 raft_consensus.cc:3037] T f9e7cadcd24d45fcbfa84209225e8f2d P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Advancing to term 7
I20260809 15:47:02.244921 27507 raft_consensus.cc:482] T 30c1740ef6874df0a332fbd6fd4b03ce P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:02.245389 27507 raft_consensus.cc:504] T 30c1740ef6874df0a332fbd6fd4b03ce P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:02.246805 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee02fff2c9c48a19f43eb358a89e7b0" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:47:02.247560 27772 raft_consensus.cc:482] T 3924b648461e463b9d7925e5671d68ae P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
I20260809 15:47:02.229125 23280 raft_consensus.cc:2932] T 4b30f28311964a0b9149a50495e83d8f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index -1 to 4, NON_VOTER 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } attrs { replace: true } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } } }
I20260809 15:47:02.248078 27772 raft_consensus.cc:504] T 3924b648461e463b9d7925e5671d68ae P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:02.249327 23519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4564dfc6d4094c1b9240b367ccecf3b1" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:47:02.250032 27772 leader_election.cc:288] T 3924b648461e463b9d7925e5671d68ae P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
W20260809 15:47:02.250568 23472 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 76638.3us
I20260809 15:47:02.251121 27797 leader_election.cc:288] T 00a1881267a14ba8b78c81040f17a014 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 9 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:02.251593 27053 raft_consensus.cc:2781] T 2db057fd10e149a9895dff6ef8c9f038 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Leader election won for term 6
I20260809 15:47:02.252518 27507 leader_election.cc:288] T 30c1740ef6874df0a332fbd6fd4b03ce P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:47:02.252213 27780 leader_election.cc:288] T a88421dba80d4d3388aa5eb62e23853b P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:02.247866 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3e94b7e50754a81b3d72a07476cb7b9" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:02.254181 27731 consensus_queue.cc:237] T ba56725e42f74a1baa4c04d9660727f9 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:02.255486 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9e7cadcd24d45fcbfa84209225e8f2d" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:02.256016 23527 raft_consensus.cc:2454] T f9e7cadcd24d45fcbfa84209225e8f2d P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 6.
I20260809 15:47:02.257766 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ae14a74a26f434db08c652e097385c8" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:02.259912 23527 raft_consensus.cc:2379] T 6ae14a74a26f434db08c652e097385c8 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 in current term 7: Already voted for candidate 65c36d164b0540f099d592f38307ee6c in this term.
I20260809 15:47:02.256404 23392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a88421dba80d4d3388aa5eb62e23853b" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:02.197552 27682 raft_consensus.cc:2781] T 538a228ddfb54bc6a36528d5ff07597b P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Leader election won for term 6
I20260809 15:47:02.262820 23468 leader_election.cc:302] T 7660dfa2ef694927b8eda4100a2bcd9b P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:02.221236 23522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00a1881267a14ba8b78c81040f17a014" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:02.203773 27677 raft_consensus.cc:504] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:02.275223 27677 leader_election.cc:288] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 6 election: Requested vote from peers 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:02.268980 23528 consensus_queue.cc:260] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:02.277117 27781 raft_consensus.cc:504] T 6ae14a74a26f434db08c652e097385c8 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:02.278967 27781 leader_election.cc:288] T 6ae14a74a26f434db08c652e097385c8 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 7 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:02.174734 27512 consensus_queue.cc:1035] T a88421dba80d4d3388aa5eb62e23853b P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.105s
I20260809 15:47:02.176239 27428 leader_election.cc:288] T 538a228ddfb54bc6a36528d5ff07597b P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 6 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:02.282636 27497 raft_consensus.cc:2726] T 4bcac4b24e7a4448abdc934bba5a624e P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 15:47:02.283802 27761 leader_election.cc:288] T 538a228ddfb54bc6a36528d5ff07597b P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:02.284385 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3aa6a9a52804df0ba9b0ca3c82f38c3" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f"
I20260809 15:47:02.288834 23528 raft_consensus.cc:3037] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Advancing to term 9
I20260809 15:47:02.289497 24410 leader_election.cc:302] T d3e94b7e50754a81b3d72a07476cb7b9 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:02.206816 24463 raft_consensus.cc:3037] T d3be1e356b254d22b995be32f1c12f6f P 61e69695258f48baa984c9d395e8ce3f [term 7 FOLLOWER]: Advancing to term 8
I20260809 15:47:02.220571 23518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472816994d814cb198b69e5248697440" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:02.264149 23228 leader_election.cc:302] T cee02fff2c9c48a19f43eb358a89e7b0 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c
I20260809 15:47:02.258569 23520 raft_consensus.cc:2454] T 538a228ddfb54bc6a36528d5ff07597b P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 6.
I20260809 15:47:02.294657 23520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a88421dba80d4d3388aa5eb62e23853b" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:02.261451 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3aa6a9a52804df0ba9b0ca3c82f38c3" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
W20260809 15:47:02.234804 27760 log.cc:924] Time spent T c5c2e425a74b4e64ae70b5def750434c P 65c36d164b0540f099d592f38307ee6c: Append to log took a long time: real 0.378s user 0.002s sys 0.000s
I20260809 15:47:02.272221 23522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c29937a80724ddb9d07a27534a146f3" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:02.298416 23228 leader_election.cc:302] T a88421dba80d4d3388aa5eb62e23853b P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:02.264427 23276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00a1881267a14ba8b78c81040f17a014" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:47:02.299434 23276 raft_consensus.cc:3037] T 00a1881267a14ba8b78c81040f17a014 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Advancing to term 9
I20260809 15:47:02.273524 24448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "907bacfa79d04a55a68c8dbe60056462" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f"
I20260809 15:47:02.260680 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73f6f44c259547f1b05a3a434616fa54" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:02.301160 23522 raft_consensus.cc:2354] T 0c29937a80724ddb9d07a27534a146f3 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 6. Current term is 7.
I20260809 15:47:02.295243 23526 raft_consensus.cc:2454] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 5.
I20260809 15:47:02.300854 23519 raft_consensus.cc:2454] T 4564dfc6d4094c1b9240b367ccecf3b1 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 6.
I20260809 15:47:02.305668 23522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "538a228ddfb54bc6a36528d5ff07597b" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:02.306962 23522 raft_consensus.cc:2354] T 538a228ddfb54bc6a36528d5ff07597b P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 5. Current term is 6.
I20260809 15:47:02.306834 23396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ae14a74a26f434db08c652e097385c8" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:47:02.291930 27787 raft_consensus.cc:2726] T d3e94b7e50754a81b3d72a07476cb7b9 P 61e69695258f48baa984c9d395e8ce3f [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260809 15:47:02.305074 23470 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 135441us
I20260809 15:47:02.307818 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba56725e42f74a1baa4c04d9660727f9" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:47:02.312201 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30c1740ef6874df0a332fbd6fd4b03ce" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:02.303399 23228 leader_election.cc:302] T 73f6f44c259547f1b05a3a434616fa54 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:02.330618 23527 raft_consensus.cc:2454] T 30c1740ef6874df0a332fbd6fd4b03ce P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 6.
W20260809 15:47:02.330684 23472 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 78349.2us
I20260809 15:47:02.329696 23396 raft_consensus.cc:3037] T 6ae14a74a26f434db08c652e097385c8 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Advancing to term 7
I20260809 15:47:02.311450 27657 consensus_queue.cc:1035] T a3622f6e3e3641cea1960e3a31498bb7 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 15:47:02.335547 23528 raft_consensus.cc:1261] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 65c36d164b0540f099d592f38307ee6c [term 9 FOLLOWER]: Refusing update from remote peer 61e69695258f48baa984c9d395e8ce3f: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 9 index: 8. (term mismatch)
I20260809 15:47:02.331493 23472 leader_election.cc:302] T f9e7cadcd24d45fcbfa84209225e8f2d P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:47:02.331974 23228 leader_election.cc:302] T 4564dfc6d4094c1b9240b367ccecf3b1 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters: c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:02.334302 23276 raft_consensus.cc:2454] T 00a1881267a14ba8b78c81040f17a014 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 9.
I20260809 15:47:02.343693 23528 pending_rounds.cc:76] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 65c36d164b0540f099d592f38307ee6c: Aborting all ops after (but not including) 7
I20260809 15:47:02.304998 27032 raft_consensus.cc:2726] T 7660dfa2ef694927b8eda4100a2bcd9b P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260809 15:47:02.345726 23528 pending_rounds.cc:97] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 65c36d164b0540f099d592f38307ee6c: Aborting uncommitted NO_OP operation due to leader change: 8.8
I20260809 15:47:02.338078 24453 raft_consensus.cc:2454] T c0741ab44d99439abc3aa3bdd27ee160 P 61e69695258f48baa984c9d395e8ce3f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 6.
I20260809 15:47:02.335548 27053 raft_consensus.cc:686] T 2db057fd10e149a9895dff6ef8c9f038 P 65c36d164b0540f099d592f38307ee6c [term 6 LEADER]: Becoming Leader. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:47:02.347827 27053 consensus_queue.cc:237] T 2db057fd10e149a9895dff6ef8c9f038 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:02.335134 27761 raft_consensus.cc:2726] T 73f6f44c259547f1b05a3a434616fa54 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260809 15:47:02.339689 27762 raft_consensus.cc:2726] T a88421dba80d4d3388aa5eb62e23853b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260809 15:47:02.340176 23394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "538a228ddfb54bc6a36528d5ff07597b" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:02.345196 27305 consensus_queue.cc:1035] T a3622f6e3e3641cea1960e3a31498bb7 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.007s
I20260809 15:47:02.343333 23522 raft_consensus.cc:1261] T 0c48a3049897452383257aa757834fb5 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Refusing update from remote peer 61e69695258f48baa984c9d395e8ce3f: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260809 15:47:02.340950 23517 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56766 (request call id 1997) took 1161 ms. Trace:
I20260809 15:47:02.297947 27682 raft_consensus.cc:686] T 538a228ddfb54bc6a36528d5ff07597b P c0f38c1908414566aa9115bb2cbdd85d [term 6 LEADER]: Becoming Leader. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:47:02.344759 23520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3aa6a9a52804df0ba9b0ca3c82f38c3" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:02.355906 23520 raft_consensus.cc:2454] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 5.
I20260809 15:47:02.302888 27781 raft_consensus.cc:2726] T cee02fff2c9c48a19f43eb358a89e7b0 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
W20260809 15:47:02.337838 27737 log.cc:924] Time spent T 00000000000000000000000000000000 P a86a2513b8024a869d80303dcbf44163: Append to log took a long time: real 0.564s user 0.003s sys 0.000s
I20260809 15:47:02.346535 27480 raft_consensus.cc:504] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 61e69695258f48baa984c9d395e8ce3f [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
W20260809 15:47:02.362396 23468 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.68:34689, user_credentials={real_user=slave}} blocked reactor thread for 75425.8us
I20260809 15:47:02.353189 23517 rpcz_store.cc:276] 0809 15:47:01.179956 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:01.316348 (+136392us) service_pool.cc:227] Handling call
0809 15:47:01.316718 (+ 370us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 15:47:01.419560 (+102842us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:47:01.419605 (+ 45us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 15:47:01.430789 (+ 11184us) log.cc:842] Serialized 26 byte log entry
0809 15:47:01.670455 (+239666us) raft_consensus.cc:1631] Marking committed up to 4
0809 15:47:01.693455 (+ 23000us) log.cc:842] Serialized 14 byte log entry
0809 15:47:02.132326 (+438871us) spinlock_profiling.cc:243] Waited 4.43 ms on lock 0x7b18003fc1c0. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34ef1978b1 00007f34ef1a4661 00007f34e626f4ca 00007f34ef950f99 00007f34e6133990 00007f34e62f5864 00007f34e62f5155 00007f34e92183d0 00007f34e922a576 00007f34e923b0d7 00007f34e923b058
0809 15:47:02.132487 (+ 161us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:47:02.132550 (+ 63us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 15:47:02.234171 (+101621us) spinlock_profiling.cc:243] Waited 2.7 ms on lock 0x7b44000232cc. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e613cf11 00007f34e620aa95 00007f34e620b4b2 00007f34e8eb3511 00007f34e8eaee1a 00007f34e8ea2bee 00007f34e8ea1f36 00007f34e8ea1b58 00007f34e8eab551 00007f34e8eab509 00007f34e8eab499
0809 15:47:02.340727 (+106556us) raft_consensus.cc:1671] finished
0809 15:47:02.340746 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:47:02.340884 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":270800,"prepare.queue_time_us":22789,"prepare.run_cpu_time_us":31,"prepare.run_wall_time_us":32,"raft.queue_time_us":496006,"raft.run_cpu_time_us":226,"raft.run_wall_time_us":25468,"spinlock_wait_cycles":16383616,"thread_start_us":147493,"threads_started":1,"wal-append.queue_time_us":343237}
I20260809 15:47:02.358508 27762 raft_consensus.cc:2781] T 4564dfc6d4094c1b9240b367ccecf3b1 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Leader election won for term 6
I20260809 15:47:02.338430 23518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ae14a74a26f434db08c652e097385c8" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:47:02.372864 23518 raft_consensus.cc:2379] T 6ae14a74a26f434db08c652e097385c8 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 in current term 7: Already voted for candidate 65c36d164b0540f099d592f38307ee6c in this term.
I20260809 15:47:02.354703 23228 leader_election.cc:398] T 0c29937a80724ddb9d07a27534a146f3 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 6 pre-election: Vote denied by peer 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553) with higher term. Message: Invalid argument: T 0c29937a80724ddb9d07a27534a146f3 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 6. Current term is 7.
I20260809 15:47:02.344771 23345 leader_election.cc:302] T 907bacfa79d04a55a68c8dbe60056462 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 61e69695258f48baa984c9d395e8ce3f, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:47:02.356365 23351 leader_election.cc:302] T 00a1881267a14ba8b78c81040f17a014 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:47:02.368332 23470 leader_election.cc:302] T 4b30f28311964a0b9149a50495e83d8f P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters: c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:02.374148 23228 leader_election.cc:401] T 0c29937a80724ddb9d07a27534a146f3 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 6 pre-election: Cancelling election due to peer responding with higher term
I20260809 15:47:02.346180 23528 raft_consensus.cc:2866] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 65c36d164b0540f099d592f38307ee6c [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 15:47:02.375308 23228 leader_election.cc:398] T 538a228ddfb54bc6a36528d5ff07597b P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 pre-election: Vote denied by peer 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553) with higher term. Message: Invalid argument: T 538a228ddfb54bc6a36528d5ff07597b P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 5. Current term is 6.
I20260809 15:47:02.375708 23228 leader_election.cc:401] T 538a228ddfb54bc6a36528d5ff07597b P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 pre-election: Cancelling election due to peer responding with higher term
I20260809 15:47:02.375834 23528 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44018 (request call id 273) took 1676 ms. Trace:
W20260809 15:47:02.377978 23472 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 46560.4us
I20260809 15:47:02.379999 23396 raft_consensus.cc:2454] T 6ae14a74a26f434db08c652e097385c8 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 7.
I20260809 15:47:02.352840 27790 consensus_queue.cc:1046] T 907bacfa79d04a55a68c8dbe60056462 P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.016s
I20260809 15:47:02.380270 23472 leader_election.cc:302] T 297c82ed906d4b48b184985eed3de601 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:47:02.379804 23528 rpcz_store.cc:276] 0809 15:47:00.699875 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:00.700103 (+ 228us) service_pool.cc:227] Handling call
0809 15:47:00.700389 (+ 286us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:47:01.435779 (+735390us) spinlock_profiling.cc:243] Waited 67.4 ms on lock 0x7b54002371a0. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e91b0730 00007f34e9222300 00007f34e9223371 00007f34e921fedf 00007f34e921f99e 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc
0809 15:47:02.268700 (+832921us) spinlock_profiling.cc:243] Waited 20.6 ms on lock 0x7b54002371a0. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e91b6951 00007f34e9217321 00007f34e9218b4e 00007f34e9222c89 00007f34e9223556 00007f34e921fedf 00007f34e921f99e 00007f34ee1612e9 00007f34e7847684 00007f34e7847612
0809 15:47:02.375620 (+106920us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:47:02.375786 (+ 166us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":202308224}
I20260809 15:47:02.378161 27761 raft_consensus.cc:3037] T 0c29937a80724ddb9d07a27534a146f3 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Advancing to term 7
I20260809 15:47:02.376744 27677 raft_consensus.cc:2726] T 907bacfa79d04a55a68c8dbe60056462 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260809 15:47:02.383299 27294 raft_consensus.cc:2781] T f9e7cadcd24d45fcbfa84209225e8f2d P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260809 15:47:02.389741 27512 raft_consensus.cc:2781] T 297c82ed906d4b48b184985eed3de601 P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Leader election won for term 8
I20260809 15:47:02.390247 27294 raft_consensus.cc:482] T f9e7cadcd24d45fcbfa84209225e8f2d P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:47:02.390657 27294 raft_consensus.cc:3037] T f9e7cadcd24d45fcbfa84209225e8f2d P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Advancing to term 7
I20260809 15:47:02.389389 27735 consensus_queue.cc:1035] T 3d880de657e544468c82157769622a26 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.003s
I20260809 15:47:02.387454 26619 consensus_queue.cc:1035] T d3e94b7e50754a81b3d72a07476cb7b9 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.007s
I20260809 15:47:02.381668 27686 raft_consensus.cc:2781] T 00a1881267a14ba8b78c81040f17a014 P c0f38c1908414566aa9115bb2cbdd85d [term 9 FOLLOWER]: Leader election won for term 9
I20260809 15:47:02.395874 27686 raft_consensus.cc:686] T 00a1881267a14ba8b78c81040f17a014 P c0f38c1908414566aa9115bb2cbdd85d [term 9 LEADER]: Becoming Leader. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:47:02.397090 27686 consensus_queue.cc:237] T 00a1881267a14ba8b78c81040f17a014 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:47:02.379947 27334 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 9 is higher.
I20260809 15:47:02.398767 27513 consensus_queue.cc:1035] T 3d880de657e544468c82157769622a26 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260809 15:47:02.400524 23470 leader_election.cc:302] T 0c48a3049897452383257aa757834fb5 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 61e69695258f48baa984c9d395e8ce3f
I20260809 15:47:02.403587 23468 leader_election.cc:302] T c0741ab44d99439abc3aa3bdd27ee160 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c; no voters:
I20260809 15:47:02.403681 23392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ae14a74a26f434db08c652e097385c8" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:47:02.385303 24468 raft_consensus.cc:2379] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 61e69695258f48baa984c9d395e8ce3f [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d in current term 6: Already voted for candidate 61e69695258f48baa984c9d395e8ce3f in this term.
I20260809 15:47:02.404413 23392 raft_consensus.cc:2379] T 6ae14a74a26f434db08c652e097385c8 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c in current term 7: Already voted for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in this term.
I20260809 15:47:02.406154 23228 leader_election.cc:302] T 30c1740ef6874df0a332fbd6fd4b03ce P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters: c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:02.418776 23392 raft_consensus.cc:1226] T cef3e7277d7d4760824a3436ed4a847d P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 6. Current term is 7. Ops: []
I20260809 15:47:02.415442 23353 leader_election.cc:302] T 472816994d814cb198b69e5248697440 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c
I20260809 15:47:02.419935 23517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3aa6a9a52804df0ba9b0ca3c82f38c3" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:47:02.420476 23517 raft_consensus.cc:3037] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Advancing to term 6
I20260809 15:47:02.363915 27544 consensus_queue.cc:1035] T 0c48a3049897452383257aa757834fb5 P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.021s
I20260809 15:47:02.390533 27512 raft_consensus.cc:686] T 297c82ed906d4b48b184985eed3de601 P 65c36d164b0540f099d592f38307ee6c [term 8 LEADER]: Becoming Leader. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:47:02.388860 27322 raft_consensus.cc:2781] T 4b30f28311964a0b9149a50495e83d8f P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260809 15:47:02.448797 23470 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 43729.5us
I20260809 15:47:02.427389 23396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "538a228ddfb54bc6a36528d5ff07597b" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:02.449915 23395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c29937a80724ddb9d07a27534a146f3" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 7 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:47:02.453604 23395 raft_consensus.cc:3037] T 0c29937a80724ddb9d07a27534a146f3 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Advancing to term 7
I20260809 15:47:02.423878 23519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00a1881267a14ba8b78c81040f17a014" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:47:02.468767 23276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df20671a7b874dbb98c0ef28371c1308" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:47:02.469370 23277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "538a228ddfb54bc6a36528d5ff07597b" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:02.470049 23280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2424fd9bd7f14e508fd106cbfda786a4" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 7 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:47:02.471113 23273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0741ab44d99439abc3aa3bdd27ee160" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:47:02.471679 23273 raft_consensus.cc:3037] T c0741ab44d99439abc3aa3bdd27ee160 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Advancing to term 6
I20260809 15:47:02.474896 23280 raft_consensus.cc:2379] T 2424fd9bd7f14e508fd106cbfda786a4 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c in current term 7: Already voted for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in this term.
I20260809 15:47:02.418015 27735 raft_consensus.cc:2781] T 0c48a3049897452383257aa757834fb5 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260809 15:47:02.477438 23280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ae14a74a26f434db08c652e097385c8" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:47:02.478492 23286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c29937a80724ddb9d07a27534a146f3" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 7 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:47:02.487551 23273 raft_consensus.cc:2454] T c0741ab44d99439abc3aa3bdd27ee160 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 6.
I20260809 15:47:02.422977 27334 raft_consensus.cc:2781] T c0741ab44d99439abc3aa3bdd27ee160 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Leader election won for term 6
I20260809 15:47:02.488313 23280 raft_consensus.cc:2379] T 6ae14a74a26f434db08c652e097385c8 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c in current term 7: Already voted for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in this term.
I20260809 15:47:02.479744 23392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30c1740ef6874df0a332fbd6fd4b03ce" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:02.496994 23521 raft_consensus.cc:2454] T 28e17dcb5efe45b8b6a8c42da03b4819 P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 8.
I20260809 15:47:02.417987 27032 consensus_queue.cc:1046] T cee02fff2c9c48a19f43eb358a89e7b0 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 6, Time since last communication: 0.073s
I20260809 15:47:02.501668 27798 raft_consensus.cc:2781] T 30c1740ef6874df0a332fbd6fd4b03ce P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260809 15:47:02.502141 27798 raft_consensus.cc:482] T 30c1740ef6874df0a332fbd6fd4b03ce P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:47:02.502679 27798 raft_consensus.cc:3037] T 30c1740ef6874df0a332fbd6fd4b03ce P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Advancing to term 7
I20260809 15:47:02.479127 27682 consensus_queue.cc:237] T 538a228ddfb54bc6a36528d5ff07597b P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:02.491045 27735 raft_consensus.cc:482] T 0c48a3049897452383257aa757834fb5 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:47:02.507752 27735 raft_consensus.cc:3037] T 0c48a3049897452383257aa757834fb5 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Advancing to term 5
I20260809 15:47:02.474606 24453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3924b648461e463b9d7925e5671d68ae" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:47:02.514786 23280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30c1740ef6874df0a332fbd6fd4b03ce" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:02.515894 23283 raft_consensus.cc:1261] T 763e58d6ba644eea830a6c1664d39c9b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260809 15:47:02.518467 23470 leader_election.cc:302] T df20671a7b874dbb98c0ef28371c1308 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d
W20260809 15:47:02.491909 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 85820.5us
I20260809 15:47:02.417802 27790 consensus_queue.cc:1225] T 907bacfa79d04a55a68c8dbe60056462 P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Peer c0f38c1908414566aa9115bb2cbdd85d log is divergent from this leader: its last log entry 5.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260809 15:47:02.419782 27661 raft_consensus.cc:482] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:47:02.449558 27493 raft_consensus.cc:2726] T 472816994d814cb198b69e5248697440 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260809 15:47:02.523975 23228 leader_election.cc:302] T ba56725e42f74a1baa4c04d9660727f9 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:02.443459 27347 raft_consensus.cc:2726] T 538a228ddfb54bc6a36528d5ff07597b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: Vote denied by peer 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553) with higher term. Message: Invalid argument: T 538a228ddfb54bc6a36528d5ff07597b P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 5. Current term is 6.
I20260809 15:47:02.536055 23285 raft_consensus.cc:1226] T 00a1881267a14ba8b78c81040f17a014 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 9 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 8. Current term is 9. Ops: [5.4-8.6]
I20260809 15:47:02.393427 27796 raft_consensus.cc:504] T f9e7cadcd24d45fcbfa84209225e8f2d P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:02.422961 27762 raft_consensus.cc:686] T 4564dfc6d4094c1b9240b367ccecf3b1 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 LEADER]: Becoming Leader. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Running, Role: LEADER
I20260809 15:47:02.531848 23274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9e7cadcd24d45fcbfa84209225e8f2d" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:02.529575 23395 raft_consensus.cc:2454] T 0c29937a80724ddb9d07a27534a146f3 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 7.
I20260809 15:47:02.544222 23277 raft_consensus.cc:2379] T 538a228ddfb54bc6a36528d5ff07597b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c in current term 6: Already voted for candidate c0f38c1908414566aa9115bb2cbdd85d in this term.
I20260809 15:47:02.452236 27322 raft_consensus.cc:482] T 4b30f28311964a0b9149a50495e83d8f P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Starting leader election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:47:02.546351 23228 leader_election.cc:302] T 6ae14a74a26f434db08c652e097385c8 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c
I20260809 15:47:02.550915 23518 raft_consensus.cc:1261] T 28e17dcb5efe45b8b6a8c42da03b4819 P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260809 15:47:02.565224 23396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3924b648461e463b9d7925e5671d68ae" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:02.577517 23274 raft_consensus.cc:2379] T f9e7cadcd24d45fcbfa84209225e8f2d P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c in current term 7: Already voted for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in this term.
I20260809 15:47:02.535519 27404 raft_consensus.cc:2726] T df20671a7b874dbb98c0ef28371c1308 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260809 15:47:02.545253 27661 raft_consensus.cc:504] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:02.575917 27347 raft_consensus.cc:2781] T 6ae14a74a26f434db08c652e097385c8 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Leader election won for term 7
I20260809 15:47:02.535651 27334 raft_consensus.cc:686] T c0741ab44d99439abc3aa3bdd27ee160 P 65c36d164b0540f099d592f38307ee6c [term 6 LEADER]: Becoming Leader. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:47:02.628582 27751 raft_consensus.cc:482] T 4564dfc6d4094c1b9240b367ccecf3b1 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:02.636302 23396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3aa6a9a52804df0ba9b0ca3c82f38c3" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:47:02.561764 27363 raft_consensus.cc:3032] T cee02fff2c9c48a19f43eb358a89e7b0 P 65c36d164b0540f099d592f38307ee6c [term 6 LEADER]: Stepping down as leader of term 6
W20260809 15:47:02.637135 23470 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 77777.4us
I20260809 15:47:02.423628 27787 consensus_queue.cc:1035] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.126s
I20260809 15:47:02.566512 27322 raft_consensus.cc:3037] T 4b30f28311964a0b9149a50495e83d8f P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:47:02.465821 27749 consensus_queue.cc:1035] T 2424fd9bd7f14e508fd106cbfda786a4 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.098s
I20260809 15:47:02.646765 27704 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.BulkChangeConfig from 127.0.0.1:52842 (request call id 10) took 2664 ms. Trace:
I20260809 15:47:02.662957 23392 raft_consensus.cc:1226] T 00a1881267a14ba8b78c81040f17a014 P c0f38c1908414566aa9115bb2cbdd85d [term 9 LEADER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 8. Current term is 9. Ops: [8.6-8.6]
I20260809 15:47:02.676229 23521 raft_consensus.cc:1261] T df20671a7b874dbb98c0ef28371c1308 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Refusing update from remote peer 67b3aeadfa7a432abe8af4d292e04ab9: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20260809 15:47:02.681563 23396 raft_consensus.cc:2379] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 61e69695258f48baa984c9d395e8ce3f in current term 6: Already voted for candidate c0f38c1908414566aa9115bb2cbdd85d in this term.
I20260809 15:47:02.636046 27395 consensus_queue.cc:1035] T 28e17dcb5efe45b8b6a8c42da03b4819 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.042s
I20260809 15:47:02.699563 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3aa6a9a52804df0ba9b0ca3c82f38c3" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:47:02.667943 27493 raft_consensus.cc:482] T d0721a4403a440f383318a3cea9d1442 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:47:02.544044 27518 raft_consensus.cc:2726] T ba56725e42f74a1baa4c04d9660727f9 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260809 15:47:02.643743 24463 raft_consensus.cc:2454] T d3be1e356b254d22b995be32f1c12f6f P 61e69695258f48baa984c9d395e8ce3f [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 8.
I20260809 15:47:02.671845 27480 leader_election.cc:288] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 6 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:02.646104 27531 consensus_queue.cc:1035] T 763e58d6ba644eea830a6c1664d39c9b P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.042s
I20260809 15:47:02.675464 27347 raft_consensus.cc:686] T 6ae14a74a26f434db08c652e097385c8 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 LEADER]: Becoming Leader. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Running, Role: LEADER
I20260809 15:47:02.682400 27363 raft_consensus.cc:729] T cee02fff2c9c48a19f43eb358a89e7b0 P 65c36d164b0540f099d592f38307ee6c [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:47:02.685071 27762 consensus_queue.cc:237] T 4564dfc6d4094c1b9240b367ccecf3b1 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:02.695458 27704 rpcz_store.cc:276] 0809 15:46:59.973185 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:59.978039 (+ 4854us) service_pool.cc:227] Handling call
0809 15:47:00.304922 (+326883us) log.cc:842] Serialized 506 byte log entry
0809 15:47:00.801814 (+496892us) spinlock_profiling.cc:243] Waited 2.77 ms on lock 0x7b1800248aa0. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34ef1978b1 00007f34ef1a4661 00007f34e626f4ca 00007f34ef950f99 00007f34ed0c8bcf 00007f34e62f87b6 00007f34e62fbb11 00007f34e62fbac9 00007f34e62fba59 00007f34e62fba21 00007f34e62fad1d
0809 15:47:02.606191 (+1804377us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":419374,"raft.queue_time_us":100753,"raft.run_cpu_time_us":1876,"raft.run_wall_time_us":162196,"spinlock_wait_cycles":6363904,"thread_start_us":82841,"threads_started":1,"wal-append.queue_time_us":152739,"wal-append.run_cpu_time_us":1708,"wal-append.run_wall_time_us":1004993}
W20260809 15:47:02.688802 24406 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 119286us
I20260809 15:47:02.704828 24410 leader_election.cc:302] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:02.706508 23285 raft_consensus.cc:1261] T d1a134ba33ea4076bbaac0ae460c17ba P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260809 15:47:02.706707 23277 raft_consensus.cc:1261] T 07fcabb569414bddbd39df6cedd24e64 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260809 15:47:02.630235 27512 consensus_queue.cc:237] T 297c82ed906d4b48b184985eed3de601 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:47:02.721534 27829 log.cc:924] Time spent T ba56725e42f74a1baa4c04d9660727f9 P 65c36d164b0540f099d592f38307ee6c: Append to log took a long time: real 0.059s user 0.000s sys 0.000s
I20260809 15:47:02.744731 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9e7cadcd24d45fcbfa84209225e8f2d" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:47:02.727695 23472 leader_election.cc:302] T 6ae14a74a26f434db08c652e097385c8 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:02.728758 27695 raft_consensus.cc:3032] T 907bacfa79d04a55a68c8dbe60056462 P 61e69695258f48baa984c9d395e8ce3f [term 6 LEADER]: Stepping down as leader of term 6
I20260809 15:47:02.728812 27796 leader_election.cc:288] T f9e7cadcd24d45fcbfa84209225e8f2d P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 7 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:02.751920 27781 raft_consensus.cc:482] T d3be1e356b254d22b995be32f1c12f6f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:47:02.712707 27493 raft_consensus.cc:504] T d0721a4403a440f383318a3cea9d1442 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:02.730786 27347 consensus_queue.cc:237] T 6ae14a74a26f434db08c652e097385c8 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:02.732015 27363 consensus_queue.cc:260] T cee02fff2c9c48a19f43eb358a89e7b0 P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:02.767042 27363 raft_consensus.cc:3037] T cee02fff2c9c48a19f43eb358a89e7b0 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Advancing to term 7
I20260809 15:47:02.768414 23277 raft_consensus.cc:1261] T 0a51c0baaa6e432997c9d3552af4f254 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Refusing update from remote peer 61e69695258f48baa984c9d395e8ce3f: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
W20260809 15:47:02.769984 27821 log.cc:924] Time spent T 0c48a3049897452383257aa757834fb5 P 67b3aeadfa7a432abe8af4d292e04ab9: Append to log took a long time: real 0.141s user 0.001s sys 0.000s
I20260809 15:47:02.734931 27661 leader_election.cc:288] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:02.712451 27404 consensus_queue.cc:1046] T 00a1881267a14ba8b78c81040f17a014 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.027s
I20260809 15:47:02.722239 27334 consensus_queue.cc:237] T c0741ab44d99439abc3aa3bdd27ee160 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:02.771765 23270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0721a4403a440f383318a3cea9d1442" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:02.773262 23270 raft_consensus.cc:1261] T 28e17dcb5efe45b8b6a8c42da03b4819 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260809 15:47:02.752763 27796 consensus_queue.cc:1035] T df20671a7b874dbb98c0ef28371c1308 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.023s
I20260809 15:47:02.776000 27797 consensus_queue.cc:1035] T 28e17dcb5efe45b8b6a8c42da03b4819 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260809 15:47:02.740854 27294 raft_consensus.cc:504] T f9e7cadcd24d45fcbfa84209225e8f2d P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:02.734508 27735 raft_consensus.cc:504] T 0c48a3049897452383257aa757834fb5 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:02.752243 23519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ced718c1a5c4da79cb8c1ebd1435b87" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
W20260809 15:47:02.755817 24410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 51071us
I20260809 15:47:02.778959 27294 leader_election.cc:288] T f9e7cadcd24d45fcbfa84209225e8f2d P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 7 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:47:02.779589 23518 raft_consensus.cc:1226] T 0c48a3049897452383257aa757834fb5 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Rejecting Update request from peer 61e69695258f48baa984c9d395e8ce3f for earlier term 4. Current term is 5. Ops: [4.7-4.7]
I20260809 15:47:02.757225 23399 raft_consensus.cc:1261] T df20671a7b874dbb98c0ef28371c1308 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Refusing update from remote peer 67b3aeadfa7a432abe8af4d292e04ab9: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20260809 15:47:02.759488 27686 raft_consensus.cc:482] T 30c1740ef6874df0a332fbd6fd4b03ce P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:47:02.760541 27544 raft_consensus.cc:2726] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 61e69695258f48baa984c9d395e8ce3f [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260809 15:47:02.761188 23404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ced718c1a5c4da79cb8c1ebd1435b87" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:02.724160 27780 raft_consensus.cc:482] T 953eea268e17491ca76f3a555860a512 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:47:02.752725 27781 raft_consensus.cc:504] T d3be1e356b254d22b995be32f1c12f6f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:02.762161 23396 raft_consensus.cc:1261] T 763e58d6ba644eea830a6c1664d39c9b P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260809 15:47:02.764377 23393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9e7cadcd24d45fcbfa84209225e8f2d" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:47:02.790191 23393 raft_consensus.cc:3037] T f9e7cadcd24d45fcbfa84209225e8f2d P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Advancing to term 7
I20260809 15:47:02.781698 23526 raft_consensus.cc:2379] T f9e7cadcd24d45fcbfa84209225e8f2d P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 in current term 7: Already voted for candidate 65c36d164b0540f099d592f38307ee6c in this term.
I20260809 15:47:02.684576 27751 raft_consensus.cc:504] T 4564dfc6d4094c1b9240b367ccecf3b1 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:02.807072 27659 consensus_queue.cc:1035] T df20671a7b874dbb98c0ef28371c1308 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260809 15:47:02.797185 27780 raft_consensus.cc:504] T 953eea268e17491ca76f3a555860a512 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:02.809603 27735 leader_election.cc:288] T 0c48a3049897452383257aa757834fb5 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 5 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:02.810707 27798 raft_consensus.cc:504] T 30c1740ef6874df0a332fbd6fd4b03ce P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:02.812075 23280 raft_consensus.cc:2379] T 30c1740ef6874df0a332fbd6fd4b03ce P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c in current term 7: Already voted for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in this term.
I20260809 15:47:02.813916 23282 raft_consensus.cc:1261] T 00a1881267a14ba8b78c81040f17a014 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 9 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260809 15:47:02.814033 27798 leader_election.cc:288] T 30c1740ef6874df0a332fbd6fd4b03ce P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 7 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:02.816109 27704 consensus_queue.cc:1035] T 00a1881267a14ba8b78c81040f17a014 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.000s
I20260809 15:47:02.817277 27781 leader_election.cc:288] T d3be1e356b254d22b995be32f1c12f6f P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:02.752996 27053 consensus_queue.cc:1035] T 07fcabb569414bddbd39df6cedd24e64 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.065s
W20260809 15:47:02.821434 23470 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 72560.9us
I20260809 15:47:02.761137 27493 leader_election.cc:288] T d0721a4403a440f383318a3cea9d1442 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:02.824105 23270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9e7cadcd24d45fcbfa84209225e8f2d" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:47:02.833817 23270 raft_consensus.cc:2379] T f9e7cadcd24d45fcbfa84209225e8f2d P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c in current term 7: Already voted for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in this term.
I20260809 15:47:02.834549 23270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c48a3049897452383257aa757834fb5" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:47:02.825935 27751 leader_election.cc:288] T 4564dfc6d4094c1b9240b367ccecf3b1 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:02.826721 24448 raft_consensus.cc:1203] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 61e69695258f48baa984c9d395e8ce3f [term 9 LEADER]: Deduplicated request from leader. Original: 6.7->[8.8-8.8] Dedup: 8.8->[]
I20260809 15:47:02.829723 27404 consensus_queue.cc:1035] T d1a134ba33ea4076bbaac0ae460c17ba P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20260809 15:47:02.830569 27492 raft_consensus.cc:3032] T 00a1881267a14ba8b78c81040f17a014 P 65c36d164b0540f099d592f38307ee6c [term 8 LEADER]: Stepping down as leader of term 8
I20260809 15:47:02.753901 27695 raft_consensus.cc:729] T 907bacfa79d04a55a68c8dbe60056462 P 61e69695258f48baa984c9d395e8ce3f [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 61e69695258f48baa984c9d395e8ce3f, State: Running, Role: LEADER
W20260809 15:47:02.877493 23472 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 149846us
I20260809 15:47:02.881243 23397 raft_consensus.cc:1226] T 907bacfa79d04a55a68c8dbe60056462 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Rejecting Update request from peer 61e69695258f48baa984c9d395e8ce3f for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20260809 15:47:02.832851 27544 raft_consensus.cc:482] T 4bcac4b24e7a4448abdc934bba5a624e P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 15:47:02.869750 27830 log.cc:924] Time spent T 2db057fd10e149a9895dff6ef8c9f038 P 65c36d164b0540f099d592f38307ee6c: Append to log took a long time: real 0.166s user 0.000s sys 0.001s
I20260809 15:47:02.810966 27780 leader_election.cc:288] T 953eea268e17491ca76f3a555860a512 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:02.882848 23519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4564dfc6d4094c1b9240b367ccecf3b1" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:02.890251 23519 raft_consensus.cc:2454] T 4564dfc6d4094c1b9240b367ccecf3b1 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 6.
I20260809 15:47:02.864809 23270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4564dfc6d4094c1b9240b367ccecf3b1" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:02.874588 24448 raft_consensus.cc:1226] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 61e69695258f48baa984c9d395e8ce3f [term 9 LEADER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 8. Current term is 9. Ops: [8.8-8.8]
I20260809 15:47:02.885495 23518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0721a4403a440f383318a3cea9d1442" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:02.905649 23345 leader_election.cc:302] T d3be1e356b254d22b995be32f1c12f6f P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d; no voters: 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:47:02.910192 23517 raft_consensus.cc:2454] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 6.
I20260809 15:47:02.865015 27492 raft_consensus.cc:729] T 00a1881267a14ba8b78c81040f17a014 P 65c36d164b0540f099d592f38307ee6c [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:47:02.796821 27686 raft_consensus.cc:504] T 30c1740ef6874df0a332fbd6fd4b03ce P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:47:02.913638 23470 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 90641.7us
W20260809 15:47:02.915006 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 128125us
I20260809 15:47:02.921631 23472 leader_election.cc:302] T 538a228ddfb54bc6a36528d5ff07597b P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:02.933804 23519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30c1740ef6874df0a332fbd6fd4b03ce" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:47:02.934379 23519 raft_consensus.cc:3037] T 30c1740ef6874df0a332fbd6fd4b03ce P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Advancing to term 7
I20260809 15:47:02.935393 23393 raft_consensus.cc:2454] T f9e7cadcd24d45fcbfa84209225e8f2d P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 7.
I20260809 15:47:02.874641 27197 consensus_queue.cc:1035] T 0a51c0baaa6e432997c9d3552af4f254 P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.039s
I20260809 15:47:02.902683 27689 raft_consensus.cc:2726] T 6ae14a74a26f434db08c652e097385c8 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260809 15:47:02.936535 23228 leader_election.cc:302] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:02.911321 24463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3be1e356b254d22b995be32f1c12f6f" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:47:02.945122 23228 leader_election.cc:302] T f9e7cadcd24d45fcbfa84209225e8f2d P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:47:02.947003 23404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3be1e356b254d22b995be32f1c12f6f" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:02.947671 23397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30c1740ef6874df0a332fbd6fd4b03ce" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:47:02.951408 23397 raft_consensus.cc:3037] T 30c1740ef6874df0a332fbd6fd4b03ce P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Advancing to term 7
I20260809 15:47:02.948258 23395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "953eea268e17491ca76f3a555860a512" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:02.952323 23395 raft_consensus.cc:2454] T 953eea268e17491ca76f3a555860a512 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 7.
I20260809 15:47:02.948832 23518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "953eea268e17491ca76f3a555860a512" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:02.954548 23518 raft_consensus.cc:2379] T 953eea268e17491ca76f3a555860a512 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 in current term 7: Already voted for candidate 65c36d164b0540f099d592f38307ee6c in this term.
I20260809 15:47:02.919569 27395 raft_consensus.cc:2781] T d3be1e356b254d22b995be32f1c12f6f P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Leader election won for term 8
I20260809 15:47:02.879534 27322 raft_consensus.cc:504] T 4b30f28311964a0b9149a50495e83d8f P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } attrs { replace: true } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } }
I20260809 15:47:02.921322 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c48a3049897452383257aa757834fb5" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f"
W20260809 15:47:02.973456 23472 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 51918.3us
I20260809 15:47:03.025913 23228 leader_election.cc:302] T 953eea268e17491ca76f3a555860a512 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:47:02.888269 27544 raft_consensus.cc:504] T 4bcac4b24e7a4448abdc934bba5a624e P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:02.942533 27704 raft_consensus.cc:482] T cef3e7277d7d4760824a3436ed4a847d P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:02.947603 27661 raft_consensus.cc:2726] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260809 15:47:03.031957 23468 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.68:34689, user_credentials={real_user=slave}} blocked reactor thread for 89678.7us
I20260809 15:47:03.036818 23519 raft_consensus.cc:2454] T 30c1740ef6874df0a332fbd6fd4b03ce P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 7.
I20260809 15:47:03.043325 23528 raft_consensus.cc:1226] T 4b30f28311964a0b9149a50495e83d8f P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Rejecting Update request from peer c0f38c1908414566aa9115bb2cbdd85d for earlier term 3. Current term is 4. Ops: []
W20260809 15:47:03.041599 23470 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 108592us
I20260809 15:47:03.029129 27772 raft_consensus.cc:2726] T 538a228ddfb54bc6a36528d5ff07597b P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260809 15:47:03.038225 24463 raft_consensus.cc:2354] T d3be1e356b254d22b995be32f1c12f6f P 61e69695258f48baa984c9d395e8ce3f [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 7. Current term is 8.
I20260809 15:47:03.030475 27032 raft_consensus.cc:482] T 472816994d814cb198b69e5248697440 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
I20260809 15:47:03.031891 27347 raft_consensus.cc:2781] T f9e7cadcd24d45fcbfa84209225e8f2d P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Leader election won for term 7
I20260809 15:47:03.058750 23472 leader_election.cc:302] T 0c29937a80724ddb9d07a27534a146f3 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:47:03.036993 27796 raft_consensus.cc:482] T cef3e7277d7d4760824a3436ed4a847d P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:47:03.031891 27780 raft_consensus.cc:482] T 213543cdbe814c89a17a963469922c1b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:47:03.084686 27780 raft_consensus.cc:504] T 213543cdbe814c89a17a963469922c1b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:02.927631 27695 consensus_queue.cc:260] T 907bacfa79d04a55a68c8dbe60056462 P 61e69695258f48baa984c9d395e8ce3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:03.072465 23470 leader_election.cc:302] T 30c1740ef6874df0a332fbd6fd4b03ce P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:03.089996 27531 raft_consensus.cc:2726] T 30c1740ef6874df0a332fbd6fd4b03ce P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260809 15:47:03.078696 27689 raft_consensus.cc:2781] T 0c29937a80724ddb9d07a27534a146f3 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Leader election won for term 7
W20260809 15:47:03.093886 27827 log.cc:924] Time spent T ae6d2c08a7a94f58ac8dd698e3fd456c P c0f38c1908414566aa9115bb2cbdd85d: Append to log took a long time: real 0.419s user 0.002s sys 0.000s
I20260809 15:47:03.095670 27689 raft_consensus.cc:686] T 0c29937a80724ddb9d07a27534a146f3 P 65c36d164b0540f099d592f38307ee6c [term 7 LEADER]: Becoming Leader. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:47:03.096755 27659 raft_consensus.cc:482] T d0721a4403a440f383318a3cea9d1442 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:47:03.097560 27363 raft_consensus.cc:482] T c5c2e425a74b4e64ae70b5def750434c P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:47:03.097460 27659 raft_consensus.cc:504] T d0721a4403a440f383318a3cea9d1442 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:03.098318 27363 raft_consensus.cc:504] T c5c2e425a74b4e64ae70b5def750434c P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:47:03.100718 27846 log.cc:924] Time spent T 3d880de657e544468c82157769622a26 P 67b3aeadfa7a432abe8af4d292e04ab9: Append to log took a long time: real 0.173s user 0.000s sys 0.001s
I20260809 15:47:03.101318 27363 leader_election.cc:288] T c5c2e425a74b4e64ae70b5def750434c P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:47:03.103649 27780 leader_election.cc:288] T 213543cdbe814c89a17a963469922c1b P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:03.097167 27689 consensus_queue.cc:237] T 0c29937a80724ddb9d07a27534a146f3 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.2, Last appended by leader: 2, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:47:03.106019 27847 log.cc:924] Time spent T a3622f6e3e3641cea1960e3a31498bb7 P 67b3aeadfa7a432abe8af4d292e04ab9: Append to log took a long time: real 0.073s user 0.000s sys 0.000s
I20260809 15:47:03.115633 27032 raft_consensus.cc:504] T 472816994d814cb198b69e5248697440 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:03.118625 27032 leader_election.cc:288] T 472816994d814cb198b69e5248697440 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
W20260809 15:47:03.119349 27844 consensus_queue.cc:1159] T 907bacfa79d04a55a68c8dbe60056462 P 61e69695258f48baa984c9d395e8ce3f [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "c0f38c1908414566aa9115bb2cbdd85d" responder_term: 7 status { last_received { term: 5 index: 7 } last_committed_idx: 6 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 61e69695258f48baa984c9d395e8ce3f for earlier term 6. Current term is 7. Ops: [6.7-6.7]" } } last_received_current_leader { term: 0 index: 0 } }
I20260809 15:47:03.122361 27347 raft_consensus.cc:686] T f9e7cadcd24d45fcbfa84209225e8f2d P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 LEADER]: Becoming Leader. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Running, Role: LEADER
I20260809 15:47:03.123818 27347 consensus_queue.cc:237] T f9e7cadcd24d45fcbfa84209225e8f2d P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:03.059438 27870 raft_consensus.cc:2781] T 953eea268e17491ca76f3a555860a512 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260809 15:47:03.080529 27796 raft_consensus.cc:504] T cef3e7277d7d4760824a3436ed4a847d P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:03.082942 23472 leader_election.cc:302] T 3924b648461e463b9d7925e5671d68ae P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d
W20260809 15:47:03.080142 27845 log.cc:924] Time spent T a88421dba80d4d3388aa5eb62e23853b P 67b3aeadfa7a432abe8af4d292e04ab9: Append to log took a long time: real 0.113s user 0.000s sys 0.001s
I20260809 15:47:03.056090 27661 raft_consensus.cc:482] T f7715d71dbac447b9351fc5eb36c5e80 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
W20260809 15:47:03.142514 23472 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 59658us
I20260809 15:47:03.077356 27677 raft_consensus.cc:482] T df20671a7b874dbb98c0ef28371c1308 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:03.067155 27792 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.0.0.1:38724 (request call id 523) took 1360 ms. Trace:
I20260809 15:47:03.031143 27492 consensus_queue.cc:260] T 00a1881267a14ba8b78c81040f17a014 P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:03.045184 27704 raft_consensus.cc:504] T cef3e7277d7d4760824a3436ed4a847d P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:03.064517 27196 consensus_queue.cc:1046] T 0c48a3049897452383257aa757834fb5 P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.080s
I20260809 15:47:03.139029 27870 raft_consensus.cc:482] T 953eea268e17491ca76f3a555860a512 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Starting leader election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
W20260809 15:47:03.162743 24406 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 233049us
I20260809 15:47:03.166929 23468 leader_election.cc:302] T 0c48a3049897452383257aa757834fb5 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 61e69695258f48baa984c9d395e8ce3f, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:47:03.168077 27870 raft_consensus.cc:3037] T 953eea268e17491ca76f3a555860a512 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Advancing to term 7
I20260809 15:47:03.154637 27513 raft_consensus.cc:2726] T 3924b648461e463b9d7925e5671d68ae P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260809 15:47:03.171052 23223 leader_election.cc:398] T d3be1e356b254d22b995be32f1c12f6f P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 7 pre-election: Vote denied by peer 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689) with higher term. Message: Invalid argument: T d3be1e356b254d22b995be32f1c12f6f P 61e69695258f48baa984c9d395e8ce3f [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 7. Current term is 8.
I20260809 15:47:03.172255 23393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9e7cadcd24d45fcbfa84209225e8f2d" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:47:03.185214 23223 leader_election.cc:401] T d3be1e356b254d22b995be32f1c12f6f P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 7 pre-election: Cancelling election due to peer responding with higher term
I20260809 15:47:03.174520 23399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5c2e425a74b4e64ae70b5def750434c" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:03.175175 23392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472816994d814cb198b69e5248697440" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:03.162493 27363 consensus_queue.cc:1035] T 763e58d6ba644eea830a6c1664d39c9b P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.016s
I20260809 15:47:03.180198 23394 raft_consensus.cc:1261] T ba56725e42f74a1baa4c04d9660727f9 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260809 15:47:03.180941 27659 leader_election.cc:288] T d0721a4403a440f383318a3cea9d1442 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:03.163602 27492 raft_consensus.cc:3037] T 00a1881267a14ba8b78c81040f17a014 P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Advancing to term 9
I20260809 15:47:03.185529 23393 raft_consensus.cc:2379] T f9e7cadcd24d45fcbfa84209225e8f2d P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c in current term 7: Already voted for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in this term.
I20260809 15:47:03.190037 23400 raft_consensus.cc:1261] T 4bcac4b24e7a4448abdc934bba5a624e P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260809 15:47:03.185933 23353 leader_election.cc:302] T 4564dfc6d4094c1b9240b367ccecf3b1 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters: 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:47:03.170078 27796 leader_election.cc:288] T cef3e7277d7d4760824a3436ed4a847d P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:03.196525 23395 raft_consensus.cc:1261] T d1a134ba33ea4076bbaac0ae460c17ba P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
W20260809 15:47:03.407107 23472 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 199536us
I20260809 15:47:03.203728 24455 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472816994d814cb198b69e5248697440" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
W20260809 15:47:03.258308 23353 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 72456.1us
I20260809 15:47:03.415704 23353 leader_election.cc:302] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters: 61e69695258f48baa984c9d395e8ce3f
I20260809 15:47:03.187745 27032 raft_consensus.cc:2726] T 0c48a3049897452383257aa757834fb5 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260809 15:47:03.399430 27704 leader_election.cc:288] T cef3e7277d7d4760824a3436ed4a847d P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:03.403321 27686 leader_election.cc:288] T 30c1740ef6874df0a332fbd6fd4b03ce P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:03.404368 27544 leader_election.cc:288] T 4bcac4b24e7a4448abdc934bba5a624e P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
W20260809 15:47:03.404961 27883 log.cc:924] Time spent T 07fcabb569414bddbd39df6cedd24e64 P 67b3aeadfa7a432abe8af4d292e04ab9: Append to log took a long time: real 0.220s user 0.000s sys 0.000s
W20260809 15:47:03.405140 27862 log.cc:924] Time spent T 28e17dcb5efe45b8b6a8c42da03b4819 P 65c36d164b0540f099d592f38307ee6c: Append to log took a long time: real 0.241s user 0.000s sys 0.000s
I20260809 15:47:03.154791 27661 raft_consensus.cc:504] T f7715d71dbac447b9351fc5eb36c5e80 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:47:03.467445 27761 consensus_meta.cc:287] T 0c29937a80724ddb9d07a27534a146f3 P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent flushing consensus metadata: real 1.078s user 0.007s sys 0.004s
I20260809 15:47:03.467618 27661 leader_election.cc:288] T f7715d71dbac447b9351fc5eb36c5e80 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
W20260809 15:47:03.468556 23468 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.68:34689, user_credentials={real_user=slave}} blocked reactor thread for 53419.7us
W20260809 15:47:03.205906 27861 log.cc:924] Time spent T 297c82ed906d4b48b184985eed3de601 P 65c36d164b0540f099d592f38307ee6c: Append to log took a long time: real 0.133s user 0.001s sys 0.000s
I20260809 15:47:03.160400 27677 raft_consensus.cc:504] T df20671a7b874dbb98c0ef28371c1308 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:03.471313 27677 leader_election.cc:288] T df20671a7b874dbb98c0ef28371c1308 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:03.416832 23353 leader_election.cc:302] T d0721a4403a440f383318a3cea9d1442 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
W20260809 15:47:03.472236 23353 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 55482.2us
I20260809 15:47:03.398778 24468 raft_consensus.cc:1261] T 4bcac4b24e7a4448abdc934bba5a624e P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260809 15:47:03.424304 23228 leader_election.cc:302] T 30c1740ef6874df0a332fbd6fd4b03ce P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:47:03.186221 27519 raft_consensus.cc:482] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
I20260809 15:47:03.452329 23472 leader_election.cc:302] T f9e7cadcd24d45fcbfa84209225e8f2d P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:03.475106 27519 raft_consensus.cc:504] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:03.477038 27519 leader_election.cc:288] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:03.477514 27519 raft_consensus.cc:2726] T d0721a4403a440f383318a3cea9d1442 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260809 15:47:03.480667 26966 raft_consensus.cc:2726] T f9e7cadcd24d45fcbfa84209225e8f2d P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260809 15:47:03.482514 26966 consensus_queue.cc:1035] T 4bcac4b24e7a4448abdc934bba5a624e P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260809 15:47:03.484655 27686 consensus_queue.cc:1046] T 4b30f28311964a0b9149a50495e83d8f P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } attrs { replace: true }, Status: INVALID_TERM, Last received: 3.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.001s
W20260809 15:47:03.435611 24410 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.508s user 0.012s sys 0.036s
W20260809 15:47:03.488569 24410 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.561s user 0.012s sys 0.036s
I20260809 15:47:03.461282 27497 raft_consensus.cc:2781] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Leader election won for term 6
I20260809 15:47:03.448450 27878 raft_consensus.cc:482] T cee02fff2c9c48a19f43eb358a89e7b0 P 61e69695258f48baa984c9d395e8ce3f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:47:03.195722 27518 raft_consensus.cc:3037] T d3be1e356b254d22b995be32f1c12f6f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Advancing to term 8
W20260809 15:47:03.492091 23397 consensus_meta.cc:287] T 30c1740ef6874df0a332fbd6fd4b03ce P c0f38c1908414566aa9115bb2cbdd85d: Time spent flushing consensus metadata: real 0.537s user 0.008s sys 0.000s
W20260809 15:47:03.488991 24410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 569175us
W20260809 15:47:03.498379 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 74158.1us
I20260809 15:47:03.197906 27780 raft_consensus.cc:482] T 2424fd9bd7f14e508fd106cbfda786a4 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:03.457638 27322 leader_election.cc:288] T 4b30f28311964a0b9149a50495e83d8f P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 4 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:47:03.508275 23397 raft_consensus.cc:2454] T 30c1740ef6874df0a332fbd6fd4b03ce P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 7.
W20260809 15:47:03.492867 27835 thread.cc:636] apply [worker] (thread pool) Time spent creating pthread: real 0.769s user 0.005s sys 0.019s
I20260809 15:47:03.486366 27704 raft_consensus.cc:2781] T 4564dfc6d4094c1b9240b367ccecf3b1 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260809 15:47:03.493382 27032 consensus_queue.cc:1035] T ba56725e42f74a1baa4c04d9660727f9 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.058s
I20260809 15:47:03.548467 23521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30c1740ef6874df0a332fbd6fd4b03ce" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:03.522778 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cef3e7277d7d4760824a3436ed4a847d" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:04.059309 23526 raft_consensus.cc:2454] T cef3e7277d7d4760824a3436ed4a847d P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 7.
I20260809 15:47:03.509574 23522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a823d35f4f348d1b52ba9e6d80b00d2" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:04.057947 23397 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55332 (request call id 1415) took 1130 ms. Trace:
I20260809 15:47:03.499045 27761 raft_consensus.cc:2726] T 0c29937a80724ddb9d07a27534a146f3 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: Vote denied by peer 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553) with higher term. Message: Invalid argument: T 0c29937a80724ddb9d07a27534a146f3 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 6. Current term is 7.
W20260809 15:47:04.057091 23223 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.650s user 0.000s sys 0.010s
W20260809 15:47:04.069640 23223 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.707s user 0.000s sys 0.010s
W20260809 15:47:03.687879 27790 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.547s user 0.018s sys 0.012s
I20260809 15:47:04.060240 23397 rpcz_store.cc:276] 0809 15:47:02.927766 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:02.927894 (+ 128us) service_pool.cc:227] Handling call
0809 15:47:04.057885 (+1129991us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:47:03.523038 23518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df20671a7b874dbb98c0ef28371c1308" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:03.554859 23468 leader_election.cc:302] T 472816994d814cb198b69e5248697440 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d
W20260809 15:47:03.850679 27762 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.106s user 0.026s sys 0.068s
I20260809 15:47:03.498423 27677 raft_consensus.cc:3032] T 4b30f28311964a0b9149a50495e83d8f P c0f38c1908414566aa9115bb2cbdd85d [term 3 LEADER]: Stepping down as leader of term 3
W20260809 15:47:03.500939 27871 log.cc:924] Time spent T c0741ab44d99439abc3aa3bdd27ee160 P 65c36d164b0540f099d592f38307ee6c: Append to log took a long time: real 0.369s user 0.000s sys 0.002s
W20260809 15:47:03.687750 23517 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.622s user 0.018s sys 0.014s
W20260809 15:47:04.060285 23345 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.721s user 0.011s sys 0.006s
W20260809 15:47:04.081099 23517 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.015s user 0.019s sys 0.028s
W20260809 15:47:04.081197 23345 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.742s user 0.011s sys 0.006s
W20260809 15:47:03.630353 27395 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.578s user 0.022s sys 0.044s
W20260809 15:47:04.057482 27835 thread.cc:602] apply [worker] (thread pool) Time spent starting thread: real 1.335s user 0.009s sys 0.037s
W20260809 15:47:03.896670 27689 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.787s user 0.030s sys 0.047s
I20260809 15:47:04.073920 23286 raft_consensus.cc:2454] T 0c29937a80724ddb9d07a27534a146f3 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 7.
I20260809 15:47:04.058735 23521 raft_consensus.cc:2354] T 30c1740ef6874df0a332fbd6fd4b03ce P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d for earlier term 6. Current term is 7.
W20260809 15:47:04.056882 27688 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.114s user 0.007s sys 0.027s
I20260809 15:47:03.503808 27497 raft_consensus.cc:686] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P c0f38c1908414566aa9115bb2cbdd85d [term 6 LEADER]: Becoming Leader. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:47:03.502115 27892 raft_consensus.cc:2781] T 30c1740ef6874df0a332fbd6fd4b03ce P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Leader election won for term 7
W20260809 15:47:03.836591 27347 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.667s user 0.007s sys 0.011s
I20260809 15:47:04.058653 27704 raft_consensus.cc:482] T 4564dfc6d4094c1b9240b367ccecf3b1 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260809 15:47:03.941488 27832 log.cc:924] Time spent T ae6d2c08a7a94f58ac8dd698e3fd456c P 65c36d164b0540f099d592f38307ee6c: Append to log took a long time: real 1.225s user 0.009s sys 0.005s
I20260809 15:47:03.832669 27661 raft_consensus.cc:482] T 28e17dcb5efe45b8b6a8c42da03b4819 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 15:47:04.070812 26619 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.587s user 0.002s sys 0.010s
I20260809 15:47:03.498641 27363 consensus_queue.cc:1035] T 4bcac4b24e7a4448abdc934bba5a624e P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.570s
I20260809 15:47:03.524587 27544 raft_consensus.cc:482] T 294d4bcc1e7a477cb56455581277a60f P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:47:03.861188 27796 raft_consensus.cc:482] T 7baa2492df0340c4a5ac241959ce1ac3 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
W20260809 15:47:04.075119 27790 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.937s user 0.024s sys 0.021s
W20260809 15:47:04.074204 27870 consensus_meta.cc:287] T 953eea268e17491ca76f3a555860a512 P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent flushing consensus metadata: real 0.906s user 0.009s sys 0.000s
W20260809 15:47:04.124714 23470 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.710s user 0.004s sys 0.028s
W20260809 15:47:04.082334 27395 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.030s user 0.026s sys 0.063s
W20260809 15:47:04.080616 27762 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.336s user 0.026s sys 0.076s
I20260809 15:47:03.522879 27780 raft_consensus.cc:504] T 2424fd9bd7f14e508fd106cbfda786a4 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:03.502199 27878 raft_consensus.cc:504] T cee02fff2c9c48a19f43eb358a89e7b0 P 61e69695258f48baa984c9d395e8ce3f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:04.073812 27761 raft_consensus.cc:482] T d2d4f749fcfd4fb3bfb8115af3471598 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:47:04.083122 27677 raft_consensus.cc:729] T 4b30f28311964a0b9149a50495e83d8f P c0f38c1908414566aa9115bb2cbdd85d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
W20260809 15:47:04.124599 27347 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.955s user 0.011s sys 0.011s
I20260809 15:47:04.090034 27658 raft_consensus.cc:482] T 0c48a3049897452383257aa757834fb5 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
W20260809 15:47:04.091639 27688 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.154s user 0.008s sys 0.027s
W20260809 15:47:04.090848 27689 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.981s user 0.031s sys 0.048s
I20260809 15:47:04.176568 27704 raft_consensus.cc:3037] T 4564dfc6d4094c1b9240b367ccecf3b1 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Advancing to term 7
W20260809 15:47:04.179898 23347 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.812s user 0.013s sys 0.035s
W20260809 15:47:04.177420 23470 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.763s user 0.008s sys 0.041s
W20260809 15:47:04.177256 26619 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.693s user 0.003s sys 0.010s
I20260809 15:47:03.157243 27792 rpcz_store.cc:276] 0809 15:47:01.704353 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:01.727545 (+ 23192us) service_pool.cc:227] Handling call
0809 15:47:01.926205 (+198660us) ts_tablet_manager.cc:928] T 4b30f28311964a0b9149a50495e83d8f P 61e69695258f48baa984c9d395e8ce3f: Initiating tablet copy from peer c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
0809 15:47:02.069603 (+143398us) tablet_copy_client.cc:361] Tablet copy session begun
0809 15:47:02.759991 (+690388us) tablet_metadata.cc:641] Metadata flushed
0809 15:47:02.760295 (+ 304us) tablet_copy_client.cc:490] Wrote new tablet metadata
0809 15:47:02.878867 (+118572us) tablet_copy_client.cc:411] Wrote new consensus metadata
0809 15:47:02.939415 (+ 60548us) tablet_replica.cc:205] Creating consensus instance
0809 15:47:03.043438 (+104023us) ts_tablet_manager.cc:983] Replica registered.
0809 15:47:03.047884 (+ 4446us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":277,"tablet-copy.queue_time_us":138364,"thread_start_us":94364,"threads_started":2}
I20260809 15:47:04.222575 27893 raft_consensus.cc:482] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
W20260809 15:47:04.222975 23227 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.019s user 0.006s sys 0.021s
I20260809 15:47:04.179373 27395 raft_consensus.cc:686] T d3be1e356b254d22b995be32f1c12f6f P c0f38c1908414566aa9115bb2cbdd85d [term 8 LEADER]: Becoming Leader. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:47:04.223429 27493 raft_consensus.cc:482] T 8e4ffcf83c234ae7910ddef3ec0b4bda P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
W20260809 15:47:04.228732 23347 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.861s user 0.015s sys 0.036s
I20260809 15:47:04.179982 24463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a823d35f4f348d1b52ba9e6d80b00d2" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:47:04.176867 27886 raft_consensus.cc:482] T 73f6f44c259547f1b05a3a434616fa54 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:47:04.180819 27892 raft_consensus.cc:686] T 30c1740ef6874df0a332fbd6fd4b03ce P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 LEADER]: Becoming Leader. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Running, Role: LEADER
W20260809 15:47:04.234324 23227 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.190s user 0.007s sys 0.021s
I20260809 15:47:04.184614 27909 raft_consensus.cc:482] T 0a51c0baaa6e432997c9d3552af4f254 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 15:47:04.235809 23469 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.630s user 0.000s sys 0.043s
W20260809 15:47:04.242457 23469 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.790s user 0.000s sys 0.043s
I20260809 15:47:04.200266 27544 raft_consensus.cc:504] T 294d4bcc1e7a477cb56455581277a60f P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:04.247069 27909 raft_consensus.cc:504] T 0a51c0baaa6e432997c9d3552af4f254 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:04.228667 27762 raft_consensus.cc:482] T 3296ef50be254fc29cfc77d22a8f7a23 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
W20260809 15:47:04.235191 27838 log.cc:924] Time spent T 4b30f28311964a0b9149a50495e83d8f P c0f38c1908414566aa9115bb2cbdd85d: Append to log took a long time: real 1.483s user 0.006s sys 0.015s
I20260809 15:47:04.179566 27796 raft_consensus.cc:504] T 7baa2492df0340c4a5ac241959ce1ac3 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:04.237980 27913 raft_consensus.cc:482] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:47:04.236653 27347 raft_consensus.cc:482] T 563daf42d7784db195ea99c3cb21b370 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
I20260809 15:47:04.235591 27695 raft_consensus.cc:3037] T 907bacfa79d04a55a68c8dbe60056462 P 61e69695258f48baa984c9d395e8ce3f [term 6 FOLLOWER]: Advancing to term 7
I20260809 15:47:04.238674 27689 raft_consensus.cc:2726] T 472816994d814cb198b69e5248697440 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260809 15:47:04.177444 27870 raft_consensus.cc:504] T 953eea268e17491ca76f3a555860a512 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:04.207942 27761 raft_consensus.cc:504] T d2d4f749fcfd4fb3bfb8115af3471598 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:47:04.255326 27519 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.535s user 0.039s sys 0.046s
I20260809 15:47:04.207410 27661 raft_consensus.cc:504] T 28e17dcb5efe45b8b6a8c42da03b4819 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:04.228953 27658 raft_consensus.cc:504] T 0c48a3049897452383257aa757834fb5 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:04.263202 27792 tablet_copy_client.cc:791] T 4b30f28311964a0b9149a50495e83d8f P 61e69695258f48baa984c9d395e8ce3f: tablet copy: Starting download of 0 data blocks...
I20260809 15:47:04.247341 27677 consensus_queue.cc:260] T 4b30f28311964a0b9149a50495e83d8f P c0f38c1908414566aa9115bb2cbdd85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } attrs { replace: true } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } }
I20260809 15:47:04.241032 27493 raft_consensus.cc:504] T 8e4ffcf83c234ae7910ddef3ec0b4bda P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
W20260809 15:47:04.265952 27519 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.546s user 0.039s sys 0.046s
I20260809 15:47:04.242954 27886 raft_consensus.cc:504] T 73f6f44c259547f1b05a3a434616fa54 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:04.274952 27661 leader_election.cc:288] T 28e17dcb5efe45b8b6a8c42da03b4819 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:04.258869 27913 raft_consensus.cc:504] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:04.272006 23529 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56766 (request call id 2003) took 2056 ms. Trace:
I20260809 15:47:04.298745 27939 raft_consensus.cc:482] T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
W20260809 15:47:04.299535 23472 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 217504us
I20260809 15:47:04.301493 27582 raft_consensus.cc:482] T 2db057fd10e149a9895dff6ef8c9f038 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:04.302013 23521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "213543cdbe814c89a17a963469922c1b" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:04.303094 23518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cef3e7277d7d4760824a3436ed4a847d" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:04.303771 23522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0721a4403a440f383318a3cea9d1442" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:04.304677 27934 raft_consensus.cc:482] T 297c82ed906d4b48b184985eed3de601 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:04.307049 27936 raft_consensus.cc:482] T 998c6cee72cf40228c6dba064c0db2da P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:47:04.303579 27950 raft_consensus.cc:482] T ae6d2c08a7a94f58ac8dd698e3fd456c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:47:04.340454 27936 raft_consensus.cc:504] T 998c6cee72cf40228c6dba064c0db2da P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:04.341627 27913 leader_election.cc:288] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:04.342145 27913 raft_consensus.cc:482] T 4b30f28311964a0b9149a50495e83d8f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:47:04.342497 27936 leader_election.cc:288] T 998c6cee72cf40228c6dba064c0db2da P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:04.313750 23520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7715d71dbac447b9351fc5eb36c5e80" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:04.317056 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bcac4b24e7a4448abdc934bba5a624e" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:04.236340 27497 consensus_queue.cc:237] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:04.265992 27197 raft_consensus.cc:482] T c0741ab44d99439abc3aa3bdd27ee160 P 61e69695258f48baa984c9d395e8ce3f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:04.329073 23529 rpcz_store.cc:276] 0809 15:47:02.216063 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:02.219314 (+ 3251us) service_pool.cc:227] Handling call
0809 15:47:02.219672 (+ 358us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 15:47:02.292174 (+ 72502us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:47:02.292202 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 15:47:02.292633 (+ 431us) log.cc:842] Serialized 26 byte log entry
0809 15:47:02.549509 (+256876us) raft_consensus.cc:1631] Marking committed up to 2
0809 15:47:02.549543 (+ 34us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:47:02.549600 (+ 57us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 15:47:03.705215 (+1155615us) spinlock_profiling.cc:243] Waited 13.7 ms on lock 0x7b44000232cc. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e613cf11 00007f34e620aa95 00007f34e620b4b2 00007f34e8eb3511 00007f34e8eaee1a 00007f34e8ea2bee 00007f34e8ea1f36 00007f34e8ea1b58 00007f34e8eab551 00007f34e8eab509 00007f34e8eab499
0809 15:47:04.262739 (+557524us) raft_consensus.cc:1671] finished
0809 15:47:04.262761 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:47:04.271957 (+ 9196us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":121147,"raft.queue_time_us":84849,"raft.run_cpu_time_us":102,"raft.run_wall_time_us":97,"spinlock_wait_cycles":31648384,"thread_start_us":196525,"threads_started":1,"wal-append.queue_time_us":354574}
I20260809 15:47:04.335609 23518 raft_consensus.cc:2454] T cef3e7277d7d4760824a3436ed4a847d P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 7.
I20260809 15:47:04.237941 27893 raft_consensus.cc:504] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:04.280946 27661 raft_consensus.cc:482] T 0acc47ce21c3422698d1ef7277c3ed63 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:47:04.340552 27950 raft_consensus.cc:504] T ae6d2c08a7a94f58ac8dd698e3fd456c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:04.311350 23526 raft_consensus.cc:1261] T 6ae14a74a26f434db08c652e097385c8 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Refusing update from remote peer 67b3aeadfa7a432abe8af4d292e04ab9: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260809 15:47:04.254904 27762 raft_consensus.cc:504] T 3296ef50be254fc29cfc77d22a8f7a23 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:04.318333 23519 raft_consensus.cc:1226] T 0c48a3049897452383257aa757834fb5 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Rejecting Update request from peer 61e69695258f48baa984c9d395e8ce3f for earlier term 4. Current term is 5. Ops: [4.7-4.7]
I20260809 15:47:04.268502 27921 raft_consensus.cc:482] T 3d880de657e544468c82157769622a26 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:47:04.261987 27347 raft_consensus.cc:504] T 563daf42d7784db195ea99c3cb21b370 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:04.331573 27906 raft_consensus.cc:482] T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
I20260809 15:47:04.264376 27915 consensus_queue.cc:1035] T d1a134ba33ea4076bbaac0ae460c17ba P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.098s
I20260809 15:47:04.352479 27792 tablet_copy_client.cc:655] T 4b30f28311964a0b9149a50495e83d8f P 61e69695258f48baa984c9d395e8ce3f: tablet copy: Starting download of 1 WAL segments...
I20260809 15:47:04.337783 27892 consensus_queue.cc:237] T 30c1740ef6874df0a332fbd6fd4b03ce P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:47:04.373238 27902 log.cc:924] Time spent T f9e7cadcd24d45fcbfa84209225e8f2d P 67b3aeadfa7a432abe8af4d292e04ab9: Append to log took a long time: real 0.113s user 0.000s sys 0.000s
W20260809 15:47:04.375738 27896 log.cc:924] Time spent T 0c29937a80724ddb9d07a27534a146f3 P 65c36d164b0540f099d592f38307ee6c: Append to log took a long time: real 0.110s user 0.000s sys 0.000s
I20260809 15:47:04.357082 27197 raft_consensus.cc:504] T c0741ab44d99439abc3aa3bdd27ee160 P 61e69695258f48baa984c9d395e8ce3f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:04.376199 27751 raft_consensus.cc:482] T 563daf42d7784db195ea99c3cb21b370 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
I20260809 15:47:04.377225 27493 leader_election.cc:288] T 8e4ffcf83c234ae7910ddef3ec0b4bda P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:04.378314 27881 consensus_queue.cc:1046] T 0c48a3049897452383257aa757834fb5 P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260809 15:47:04.356061 27677 raft_consensus.cc:3037] T 4b30f28311964a0b9149a50495e83d8f P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:47:04.342572 27913 raft_consensus.cc:504] T 4b30f28311964a0b9149a50495e83d8f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } attrs { replace: true } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } }
I20260809 15:47:04.272966 27395 consensus_queue.cc:237] T d3be1e356b254d22b995be32f1c12f6f P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:04.368548 27780 leader_election.cc:288] T 2424fd9bd7f14e508fd106cbfda786a4 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:04.344036 27582 raft_consensus.cc:504] T 2db057fd10e149a9895dff6ef8c9f038 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:04.395722 23397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "213543cdbe814c89a17a963469922c1b" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
W20260809 15:47:04.395689 23468 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.68:34689, user_credentials={real_user=slave}} blocked reactor thread for 840757us
W20260809 15:47:04.405459 23223 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.68:34689, user_credentials={real_user=slave}} blocked reactor thread for 98235.4us
I20260809 15:47:04.406443 23397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b30f28311964a0b9149a50495e83d8f" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:47:04.407436 23400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cef3e7277d7d4760824a3436ed4a847d" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:04.411120 23400 raft_consensus.cc:2379] T cef3e7277d7d4760824a3436ed4a847d P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 in current term 7: Already voted for candidate c0f38c1908414566aa9115bb2cbdd85d in this term.
I20260809 15:47:04.364317 27661 raft_consensus.cc:504] T 0acc47ce21c3422698d1ef7277c3ed63 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:04.457536 23273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30c1740ef6874df0a332fbd6fd4b03ce" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:04.457996 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0721a4403a440f383318a3cea9d1442" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:04.459939 23277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df20671a7b874dbb98c0ef28371c1308" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:04.460568 23285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cef3e7277d7d4760824a3436ed4a847d" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:04.484316 23285 raft_consensus.cc:2454] T cef3e7277d7d4760824a3436ed4a847d P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 6.
I20260809 15:47:04.486081 23396 raft_consensus.cc:1261] T 6ae14a74a26f434db08c652e097385c8 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Refusing update from remote peer 67b3aeadfa7a432abe8af4d292e04ab9: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260809 15:47:04.484472 23402 raft_consensus.cc:2454] T d0721a4403a440f383318a3cea9d1442 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 6.
I20260809 15:47:04.487479 23394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7715d71dbac447b9351fc5eb36c5e80" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:04.394649 27909 leader_election.cc:288] T 0a51c0baaa6e432997c9d3552af4f254 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:04.488430 23283 raft_consensus.cc:1261] T 2db057fd10e149a9895dff6ef8c9f038 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260809 15:47:04.392494 27493 raft_consensus.cc:482] T 998c6cee72cf40228c6dba064c0db2da P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:47:04.401786 27790 raft_consensus.cc:3032] T 0c48a3049897452383257aa757834fb5 P 61e69695258f48baa984c9d395e8ce3f [term 4 LEADER]: Stepping down as leader of term 4
I20260809 15:47:04.500545 23400 raft_consensus.cc:1261] T 07fcabb569414bddbd39df6cedd24e64 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260809 15:47:04.343164 27939 raft_consensus.cc:504] T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:04.353214 27934 raft_consensus.cc:504] T 297c82ed906d4b48b184985eed3de601 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:04.520658 23519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee02fff2c9c48a19f43eb358a89e7b0" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:04.522641 23519 raft_consensus.cc:2454] T cee02fff2c9c48a19f43eb358a89e7b0 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 7.
I20260809 15:47:04.525511 23529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e17dcb5efe45b8b6a8c42da03b4819" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:04.526856 23522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5712f6b6fffc4cc5a8ddfbe63e4097df" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:04.484225 27936 raft_consensus.cc:482] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
I20260809 15:47:04.394176 27751 raft_consensus.cc:504] T 563daf42d7784db195ea99c3cb21b370 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:04.528208 23277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b30f28311964a0b9149a50495e83d8f" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:47:04.371026 27921 raft_consensus.cc:504] T 3d880de657e544468c82157769622a26 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:04.380978 27906 raft_consensus.cc:504] T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:04.652746 27957 raft_consensus.cc:482] T 6ced718c1a5c4da79cb8c1ebd1435b87 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:47:04.551290 23396 raft_consensus.cc:1261] T 297c82ed906d4b48b184985eed3de601 P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260809 15:47:04.528875 23274 raft_consensus.cc:1261] T 538a228ddfb54bc6a36528d5ff07597b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260809 15:47:04.673635 24452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c48a3049897452383257aa757834fb5" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:47:04.562625 23521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2424fd9bd7f14e508fd106cbfda786a4" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 8 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:04.922734 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "563daf42d7784db195ea99c3cb21b370" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:47:04.542517 23522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c48a3049897452383257aa757834fb5" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:04.495210 27294 raft_consensus.cc:482] T e14d118b2eaa42aa8a7cd4025aba7128 P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
I20260809 15:47:04.562444 23518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "294d4bcc1e7a477cb56455581277a60f" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:04.488330 27780 raft_consensus.cc:482] T 294d4bcc1e7a477cb56455581277a60f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:47:04.528112 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998c6cee72cf40228c6dba064c0db2da" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:04.954109 23277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "294d4bcc1e7a477cb56455581277a60f" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:04.950497 23522 raft_consensus.cc:2379] T 0c48a3049897452383257aa757834fb5 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 in current term 5: Already voted for candidate 65c36d164b0540f099d592f38307ee6c in this term.
I20260809 15:47:04.544981 24410 leader_election.cc:302] T cee02fff2c9c48a19f43eb358a89e7b0 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c; no voters:
W20260809 15:47:04.943318 27305 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.571s user 0.012s sys 0.010s
W20260809 15:47:05.015185 27305 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.645s user 0.013s sys 0.010s
I20260809 15:47:04.500054 27790 raft_consensus.cc:729] T 0c48a3049897452383257aa757834fb5 P 61e69695258f48baa984c9d395e8ce3f [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 61e69695258f48baa984c9d395e8ce3f, State: Running, Role: LEADER
W20260809 15:47:04.568177 27966 log.cc:924] Time spent T 0c48a3049897452383257aa757834fb5 P 61e69695258f48baa984c9d395e8ce3f: Append to log took a long time: real 0.087s user 0.000s sys 0.000s
I20260809 15:47:04.521999 27878 leader_election.cc:288] T cee02fff2c9c48a19f43eb358a89e7b0 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:05.020998 23392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e17dcb5efe45b8b6a8c42da03b4819" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:04.552989 27796 leader_election.cc:288] T 7baa2492df0340c4a5ac241959ce1ac3 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:04.860469 27961 raft_consensus.cc:482] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
I20260809 15:47:04.563050 27908 raft_consensus.cc:482] T 953eea268e17491ca76f3a555860a512 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:04.526711 27493 raft_consensus.cc:504] T 998c6cee72cf40228c6dba064c0db2da P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:05.024333 27945 raft_consensus.cc:482] T 4bcac4b24e7a4448abdc934bba5a624e P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:05.017300 27294 raft_consensus.cc:504] T e14d118b2eaa42aa8a7cd4025aba7128 P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:05.023175 27661 leader_election.cc:288] T 0acc47ce21c3422698d1ef7277c3ed63 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
W20260809 15:47:05.017726 24410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 472799us
I20260809 15:47:05.023222 27762 leader_election.cc:288] T 3296ef50be254fc29cfc77d22a8f7a23 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:05.028609 27751 leader_election.cc:288] T 563daf42d7784db195ea99c3cb21b370 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:05.029990 27906 leader_election.cc:288] T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:04.565354 27880 raft_consensus.cc:482] T 7660dfa2ef694927b8eda4100a2bcd9b P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:47:05.031370 27921 leader_election.cc:288] T 3d880de657e544468c82157769622a26 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
W20260809 15:47:04.978222 27519 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.635s user 0.032s sys 0.062s
I20260809 15:47:05.032815 23400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a51c0baaa6e432997c9d3552af4f254" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:05.033401 23274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5c2e425a74b4e64ae70b5def750434c" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:05.035549 27956 raft_consensus.cc:482] T cee02fff2c9c48a19f43eb358a89e7b0 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:04.885016 27032 raft_consensus.cc:482] T 538a228ddfb54bc6a36528d5ff07597b P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:04.653456 27886 leader_election.cc:288] T 73f6f44c259547f1b05a3a434616fa54 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:04.655738 24463 raft_consensus.cc:1261] T c0741ab44d99439abc3aa3bdd27ee160 P 61e69695258f48baa984c9d395e8ce3f [term 6 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
W20260809 15:47:05.047704 23517 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.955s user 0.000s sys 0.070s
I20260809 15:47:04.884792 26619 raft_consensus.cc:482] T 0acc47ce21c3422698d1ef7277c3ed63 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:47:04.919469 27958 raft_consensus.cc:482] T 3296ef50be254fc29cfc77d22a8f7a23 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
I20260809 15:47:05.067987 23392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "953eea268e17491ca76f3a555860a512" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:47:04.655550 27347 leader_election.cc:288] T 563daf42d7784db195ea99c3cb21b370 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:04.527082 27544 leader_election.cc:288] T 294d4bcc1e7a477cb56455581277a60f P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:04.648720 27893 leader_election.cc:288] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:04.648952 27658 leader_election.cc:288] T 0c48a3049897452383257aa757834fb5 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:04.571622 27903 raft_consensus.cc:482] T d1a134ba33ea4076bbaac0ae460c17ba P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:04.899691 27949 raft_consensus.cc:482] T d2d4f749fcfd4fb3bfb8115af3471598 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:47:04.650087 27197 leader_election.cc:288] T c0741ab44d99439abc3aa3bdd27ee160 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
W20260809 15:47:05.069106 23399 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.566s user 0.014s sys 0.010s
I20260809 15:47:04.656780 27870 leader_election.cc:288] T 953eea268e17491ca76f3a555860a512 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 7 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
W20260809 15:47:05.070176 23470 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.570s user 0.002s sys 0.023s
I20260809 15:47:04.652094 27582 leader_election.cc:288] T 2db057fd10e149a9895dff6ef8c9f038 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:04.671722 27909 raft_consensus.cc:482] T 07fcabb569414bddbd39df6cedd24e64 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:47:05.072480 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f9d6565aa4c45a3b3f01210d0eb71f7" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:47:04.654021 27950 leader_election.cc:288] T ae6d2c08a7a94f58ac8dd698e3fd456c P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:05.073909 24452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e4ffcf83c234ae7910ddef3ec0b4bda" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:47:05.023284 27963 raft_consensus.cc:482] T 73f6f44c259547f1b05a3a434616fa54 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
W20260809 15:47:05.089066 23273 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.593s user 0.007s sys 0.017s
W20260809 15:47:05.090986 23472 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.595s user 0.000s sys 0.013s
W20260809 15:47:05.088914 23270 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.566s user 0.016s sys 0.009s
I20260809 15:47:04.946645 27936 raft_consensus.cc:504] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:05.046734 27918 raft_consensus.cc:482] T d3e94b7e50754a81b3d72a07476cb7b9 P 61e69695258f48baa984c9d395e8ce3f [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
W20260809 15:47:05.080006 27492 consensus_meta.cc:287] T 00a1881267a14ba8b78c81040f17a014 P 65c36d164b0540f099d592f38307ee6c: Time spent flushing consensus metadata: real 1.672s user 0.005s sys 0.006s
I20260809 15:47:05.016966 27780 raft_consensus.cc:504] T 294d4bcc1e7a477cb56455581277a60f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:04.983184 27957 raft_consensus.cc:504] T 6ced718c1a5c4da79cb8c1ebd1435b87 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:05.072876 27878 raft_consensus.cc:482] T d3be1e356b254d22b995be32f1c12f6f P 61e69695258f48baa984c9d395e8ce3f [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:05.376915 27934 leader_election.cc:288] T 297c82ed906d4b48b184985eed3de601 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
W20260809 15:47:05.480816 23399 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.978s user 0.016s sys 0.024s
W20260809 15:47:05.394956 23517 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.302s user 0.000s sys 0.076s
W20260809 15:47:05.379333 23395 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.707s user 0.024s sys 0.001s
W20260809 15:47:05.486701 23395 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.815s user 0.027s sys 0.004s
W20260809 15:47:05.482719 23273 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.987s user 0.010s sys 0.029s
I20260809 15:47:05.090830 27881 raft_consensus.cc:2781] T cee02fff2c9c48a19f43eb358a89e7b0 P 61e69695258f48baa984c9d395e8ce3f [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260809 15:47:05.327998 23286 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53554 (request call id 1678) took 2858 ms (client timeout 2002 ms). Trace:
I20260809 15:47:05.283634 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3296ef50be254fc29cfc77d22a8f7a23" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:47:05.395907 23400 raft_consensus.cc:2454] T 0a51c0baaa6e432997c9d3552af4f254 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 6.
I20260809 15:47:05.094224 27939 leader_election.cc:288] T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:05.483037 23528 raft_consensus.cc:1261] T 00a1881267a14ba8b78c81040f17a014 P 65c36d164b0540f099d592f38307ee6c [term 9 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 9 index: 6. (term mismatch)
W20260809 15:47:05.345074 23469 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.668s user 0.020s sys 0.041s
W20260809 15:47:05.157286 23280 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.581s user 0.009s sys 0.014s
W20260809 15:47:05.144876 24455 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.576s user 0.000s sys 0.021s
W20260809 15:47:05.135514 27518 consensus_meta.cc:287] T d3be1e356b254d22b995be32f1c12f6f P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent flushing consensus metadata: real 1.632s user 0.009s sys 0.002s
W20260809 15:47:05.194537 23351 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.707s user 0.005s sys 0.022s
W20260809 15:47:05.496059 23351 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.009s user 0.009s sys 0.026s
W20260809 15:47:05.395541 27519 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.053s user 0.032s sys 0.078s
W20260809 15:47:05.289417 27695 consensus_meta.cc:287] T 907bacfa79d04a55a68c8dbe60056462 P 61e69695258f48baa984c9d395e8ce3f: Time spent flushing consensus metadata: real 1.029s user 0.007s sys 0.001s
W20260809 15:47:05.502239 23228 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.976s user 0.028s sys 0.069s
W20260809 15:47:05.502557 23228 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.977s user 0.028s sys 0.069s
W20260809 15:47:05.490903 23226 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.505s user 0.023s sys 0.041s
W20260809 15:47:05.503006 23226 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.560s user 0.023s sys 0.041s
I20260809 15:47:05.502452 24452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3e94b7e50754a81b3d72a07476cb7b9" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
W20260809 15:47:05.396052 23472 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.900s user 0.000s sys 0.023s
I20260809 15:47:05.277927 27928 raft_consensus.cc:482] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
W20260809 15:47:05.459276 23468 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.802s user 0.006s sys 0.020s
W20260809 15:47:05.509516 23468 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.852s user 0.007s sys 0.020s
W20260809 15:47:05.488132 23286 rpcz_store.cc:269] 0809 15:47:02.469314 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:02.478268 (+ 8954us) service_pool.cc:227] Handling call
0809 15:47:05.327803 (+2849535us) spinlock_profiling.cc:243] Waited 1.58 s on lock 0x7b54001540d4. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e92242ae 00007f34ee161c27 00007f34e7849c04 00007f34e7849b92 00007f34e7849ac1 00007f34e7849a3c 00007f34e7848cf2 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:47:05.327926 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3641761024}
I20260809 15:47:05.492571 23528 pending_rounds.cc:76] T 00a1881267a14ba8b78c81040f17a014 P 65c36d164b0540f099d592f38307ee6c: Aborting all ops after (but not including) 5
I20260809 15:47:05.511158 23528 pending_rounds.cc:97] T 00a1881267a14ba8b78c81040f17a014 P 65c36d164b0540f099d592f38307ee6c: Aborting uncommitted NO_OP operation due to leader change: 8.6
W20260809 15:47:05.492664 23469 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.816s user 0.022s sys 0.065s
W20260809 15:47:05.492909 23280 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.916s user 0.017s sys 0.028s
W20260809 15:47:05.350574 27704 consensus_meta.cc:287] T 4564dfc6d4094c1b9240b367ccecf3b1 P c0f38c1908414566aa9115bb2cbdd85d: Time spent flushing consensus metadata: real 1.114s user 0.002s sys 0.011s
W20260809 15:47:05.226960 23347 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.859s user 0.019s sys 0.028s
I20260809 15:47:05.500233 23400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7baa2492df0340c4a5ac241959ce1ac3" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:05.500921 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3924b648461e463b9d7925e5671d68ae" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
W20260809 15:47:05.514900 23347 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.151s user 0.024s sys 0.035s
I20260809 15:47:05.498044 23277 raft_consensus.cc:2454] T 294d4bcc1e7a477cb56455581277a60f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 5.
I20260809 15:47:05.152956 24453 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38734 (request call id 354) took 2567 ms. Trace:
I20260809 15:47:05.234031 27969 raft_consensus.cc:482] T 05ad7d8fdca64e3da527280ca5be9280 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
I20260809 15:47:05.503118 23392 raft_consensus.cc:2454] T 953eea268e17491ca76f3a555860a512 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 7.
W20260809 15:47:05.504084 23402 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.833s user 0.007s sys 0.010s
W20260809 15:47:05.575291 23402 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.904s user 0.008s sys 0.010s
I20260809 15:47:05.144335 27906 raft_consensus.cc:482] T 472816994d814cb198b69e5248697440 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
I20260809 15:47:05.315196 27661 raft_consensus.cc:482] T 538a228ddfb54bc6a36528d5ff07597b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 15:47:05.396289 23270 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.895s user 0.016s sys 0.019s
I20260809 15:47:05.505980 27843 raft_consensus.cc:482] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:47:05.508857 27480 raft_consensus.cc:482] T 907bacfa79d04a55a68c8dbe60056462 P 61e69695258f48baa984c9d395e8ce3f [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:05.415913 27032 raft_consensus.cc:504] T 538a228ddfb54bc6a36528d5ff07597b P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:05.234083 24451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "563daf42d7784db195ea99c3cb21b370" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:47:05.481283 27197 raft_consensus.cc:482] T e14d118b2eaa42aa8a7cd4025aba7128 P 61e69695258f48baa984c9d395e8ce3f [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
I20260809 15:47:05.484449 27395 raft_consensus.cc:482] T 213543cdbe814c89a17a963469922c1b P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:47:05.511466 23528 raft_consensus.cc:2866] T 00a1881267a14ba8b78c81040f17a014 P 65c36d164b0540f099d592f38307ee6c [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 15:47:05.512856 23274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee02fff2c9c48a19f43eb358a89e7b0" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:05.490012 26619 raft_consensus.cc:504] T 0acc47ce21c3422698d1ef7277c3ed63 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:05.489570 27881 raft_consensus.cc:482] T cee02fff2c9c48a19f43eb358a89e7b0 P 61e69695258f48baa984c9d395e8ce3f [term 6 FOLLOWER]: Starting leader election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:47:05.493957 27544 raft_consensus.cc:482] T 3d880de657e544468c82157769622a26 P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:47:05.077029 27961 raft_consensus.cc:504] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:05.505373 27518 raft_consensus.cc:2726] T d3be1e356b254d22b995be32f1c12f6f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Leader pre-election lost for term 7. Reason: Vote denied by peer 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689) with higher term. Message: Invalid argument: T d3be1e356b254d22b995be32f1c12f6f P 61e69695258f48baa984c9d395e8ce3f [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 7. Current term is 8.
I20260809 15:47:05.481114 27903 raft_consensus.cc:504] T d1a134ba33ea4076bbaac0ae460c17ba P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:05.589334 23392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998c6cee72cf40228c6dba064c0db2da" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:05.662508 23392 raft_consensus.cc:2454] T 998c6cee72cf40228c6dba064c0db2da P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 7.
I20260809 15:47:05.590631 23400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2424fd9bd7f14e508fd106cbfda786a4" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 8 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:05.093277 27908 raft_consensus.cc:504] T 953eea268e17491ca76f3a555860a512 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:05.415212 27909 raft_consensus.cc:504] T 07fcabb569414bddbd39df6cedd24e64 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:47:05.394862 23470 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.899s user 0.005s sys 0.030s
W20260809 15:47:05.637583 23353 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.793s user 0.056s sys 0.053s
W20260809 15:47:05.500906 24455 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.931s user 0.008s sys 0.038s
I20260809 15:47:05.643836 27493 leader_election.cc:288] T 998c6cee72cf40228c6dba064c0db2da P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:05.094012 27945 raft_consensus.cc:504] T 4bcac4b24e7a4448abdc934bba5a624e P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:05.517309 27519 raft_consensus.cc:482] T 07fcabb569414bddbd39df6cedd24e64 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:05.490330 27878 raft_consensus.cc:504] T d3be1e356b254d22b995be32f1c12f6f P 61e69695258f48baa984c9d395e8ce3f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:05.517131 27751 raft_consensus.cc:482] T ba56725e42f74a1baa4c04d9660727f9 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:47:05.593459 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d880de657e544468c82157769622a26" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
W20260809 15:47:05.616070 24410 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.20.202.66:36263 timed out after 2.111s (SENT) [suppressed 6 similar messages]
I20260809 15:47:05.511039 27582 raft_consensus.cc:482] T 16ca412a0dae4feebe85dd01bf1a9173 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:47:05.153328 27880 raft_consensus.cc:504] T 7660dfa2ef694927b8eda4100a2bcd9b P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:05.583237 27347 raft_consensus.cc:482] T 763e58d6ba644eea830a6c1664d39c9b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:47:05.593504 27892 raft_consensus.cc:482] T 99bdb93503db4de4a1cfb0161ea0c07c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:47:05.644248 27294 leader_election.cc:288] T e14d118b2eaa42aa8a7cd4025aba7128 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:05.508229 24463 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38734 (request call id 362) took 1092 ms. Trace:
W20260809 15:47:05.591898 27987 log.cc:924] Time spent T 28e17dcb5efe45b8b6a8c42da03b4819 P 65c36d164b0540f099d592f38307ee6c: Append to log took a long time: real 0.502s user 0.001s sys 0.000s
I20260809 15:47:05.592918 27796 raft_consensus.cc:482] T a3622f6e3e3641cea1960e3a31498bb7 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
W20260809 15:47:05.688058 23353 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.844s user 0.058s sys 0.060s
I20260809 15:47:05.489787 27949 raft_consensus.cc:504] T d2d4f749fcfd4fb3bfb8115af3471598 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:05.418807 27958 raft_consensus.cc:504] T 3296ef50be254fc29cfc77d22a8f7a23 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:05.516016 27928 raft_consensus.cc:504] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:05.481410 27963 raft_consensus.cc:504] T 73f6f44c259547f1b05a3a434616fa54 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:05.649315 27363 raft_consensus.cc:482] T 00a1881267a14ba8b78c81040f17a014 P 65c36d164b0540f099d592f38307ee6c [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:05.221400 27790 consensus_queue.cc:260] T 0c48a3049897452383257aa757834fb5 P 61e69695258f48baa984c9d395e8ce3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:05.575116 27704 raft_consensus.cc:504] T 4564dfc6d4094c1b9240b367ccecf3b1 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:05.647024 27988 raft_consensus.cc:482] T 0da4d5283c8a46878bd39b85e5c71f66 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:47:05.592767 27843 raft_consensus.cc:504] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:05.491539 27918 raft_consensus.cc:504] T d3e94b7e50754a81b3d72a07476cb7b9 P 61e69695258f48baa984c9d395e8ce3f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:05.656116 27881 raft_consensus.cc:3037] T cee02fff2c9c48a19f43eb358a89e7b0 P 61e69695258f48baa984c9d395e8ce3f [term 6 FOLLOWER]: Advancing to term 7
I20260809 15:47:05.481619 27956 raft_consensus.cc:504] T cee02fff2c9c48a19f43eb358a89e7b0 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:05.649091 27197 raft_consensus.cc:504] T e14d118b2eaa42aa8a7cd4025aba7128 P 61e69695258f48baa984c9d395e8ce3f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:05.582484 23286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e4ffcf83c234ae7910ddef3ec0b4bda" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:05.655122 27497 raft_consensus.cc:482] T 2db057fd10e149a9895dff6ef8c9f038 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:05.777808 27961 leader_election.cc:288] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:05.682592 23402 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55320 (request call id 1537) took 1177 ms. Trace:
I20260809 15:47:05.786700 23400 raft_consensus.cc:2396] T 2424fd9bd7f14e508fd106cbfda786a4 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 for term 8 because replica has last-logged OpId of term: 7 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 2.
I20260809 15:47:05.753091 27939 raft_consensus.cc:482] T 0c29937a80724ddb9d07a27534a146f3 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:05.761379 27518 raft_consensus.cc:482] T 0da4d5283c8a46878bd39b85e5c71f66 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:47:05.821200 23353 leader_election.cc:302] T cef3e7277d7d4760824a3436ed4a847d P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:47:05.644920 27661 raft_consensus.cc:504] T 538a228ddfb54bc6a36528d5ff07597b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:05.777832 27780 leader_election.cc:288] T 294d4bcc1e7a477cb56455581277a60f P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
W20260809 15:47:05.812693 27899 log.cc:924] Time spent T 4564dfc6d4094c1b9240b367ccecf3b1 P 67b3aeadfa7a432abe8af4d292e04ab9: Append to log took a long time: real 1.627s user 0.018s sys 0.038s
W20260809 15:47:05.829144 24403 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.813s user 0.007s sys 0.024s
I20260809 15:47:05.648089 27480 raft_consensus.cc:504] T 907bacfa79d04a55a68c8dbe60056462 P 61e69695258f48baa984c9d395e8ce3f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
W20260809 15:47:05.825325 27972 log.cc:924] Time spent T 3d880de657e544468c82157769622a26 P 65c36d164b0540f099d592f38307ee6c: Append to log took a long time: real 0.479s user 0.000s sys 0.001s
I20260809 15:47:05.650717 27973 raft_consensus.cc:482] T 297c82ed906d4b48b184985eed3de601 P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:05.776919 27945 leader_election.cc:288] T 4bcac4b24e7a4448abdc934bba5a624e P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:05.818837 27294 raft_consensus.cc:482] T 99bdb93503db4de4a1cfb0161ea0c07c P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:47:05.650614 27395 raft_consensus.cc:504] T 213543cdbe814c89a17a963469922c1b P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:47:05.841187 27967 log.cc:924] Time spent T 28e17dcb5efe45b8b6a8c42da03b4819 P c0f38c1908414566aa9115bb2cbdd85d: Append to log took a long time: real 0.508s user 0.001s sys 0.000s
I20260809 15:47:05.858705 23228 leader_election.cc:302] T cef3e7277d7d4760824a3436ed4a847d P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:47:05.769572 27751 raft_consensus.cc:504] T ba56725e42f74a1baa4c04d9660727f9 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:05.769773 27519 raft_consensus.cc:504] T 07fcabb569414bddbd39df6cedd24e64 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:05.863637 23393 raft_consensus.cc:1226] T 4564dfc6d4094c1b9240b367ccecf3b1 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Rejecting Update request from peer 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 6. Current term is 7. Ops: []
I20260809 15:47:05.782313 27347 raft_consensus.cc:504] T 763e58d6ba644eea830a6c1664d39c9b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:05.589478 27969 raft_consensus.cc:504] T 05ad7d8fdca64e3da527280ca5be9280 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:05.893481 27957 leader_election.cc:288] T 6ced718c1a5c4da79cb8c1ebd1435b87 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
W20260809 15:47:05.881633 28006 log.cc:924] Time spent T b3aa6a9a52804df0ba9b0ca3c82f38c3 P c0f38c1908414566aa9115bb2cbdd85d: Append to log took a long time: real 0.099s user 0.000s sys 0.000s
W20260809 15:47:05.882401 28002 log.cc:924] Time spent T d3be1e356b254d22b995be32f1c12f6f P c0f38c1908414566aa9115bb2cbdd85d: Append to log took a long time: real 0.219s user 0.000s sys 0.000s
I20260809 15:47:05.846756 27936 leader_election.cc:288] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:05.787362 27582 raft_consensus.cc:504] T 16ca412a0dae4feebe85dd01bf1a9173 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:05.984364 27197 leader_election.cc:288] T e14d118b2eaa42aa8a7cd4025aba7128 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:05.984669 27661 leader_election.cc:288] T 538a228ddfb54bc6a36528d5ff07597b P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:05.595353 27906 raft_consensus.cc:504] T 472816994d814cb198b69e5248697440 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:05.986644 27906 leader_election.cc:288] T 472816994d814cb198b69e5248697440 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:05.987035 27969 leader_election.cc:288] T 05ad7d8fdca64e3da527280ca5be9280 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:05.987288 27906 raft_consensus.cc:482] T e71a05ba1d4d42bd95c3f85f9313399e P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
I20260809 15:47:05.987805 27906 raft_consensus.cc:504] T e71a05ba1d4d42bd95c3f85f9313399e P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:05.988729 27704 leader_election.cc:288] T 4564dfc6d4094c1b9240b367ccecf3b1 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 7 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:05.989621 27906 leader_election.cc:288] T e71a05ba1d4d42bd95c3f85f9313399e P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:05.874230 23402 rpcz_store.cc:276] 0809 15:47:04.505142 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:04.505309 (+ 167us) service_pool.cc:227] Handling call
0809 15:47:04.550846 (+ 45537us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:47:04.551011 (+ 165us) raft_consensus.cc:1520] Early marking committed up to index 3
0809 15:47:04.671128 (+120117us) log.cc:842] Serialized 14 byte log entry
0809 15:47:05.682290 (+1011162us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 15:47:05.682340 (+ 50us) raft_consensus.cc:1631] Marking committed up to 3
0809 15:47:05.682377 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:47:05.682435 (+ 58us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:47:05.682539 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":106585,"prepare.queue_time_us":119984,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":42,"raft.queue_time_us":194790,"raft.run_cpu_time_us":110,"raft.run_wall_time_us":2975,"spinlock_wait_cycles":25452544,"thread_start_us":904233,"threads_started":1,"wal-append.queue_time_us":1099049}
I20260809 15:47:05.890432 23400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bcac4b24e7a4448abdc934bba5a624e" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:05.991958 23400 raft_consensus.cc:2454] T 4bcac4b24e7a4448abdc934bba5a624e P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 3.
I20260809 15:47:05.891522 23286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73f6f44c259547f1b05a3a434616fa54" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:05.854869 27790 raft_consensus.cc:3037] T 0c48a3049897452383257aa757834fb5 P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Advancing to term 5
I20260809 15:47:06.000500 24451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "907bacfa79d04a55a68c8dbe60056462" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:47:06.001138 24451 raft_consensus.cc:2454] T 907bacfa79d04a55a68c8dbe60056462 P 61e69695258f48baa984c9d395e8ce3f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 7.
I20260809 15:47:05.786002 27892 raft_consensus.cc:504] T 99bdb93503db4de4a1cfb0161ea0c07c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:05.895972 23274 raft_consensus.cc:2379] T cee02fff2c9c48a19f43eb358a89e7b0 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 61e69695258f48baa984c9d395e8ce3f in current term 7: Already voted for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in this term.
I20260809 15:47:05.897660 27980 raft_consensus.cc:482] T 7baa2492df0340c4a5ac241959ce1ac3 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:47:05.898641 24452 raft_consensus.cc:2454] T d3e94b7e50754a81b3d72a07476cb7b9 P 61e69695258f48baa984c9d395e8ce3f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 7.
I20260809 15:47:05.828356 27796 raft_consensus.cc:504] T a3622f6e3e3641cea1960e3a31498bb7 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:47:05.899727 24410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 283672us
I20260809 15:47:05.917249 23517 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44018 (request call id 286) took 2951 ms. Trace:
W20260809 15:47:05.927985 28008 log.cc:924] Time spent T 2a823d35f4f348d1b52ba9e6d80b00d2 P 65c36d164b0540f099d592f38307ee6c: Append to log took a long time: real 0.074s user 0.000s sys 0.000s
I20260809 15:47:05.888527 27493 raft_consensus.cc:482] T 2424fd9bd7f14e508fd106cbfda786a4 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:47:05.871510 27497 raft_consensus.cc:504] T 2db057fd10e149a9895dff6ef8c9f038 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:05.894302 27939 raft_consensus.cc:504] T 0c29937a80724ddb9d07a27534a146f3 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:05.926071 27935 raft_consensus.cc:482] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 65c36d164b0540f099d592f38307ee6c [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
W20260809 15:47:05.929114 28007 log.cc:924] Time spent T ba56725e42f74a1baa4c04d9660727f9 P c0f38c1908414566aa9115bb2cbdd85d: Append to log took a long time: real 0.081s user 0.000s sys 0.000s
I20260809 15:47:05.931381 23280 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53562 (request call id 1856) took 1472 ms. Trace:
I20260809 15:47:05.846194 27363 raft_consensus.cc:504] T 00a1881267a14ba8b78c81040f17a014 P 65c36d164b0540f099d592f38307ee6c [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:06.020397 27996 raft_consensus.cc:482] T 907bacfa79d04a55a68c8dbe60056462 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:06.021292 23394 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55320 (request call id 1540) took 1031 ms. Trace:
I20260809 15:47:05.662892 27544 raft_consensus.cc:504] T 3d880de657e544468c82157769622a26 P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:05.889096 27404 raft_consensus.cc:482] T 30c1740ef6874df0a332fbd6fd4b03ce P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 15:47:05.943850 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 85191.2us
I20260809 15:47:06.015130 27493 raft_consensus.cc:504] T 2424fd9bd7f14e508fd106cbfda786a4 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:05.784804 24463 rpcz_store.cc:276] 0809 15:47:04.405242 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:04.407303 (+ 2061us) service_pool.cc:227] Handling call
0809 15:47:04.527593 (+120290us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:47:05.427158 (+899565us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:47:05.497734 (+ 70576us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:47:05.517818 24453 rpcz_store.cc:276] 0809 15:47:02.525401 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:02.543766 (+ 18365us) service_pool.cc:227] Handling call
0809 15:47:02.731326 (+187560us) spinlock_profiling.cc:243] Waited 7.03 ms on lock 0x7b5800005498. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e613cf11 00007f34e62c0851 00007f34e62c0702 00007f34ef1b0b09 00007f34ee0ec730 00007f34ee1b279b 00007f34ee1b2821 00007f34ee1762b0 00007f34ee1617f9 00007f34ee1612a2 00007f34e7847684
0809 15:47:02.731547 (+ 221us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 15:47:02.759492 (+ 27945us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:47:02.759571 (+ 79us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 15:47:02.760195 (+ 624us) log.cc:842] Serialized 26 byte log entry
0809 15:47:02.946288 (+186093us) raft_consensus.cc:1631] Marking committed up to 8
0809 15:47:03.425038 (+478750us) log.cc:842] Serialized 14 byte log entry
0809 15:47:03.459830 (+ 34792us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:47:05.040079 (+1580249us) spinlock_profiling.cc:243] Waited 25.3 ms on lock 0x7b54005238a0. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e91b0730 00007f34e9223b42 00007f34e9220f2e 00007f34e921f99e 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772
0809 15:47:05.046810 (+ 6731us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 15:47:05.071704 (+ 24894us) raft_consensus.cc:1671] finished
0809 15:47:05.071719 (+ 15us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:47:05.092974 (+ 21255us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":31002,"prepare.queue_time_us":576156,"prepare.run_cpu_time_us":30,"prepare.run_wall_time_us":32,"raft.queue_time_us":32986,"raft.run_cpu_time_us":198,"raft.run_wall_time_us":2841,"spinlock_wait_cycles":74477824,"thread_start_us":321703,"threads_started":2,"wal-append.queue_time_us":395802,"wal-append.run_cpu_time_us":2634,"wal-append.run_wall_time_us":1360274}
I20260809 15:47:05.896234 27518 raft_consensus.cc:504] T 0da4d5283c8a46878bd39b85e5c71f66 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:06.041447 27518 leader_election.cc:288] T 0da4d5283c8a46878bd39b85e5c71f66 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:05.945879 27961 raft_consensus.cc:2781] T cef3e7277d7d4760824a3436ed4a847d P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260809 15:47:05.985178 27945 raft_consensus.cc:2781] T cef3e7277d7d4760824a3436ed4a847d P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260809 15:47:06.042044 27996 raft_consensus.cc:504] T 907bacfa79d04a55a68c8dbe60056462 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:06.046990 27945 raft_consensus.cc:482] T cef3e7277d7d4760824a3436ed4a847d P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:47:06.047150 27694 raft_consensus.cc:482] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 61e69695258f48baa984c9d395e8ce3f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:06.047451 27945 raft_consensus.cc:3037] T cef3e7277d7d4760824a3436ed4a847d P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Advancing to term 8
I20260809 15:47:06.048146 27880 leader_election.cc:288] T 7660dfa2ef694927b8eda4100a2bcd9b P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:06.049677 27544 leader_election.cc:288] T 3d880de657e544468c82157769622a26 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:05.993139 23286 raft_consensus.cc:2454] T 73f6f44c259547f1b05a3a434616fa54 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 7.
W20260809 15:47:05.893049 24403 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.281s user 0.007s sys 0.024s
I20260809 15:47:05.985073 27582 leader_election.cc:288] T 16ca412a0dae4feebe85dd01bf1a9173 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:05.930707 27973 raft_consensus.cc:504] T 297c82ed906d4b48b184985eed3de601 P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:06.054503 23286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "563daf42d7784db195ea99c3cb21b370" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:06.009428 23517 rpcz_store.cc:276] 0809 15:47:02.914254 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:02.914650 (+ 396us) service_pool.cc:227] Handling call
0809 15:47:02.914927 (+ 277us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 15:47:02.937177 (+ 22250us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:47:02.937205 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 15:47:02.941285 (+ 4080us) log.cc:842] Serialized 26 byte log entry
0809 15:47:03.065883 (+124598us) spinlock_profiling.cc:243] Waited 4.3 ms on lock 0x7b18002ecc60. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34ef1978b1 00007f34ef1a4661 00007f34e62da345 00007f34ef111a39 00007f34e62f78bb 00007f34e62f5cbf 00007f34e62f7e22 00007f34e8ea15bb 00007f34e8ea7063 00007f34e8ea7411 00007f34e91f45cf
0809 15:47:04.081551 (+1015668us) raft_consensus.cc:1631] Marking committed up to 8
0809 15:47:05.486476 (+1404925us) log.cc:842] Serialized 14 byte log entry
0809 15:47:05.849952 (+363476us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:47:05.850081 (+ 129us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 15:47:05.850126 (+ 45us) raft_consensus.cc:1671] finished
0809 15:47:05.865451 (+ 15325us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:47:05.865891 (+ 440us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":96450,"prepare.queue_time_us":156772,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":35,"raft.queue_time_us":26984,"raft.run_cpu_time_us":121,"raft.run_wall_time_us":23505,"spinlock_wait_cycles":9896832,"thread_start_us":2771923,"threads_started":4,"wal-append.queue_time_us":1624609,"wal-append.run_cpu_time_us":1659,"wal-append.run_wall_time_us":1041664}
I20260809 15:47:06.006880 27980 raft_consensus.cc:504] T 7baa2492df0340c4a5ac241959ce1ac3 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:05.985239 26619 leader_election.cc:288] T 0acc47ce21c3422698d1ef7277c3ed63 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:47:05.942736 27294 raft_consensus.cc:504] T 99bdb93503db4de4a1cfb0161ea0c07c P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:05.981325 27032 leader_election.cc:288] T 538a228ddfb54bc6a36528d5ff07597b P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:47:05.854058 27988 raft_consensus.cc:504] T 0da4d5283c8a46878bd39b85e5c71f66 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:06.017926 23280 rpcz_store.cc:276] 0809 15:47:04.458962 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:04.459069 (+ 107us) service_pool.cc:227] Handling call
0809 15:47:04.459386 (+ 317us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 15:47:04.488876 (+ 29490us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:47:04.488904 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 15:47:04.497309 (+ 8405us) log.cc:842] Serialized 26 byte log entry
0809 15:47:05.513287 (+1015978us) raft_consensus.cc:1631] Marking committed up to 4
0809 15:47:05.513363 (+ 76us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:47:05.513509 (+ 146us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 15:47:05.931172 (+417663us) raft_consensus.cc:1671] finished
0809 15:47:05.931193 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:47:05.931325 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":49803,"raft.queue_time_us":62305,"raft.run_cpu_time_us":105,"raft.run_wall_time_us":18739,"thread_start_us":936420,"threads_started":1,"wal-append.queue_time_us":1311569}
I20260809 15:47:06.019699 27878 leader_election.cc:288] T d3be1e356b254d22b995be32f1c12f6f P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:47:05.953822 27881 raft_consensus.cc:504] T cee02fff2c9c48a19f43eb358a89e7b0 P 61e69695258f48baa984c9d395e8ce3f [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:06.021643 23394 rpcz_store.cc:276] 0809 15:47:04.990074 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:04.990267 (+ 193us) service_pool.cc:227] Handling call
0809 15:47:05.047400 (+ 57133us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 15:47:05.069983 (+ 22583us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:47:05.070013 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 15:47:05.070389 (+ 376us) log.cc:842] Serialized 26 byte log entry
0809 15:47:05.515351 (+444962us) raft_consensus.cc:1631] Marking committed up to 2
0809 15:47:05.515391 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:47:05.515465 (+ 74us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 15:47:06.021108 (+505643us) raft_consensus.cc:1671] finished
0809 15:47:06.021125 (+ 17us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:47:06.021238 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":367159,"raft.run_cpu_time_us":124,"raft.run_wall_time_us":7170,"thread_start_us":354839,"threads_started":1,"wal-append.queue_time_us":690619}
I20260809 15:47:06.026109 28023 raft_consensus.cc:482] T ae6d2c08a7a94f58ac8dd698e3fd456c P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:47:06.064175 27633 raft_consensus.cc:482] T 4b30f28311964a0b9149a50495e83d8f P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:06.013790 27661 raft_consensus.cc:482] T c5c2e425a74b4e64ae70b5def750434c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:47:06.064734 27633 raft_consensus.cc:504] T 4b30f28311964a0b9149a50495e83d8f P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } attrs { replace: true } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } }
I20260809 15:47:06.065131 27661 raft_consensus.cc:504] T c5c2e425a74b4e64ae70b5def750434c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:06.026049 27404 raft_consensus.cc:504] T 30c1740ef6874df0a332fbd6fd4b03ce P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:06.027194 27796 leader_election.cc:288] T a3622f6e3e3641cea1960e3a31498bb7 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:06.026141 27936 consensus_queue.cc:1035] T 6ae14a74a26f434db08c652e097385c8 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.002s
I20260809 15:47:05.982712 27963 leader_election.cc:288] T 73f6f44c259547f1b05a3a434616fa54 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:06.046103 27961 raft_consensus.cc:482] T cef3e7277d7d4760824a3436ed4a847d P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:47:06.030257 27761 leader_election.cc:288] T d2d4f749fcfd4fb3bfb8115af3471598 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:06.031023 27958 leader_election.cc:288] T 3296ef50be254fc29cfc77d22a8f7a23 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:06.031558 27903 leader_election.cc:288] T d1a134ba33ea4076bbaac0ae460c17ba P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:06.037192 23228 leader_election.cc:302] T 0c48a3049897452383257aa757834fb5 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c
I20260809 15:47:06.038092 27908 leader_election.cc:288] T 953eea268e17491ca76f3a555860a512 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:47:06.038885 24453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "294d4bcc1e7a477cb56455581277a60f" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:47:06.011324 24468 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38724 (request call id 533) took 1927 ms. Trace:
I20260809 15:47:06.045384 24472 raft_consensus.cc:2454] T 3d880de657e544468c82157769622a26 P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 3.
I20260809 15:47:06.027514 27519 leader_election.cc:288] T 07fcabb569414bddbd39df6cedd24e64 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:06.024487 27909 leader_election.cc:288] T 07fcabb569414bddbd39df6cedd24e64 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:06.055071 23286 raft_consensus.cc:2454] T 563daf42d7784db195ea99c3cb21b370 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 5.
I20260809 15:47:06.058348 23280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998c6cee72cf40228c6dba064c0db2da" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:06.036257 24451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bcac4b24e7a4448abdc934bba5a624e" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:47:06.056515 28000 raft_consensus.cc:482] T 3924b648461e463b9d7925e5671d68ae P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
I20260809 15:47:06.060142 26619 raft_consensus.cc:482] T df20671a7b874dbb98c0ef28371c1308 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
I20260809 15:47:06.061803 27294 leader_election.cc:288] T 99bdb93503db4de4a1cfb0161ea0c07c P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:47:06.062273 27032 raft_consensus.cc:482] T cef3e7277d7d4760824a3436ed4a847d P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:06.063938 27881 leader_election.cc:288] T cee02fff2c9c48a19f43eb358a89e7b0 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 7 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:06.275316 27761 raft_consensus.cc:3037] T 0c48a3049897452383257aa757834fb5 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Advancing to term 5
I20260809 15:47:06.276880 24468 rpcz_store.cc:276] 0809 15:47:04.083591 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:04.084037 (+ 446us) service_pool.cc:227] Handling call
0809 15:47:05.927997 (+1843960us) spinlock_profiling.cc:243] Waited 58.7 ms on lock 0x7b5800005498. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e613cf11 00007f34e62c0851 00007f34e62c0702 00007f34ef1b0b09 00007f34ee0ec730 00007f34ee1b279b 00007f34ee1b2821 00007f34ee1762b0 00007f34ee1617f9 00007f34ee1612a2 00007f34e7847684
0809 15:47:05.939196 (+ 11199us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:47:05.946251 (+ 7055us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:47:05.954731 (+ 8480us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 15:47:05.966590 (+ 11859us) raft_consensus.cc:1631] Marking committed up to 7
0809 15:47:05.975942 (+ 9352us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:47:05.983430 (+ 7488us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:47:06.005220 (+ 21790us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":135072128}
I20260809 15:47:06.381659 27032 raft_consensus.cc:504] T cef3e7277d7d4760824a3436ed4a847d P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:06.391901 24451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16ca412a0dae4feebe85dd01bf1a9173" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:47:06.392910 27032 leader_election.cc:288] T cef3e7277d7d4760824a3436ed4a847d P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:47:06.319569 24463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e14d118b2eaa42aa8a7cd4025aba7128" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:47:06.052107 28024 raft_consensus.cc:482] T 4564dfc6d4094c1b9240b367ccecf3b1 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
I20260809 15:47:06.064493 28023 raft_consensus.cc:504] T ae6d2c08a7a94f58ac8dd698e3fd456c P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:06.029111 27347 leader_election.cc:288] T 763e58d6ba644eea830a6c1664d39c9b P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:06.020654 27935 raft_consensus.cc:504] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 65c36d164b0540f099d592f38307ee6c [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
W20260809 15:47:06.122125 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 85014.5us
I20260809 15:47:06.408368 23228 leader_election.cc:302] T 213543cdbe814c89a17a963469922c1b P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:06.268622 27973 leader_election.cc:288] T 297c82ed906d4b48b184985eed3de601 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
W20260809 15:47:06.269447 24406 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 75186.8us
W20260809 15:47:06.270180 28004 log.cc:924] Time spent T 30c1740ef6874df0a332fbd6fd4b03ce P 67b3aeadfa7a432abe8af4d292e04ab9: Append to log took a long time: real 0.117s user 0.000s sys 0.000s
I20260809 15:47:06.058846 27544 raft_consensus.cc:482] T 16ca412a0dae4feebe85dd01bf1a9173 P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
W20260809 15:47:06.077734 27677 consensus_meta.cc:287] T 4b30f28311964a0b9149a50495e83d8f P c0f38c1908414566aa9115bb2cbdd85d: Time spent flushing consensus metadata: real 1.687s user 0.009s sys 0.010s
I20260809 15:47:06.395517 27404 leader_election.cc:288] T 30c1740ef6874df0a332fbd6fd4b03ce P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:47:06.084369 27961 raft_consensus.cc:3037] T cef3e7277d7d4760824a3436ed4a847d P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Advancing to term 7
I20260809 15:47:06.318620 23280 raft_consensus.cc:2454] T 998c6cee72cf40228c6dba064c0db2da P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 7.
I20260809 15:47:06.348188 23395 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55332 (request call id 1418) took 1952 ms. Trace:
I20260809 15:47:06.438556 27781 raft_consensus.cc:2726] T 213543cdbe814c89a17a963469922c1b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260809 15:47:06.078603 27661 leader_election.cc:288] T c5c2e425a74b4e64ae70b5def750434c P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:06.349584 23286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f9d6565aa4c45a3b3f01210d0eb71f7" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:06.319875 24453 raft_consensus.cc:2454] T 294d4bcc1e7a477cb56455581277a60f P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 5.
I20260809 15:47:06.386264 28000 raft_consensus.cc:504] T 3924b648461e463b9d7925e5671d68ae P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:06.353003 27395 leader_election.cc:288] T 213543cdbe814c89a17a963469922c1b P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:06.069268 27694 raft_consensus.cc:504] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 61e69695258f48baa984c9d395e8ce3f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:06.404083 28024 raft_consensus.cc:504] T 4564dfc6d4094c1b9240b367ccecf3b1 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:06.287914 27892 leader_election.cc:288] T 99bdb93503db4de4a1cfb0161ea0c07c P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:06.066725 27633 leader_election.cc:288] T 4b30f28311964a0b9149a50495e83d8f P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:47:06.436839 23228 leader_election.cc:302] T d0721a4403a440f383318a3cea9d1442 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c
I20260809 15:47:06.381596 26619 raft_consensus.cc:504] T df20671a7b874dbb98c0ef28371c1308 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:06.439729 23280 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55434 (request call id 271) took 1949 ms. Trace:
W20260809 15:47:06.618631 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 181846us
I20260809 15:47:06.465991 27996 leader_election.cc:288] T 907bacfa79d04a55a68c8dbe60056462 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:06.456478 27544 raft_consensus.cc:504] T 16ca412a0dae4feebe85dd01bf1a9173 P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:06.476929 23397 raft_consensus.cc:2454] T 4b30f28311964a0b9149a50495e83d8f P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 4.
I20260809 15:47:06.121122 27949 leader_election.cc:288] T d2d4f749fcfd4fb3bfb8115af3471598 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:06.402254 27956 leader_election.cc:288] T cee02fff2c9c48a19f43eb358a89e7b0 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:06.438782 23395 rpcz_store.cc:276] 0809 15:47:04.396055 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:04.396215 (+ 160us) service_pool.cc:227] Handling call
0809 15:47:04.455284 (+ 59069us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 15:47:04.489978 (+ 34694us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:47:04.490006 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 15:47:04.504687 (+ 14681us) log.cc:842] Serialized 26 byte log entry
0809 15:47:05.487050 (+982363us) raft_consensus.cc:1631] Marking committed up to 2
0809 15:47:05.487076 (+ 26us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:47:05.487120 (+ 44us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 15:47:06.347988 (+860868us) raft_consensus.cc:1671] finished
0809 15:47:06.348009 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:47:06.348128 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":443865,"raft.run_cpu_time_us":119,"raft.run_wall_time_us":121,"thread_start_us":814942,"threads_started":1,"wal-append.queue_time_us":1343548}
I20260809 15:47:06.618767 23280 rpcz_store.cc:276] 0809 15:47:04.489979 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:06.438978 (+1948999us) service_pool.cc:227] Handling call
0809 15:47:06.439252 (+ 274us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:47:06.439445 (+ 193us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:47:06.439469 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 15:47:06.439505 (+ 36us) raft_consensus.cc:1631] Marking committed up to 5
0809 15:47:06.439538 (+ 33us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:47:06.439586 (+ 48us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:47:06.439679 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:47:06.619755 23228 leader_election.cc:302] T f7715d71dbac447b9351fc5eb36c5e80 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:06.463562 27939 leader_election.cc:288] T 0c29937a80724ddb9d07a27534a146f3 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:06.661662 23228 leader_election.cc:302] T 28e17dcb5efe45b8b6a8c42da03b4819 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:06.279670 27843 leader_election.cc:288] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:06.440131 27497 leader_election.cc:288] T 2db057fd10e149a9895dff6ef8c9f038 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:06.667385 27892 raft_consensus.cc:2726] T f7715d71dbac447b9351fc5eb36c5e80 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260809 15:47:06.668001 27984 raft_consensus.cc:2781] T d0721a4403a440f383318a3cea9d1442 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260809 15:47:06.668609 27984 raft_consensus.cc:482] T d0721a4403a440f383318a3cea9d1442 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:47:06.669083 23228 leader_election.cc:302] T 0a51c0baaa6e432997c9d3552af4f254 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:47:06.669236 27984 raft_consensus.cc:3037] T d0721a4403a440f383318a3cea9d1442 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Advancing to term 7
I20260809 15:47:06.669931 23228 leader_election.cc:302] T 953eea268e17491ca76f3a555860a512 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:47:06.670663 27661 raft_consensus.cc:2726] T 28e17dcb5efe45b8b6a8c42da03b4819 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260809 15:47:06.675825 27892 raft_consensus.cc:2781] T 953eea268e17491ca76f3a555860a512 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Leader election won for term 7
I20260809 15:47:06.676573 27892 raft_consensus.cc:686] T 953eea268e17491ca76f3a555860a512 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 LEADER]: Becoming Leader. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Running, Role: LEADER
I20260809 15:47:06.677894 27892 consensus_queue.cc:237] T 953eea268e17491ca76f3a555860a512 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:47:06.683202 23397 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55320 (request call id 1533) took 2277 ms (client timeout 1522 ms). Trace:
W20260809 15:47:06.683629 23397 rpcz_store.cc:269] 0809 15:47:04.406131 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:04.406234 (+ 103us) service_pool.cc:227] Handling call
0809 15:47:06.682946 (+2276712us) spinlock_profiling.cc:243] Waited 2.05 s on lock 0x7b5400193f54. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e92242ae 00007f34ee161c27 00007f34e7849c04 00007f34e7849b92 00007f34e7849ac1 00007f34e7849a3c 00007f34e7848cf2 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:47:06.683065 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4715213696}
I20260809 15:47:06.684108 26619 leader_election.cc:288] T df20671a7b874dbb98c0ef28371c1308 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:47:06.685945 24451 raft_consensus.cc:2454] T 16ca412a0dae4feebe85dd01bf1a9173 P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 5.
I20260809 15:47:06.687510 27913 leader_election.cc:288] T 4b30f28311964a0b9149a50495e83d8f P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:06.687953 27544 leader_election.cc:288] T 16ca412a0dae4feebe85dd01bf1a9173 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:06.688692 23397 raft_consensus.cc:1261] T 0a51c0baaa6e432997c9d3552af4f254 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Refusing update from remote peer 61e69695258f48baa984c9d395e8ce3f: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260809 15:47:06.673454 23280 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53554 (request call id 1704) took 2176 ms. Trace:
I20260809 15:47:06.691434 23280 rpcz_store.cc:276] 0809 15:47:04.497032 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:06.671077 (+2174045us) service_pool.cc:227] Handling call
0809 15:47:06.672989 (+ 1912us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:47:06.673160 (+ 171us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:47:06.673184 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 15:47:06.673223 (+ 39us) raft_consensus.cc:1631] Marking committed up to 2
0809 15:47:06.673253 (+ 30us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:47:06.673299 (+ 46us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:47:06.673393 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:47:06.692623 23228 leader_election.cc:302] T 998c6cee72cf40228c6dba064c0db2da P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c
I20260809 15:47:06.695573 23228 leader_election.cc:302] T 2424fd9bd7f14e508fd106cbfda786a4 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:06.699733 27913 raft_consensus.cc:2726] T 2424fd9bd7f14e508fd106cbfda786a4 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260809 15:47:06.700472 28023 leader_election.cc:288] T ae6d2c08a7a94f58ac8dd698e3fd456c P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:47:06.707365 23285 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55434 (request call id 268) took 2218 ms. Trace:
I20260809 15:47:06.707715 23285 rpcz_store.cc:276] 0809 15:47:04.489219 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:04.492188 (+ 2969us) service_pool.cc:227] Handling call
0809 15:47:04.492454 (+ 266us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:47:05.491192 (+998738us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:47:05.491220 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 15:47:05.491255 (+ 35us) raft_consensus.cc:1631] Marking committed up to 6
0809 15:47:05.491305 (+ 50us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:47:06.707137 (+1215832us) spinlock_profiling.cc:243] Waited 998 ms on lock 0x7b54003a9dd4. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e9220f51 00007f34e921f99e 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772 00007f34e764a94c 00007f34e7649d86
0809 15:47:06.707181 (+ 44us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:47:06.707308 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2296505088}
I20260809 15:47:06.663996 27518 consensus_queue.cc:1035] T 6ae14a74a26f434db08c652e097385c8 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.045s
I20260809 15:47:06.709511 23285 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53554 (request call id 1708) took 1692 ms. Trace:
I20260809 15:47:06.709836 23285 rpcz_store.cc:276] 0809 15:47:05.016714 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:06.708763 (+1692049us) service_pool.cc:227] Handling call
0809 15:47:06.709094 (+ 331us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:47:06.709240 (+ 146us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:47:06.709263 (+ 23us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 15:47:06.709299 (+ 36us) raft_consensus.cc:1631] Marking committed up to 6
0809 15:47:06.709327 (+ 28us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:47:06.709375 (+ 48us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:47:06.709465 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:47:06.624276 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f9d6565aa4c45a3b3f01210d0eb71f7" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:47:06.719908 23285 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53554 (request call id 1709) took 1696 ms. Trace:
I20260809 15:47:06.720273 23285 rpcz_store.cc:276] 0809 15:47:05.023358 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:06.710769 (+1687411us) service_pool.cc:227] Handling call
0809 15:47:06.711048 (+ 279us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:47:06.711200 (+ 152us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:47:06.711224 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 15:47:06.719634 (+ 8410us) raft_consensus.cc:1631] Marking committed up to 6
0809 15:47:06.719681 (+ 47us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:47:06.719741 (+ 60us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:47:06.719847 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 15:47:06.721074 23469 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.835s user 0.042s sys 0.073s
W20260809 15:47:06.721490 23469 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.836s user 0.042s sys 0.073s
I20260809 15:47:06.722651 23517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0741ab44d99439abc3aa3bdd27ee160" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:06.500841 27980 leader_election.cc:288] T 7baa2492df0340c4a5ac241959ce1ac3 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:47:06.470041 27945 raft_consensus.cc:504] T cef3e7277d7d4760824a3436ed4a847d P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:06.722229 23285 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53554 (request call id 1710) took 1698 ms. Trace:
I20260809 15:47:06.728510 23285 rpcz_store.cc:276] 0809 15:47:05.023707 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:06.721453 (+1697746us) service_pool.cc:227] Handling call
0809 15:47:06.721736 (+ 283us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:47:06.721892 (+ 156us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:47:06.721970 (+ 78us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 15:47:06.722008 (+ 38us) raft_consensus.cc:1631] Marking committed up to 3
0809 15:47:06.722038 (+ 30us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:47:06.722091 (+ 53us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:47:06.722183 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1280}
I20260809 15:47:06.736774 23522 raft_consensus.cc:1226] T 0c48a3049897452383257aa757834fb5 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Rejecting Update request from peer 61e69695258f48baa984c9d395e8ce3f for earlier term 4. Current term is 5. Ops: [4.7-4.7]
I20260809 15:47:06.725060 27507 raft_consensus.cc:482] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:06.799621 23517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7660dfa2ef694927b8eda4100a2bcd9b" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:07.068022 23529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16ca412a0dae4feebe85dd01bf1a9173" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:06.722648 27661 raft_consensus.cc:2781] T 998c6cee72cf40228c6dba064c0db2da P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260809 15:47:06.800756 23518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e14d118b2eaa42aa8a7cd4025aba7128" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:07.083829 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee02fff2c9c48a19f43eb358a89e7b0" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:47:07.085984 23519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3aa6a9a52804df0ba9b0ca3c82f38c3" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:07.163488 23529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7baa2492df0340c4a5ac241959ce1ac3" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:06.741158 23285 raft_consensus.cc:1261] T c0741ab44d99439abc3aa3bdd27ee160 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260809 15:47:07.085314 23521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5712f6b6fffc4cc5a8ddfbe63e4097df" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:07.164080 23529 raft_consensus.cc:2454] T 7baa2492df0340c4a5ac241959ce1ac3 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 4.
I20260809 15:47:07.164768 23285 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53554 (request call id 1711) took 2140 ms. Trace:
I20260809 15:47:07.165124 23285 rpcz_store.cc:276] 0809 15:47:05.024002 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:06.740590 (+1716588us) service_pool.cc:227] Handling call
0809 15:47:06.740866 (+ 276us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:47:07.164523 (+423657us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:47:07.164713 (+ 190us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:47:07.048679 23399 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55320 (request call id 1534) took 2637 ms. Trace:
I20260809 15:47:07.167335 23399 rpcz_store.cc:276] 0809 15:47:04.407647 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:04.408403 (+ 756us) service_pool.cc:227] Handling call
0809 15:47:04.408701 (+ 298us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 15:47:05.486064 (+1077363us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:47:05.486094 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0809 15:47:05.503888 (+ 17794us) log.cc:842] Serialized 52 byte log entry
0809 15:47:05.786964 (+283076us) raft_consensus.cc:1631] Marking committed up to 3
0809 15:47:05.931584 (+144620us) log.cc:842] Serialized 14 byte log entry
0809 15:47:06.861893 (+930309us) spinlock_profiling.cc:243] Waited 6.46 ms on lock 0x7b540006e620. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e91b3fb3 00007f34e9220f0f 00007f34e921f99e 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772 00007f34e764a94c
0809 15:47:06.861961 (+ 68us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:47:06.862041 (+ 80us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 15:47:06.862085 (+ 44us) raft_consensus.cc:1671] finished
0809 15:47:07.044571 (+182486us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:47:07.044786 (+ 215us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":47414,"prepare.queue_time_us":688001,"prepare.run_cpu_time_us":29,"prepare.run_wall_time_us":30,"raft.queue_time_us":668072,"raft.run_cpu_time_us":208,"raft.run_wall_time_us":20000,"spinlock_wait_cycles":14869248,"thread_start_us":1277792,"threads_started":3,"wal-append.queue_time_us":264930,"wal-append.run_cpu_time_us":2844,"wal-append.run_wall_time_us":1260336}
I20260809 15:47:06.832858 27761 raft_consensus.cc:2726] T 0c48a3049897452383257aa757834fb5 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260809 15:47:06.730355 27694 leader_election.cc:288] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
W20260809 15:47:06.818851 27925 log.cc:924] Time spent T 00000000000000000000000000000000 P a86a2513b8024a869d80303dcbf44163: Append to log took a long time: real 2.542s user 0.012s sys 0.040s
I20260809 15:47:07.178588 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d880de657e544468c82157769622a26" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:07.185878 23286 raft_consensus.cc:1226] T 0c48a3049897452383257aa757834fb5 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Rejecting Update request from peer 61e69695258f48baa984c9d395e8ce3f for earlier term 4. Current term is 5. Ops: []
W20260809 15:47:07.120594 27961 consensus_meta.cc:287] T cef3e7277d7d4760824a3436ed4a847d P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent flushing consensus metadata: real 0.682s user 0.003s sys 0.007s
I20260809 15:47:06.728492 28024 leader_election.cc:288] T 4564dfc6d4094c1b9240b367ccecf3b1 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:47:06.811676 27988 leader_election.cc:288] T 0da4d5283c8a46878bd39b85e5c71f66 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:06.800666 27363 leader_election.cc:288] T 00a1881267a14ba8b78c81040f17a014 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:47:07.198724 23518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "953eea268e17491ca76f3a555860a512" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:47:07.056237 27945 leader_election.cc:288] T cef3e7277d7d4760824a3436ed4a847d P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 8 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:07.199280 27961 raft_consensus.cc:504] T cef3e7277d7d4760824a3436ed4a847d P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:07.125898 27954 raft_consensus.cc:482] T 7660dfa2ef694927b8eda4100a2bcd9b P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:47:07.174640 27507 raft_consensus.cc:504] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:07.206993 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae6d2c08a7a94f58ac8dd698e3fd456c" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:07.221284 23527 raft_consensus.cc:2454] T ae6d2c08a7a94f58ac8dd698e3fd456c P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 7.
I20260809 15:47:07.222432 23527 raft_consensus.cc:1261] T f9e7cadcd24d45fcbfa84209225e8f2d P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Refusing update from remote peer 67b3aeadfa7a432abe8af4d292e04ab9: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260809 15:47:07.212867 23526 raft_consensus.cc:2454] T cee02fff2c9c48a19f43eb358a89e7b0 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 7.
I20260809 15:47:07.163376 27661 raft_consensus.cc:482] T 998c6cee72cf40228c6dba064c0db2da P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:47:07.215128 23529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0acc47ce21c3422698d1ef7277c3ed63" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:07.224975 23529 raft_consensus.cc:2454] T 0acc47ce21c3422698d1ef7277c3ed63 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 7.
I20260809 15:47:07.215755 23518 raft_consensus.cc:2379] T 953eea268e17491ca76f3a555860a512 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 in current term 7: Already voted for candidate 65c36d164b0540f099d592f38307ee6c in this term.
W20260809 15:47:07.216136 24452 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38724 (request call id 539) took 2034 ms (client timeout 1837 ms). Trace:
W20260809 15:47:07.191994 27790 consensus_meta.cc:287] T 0c48a3049897452383257aa757834fb5 P 61e69695258f48baa984c9d395e8ce3f: Time spent flushing consensus metadata: real 1.191s user 0.008s sys 0.004s
I20260809 15:47:07.217543 23521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2db057fd10e149a9895dff6ef8c9f038" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
W20260809 15:47:07.190707 23228 connection.cc:232] RPC call timeout handler was delayed by -2.1072428461629897s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18510000000000001s for a response
I20260809 15:47:07.218539 23522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d880de657e544468c82157769622a26" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
W20260809 15:47:07.220217 23353 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.331s user 0.021s sys 0.035s
W20260809 15:47:07.229027 23353 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.340s user 0.021s sys 0.035s
W20260809 15:47:07.229583 23353 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 1.4084e+06us
I20260809 15:47:07.230185 23519 raft_consensus.cc:2454] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 6.
W20260809 15:47:07.227815 24452 rpcz_store.cc:269] 0809 15:47:05.157802 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:05.485559 (+327757us) service_pool.cc:227] Handling call
0809 15:47:07.192172 (+1706613us) spinlock_profiling.cc:243] Waited 394 ms on lock 0x7b5400522bd4. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e92242ae 00007f34ee161c27 00007f34e7849c04 00007f34e7849b92 00007f34e7849ac1 00007f34e7849a3c 00007f34e7848cf2 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:47:07.192339 (+ 167us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":905159040}
I20260809 15:47:07.230726 24475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a51c0baaa6e432997c9d3552af4f254" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:47:07.230211 23353 leader_election.cc:302] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c
W20260809 15:47:07.234191 24475 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38722 (request call id 502) took 2750 ms (client timeout 1597 ms). Trace:
I20260809 15:47:07.233626 23518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "297c82ed906d4b48b184985eed3de601" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
W20260809 15:47:07.234550 24475 rpcz_store.cc:269] 0809 15:47:04.484980 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:04.494232 (+ 9252us) service_pool.cc:227] Handling call
0809 15:47:07.230573 (+2736341us) spinlock_profiling.cc:243] Waited 5.78 ms on lock 0x7b5800005498. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e613cf11 00007f34e62c0851 00007f34e62c0702 00007f34ef1b0b09 00007f34ee0ec730 00007f34ee1b279b 00007f34ee1b2821 00007f34ee1621b0 00007f34ee161a94 00007f34e7849c04 00007f34e7849b92
0809 15:47:07.231474 (+ 901us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":13283584}
I20260809 15:47:07.226527 27661 raft_consensus.cc:3037] T 998c6cee72cf40228c6dba064c0db2da P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Advancing to term 8
I20260809 15:47:07.238849 23353 leader_election.cc:398] T 30c1740ef6874df0a332fbd6fd4b03ce P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 6 pre-election: Vote denied by peer 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553) with higher term. Message: Invalid argument: T 30c1740ef6874df0a332fbd6fd4b03ce P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d for earlier term 6. Current term is 7.
I20260809 15:47:07.251545 23353 leader_election.cc:401] T 30c1740ef6874df0a332fbd6fd4b03ce P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 6 pre-election: Cancelling election due to peer responding with higher term
I20260809 15:47:07.230080 27694 raft_consensus.cc:482] T 0c48a3049897452383257aa757834fb5 P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:07.229154 27518 raft_consensus.cc:482] T c0741ab44d99439abc3aa3bdd27ee160 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:07.236971 27945 raft_consensus.cc:2726] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260809 15:47:07.254817 27988 raft_consensus.cc:2726] T 30c1740ef6874df0a332fbd6fd4b03ce P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: Vote denied by peer 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553) with higher term. Message: Invalid argument: T 30c1740ef6874df0a332fbd6fd4b03ce P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d for earlier term 6. Current term is 7.
W20260809 15:47:07.270229 27984 consensus_meta.cc:287] T d0721a4403a440f383318a3cea9d1442 P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent flushing consensus metadata: real 0.600s user 0.006s sys 0.004s
I20260809 15:47:07.290799 27984 raft_consensus.cc:504] T d0721a4403a440f383318a3cea9d1442 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:07.294075 27984 leader_election.cc:288] T d0721a4403a440f383318a3cea9d1442 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 7 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
W20260809 15:47:07.297039 23402 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.305s user 0.000s sys 0.013s
W20260809 15:47:07.297344 23402 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.306s user 0.000s sys 0.013s
I20260809 15:47:07.223752 27954 raft_consensus.cc:504] T 7660dfa2ef694927b8eda4100a2bcd9b P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:07.268849 27518 raft_consensus.cc:504] T c0741ab44d99439abc3aa3bdd27ee160 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:07.267210 27694 raft_consensus.cc:504] T 0c48a3049897452383257aa757834fb5 P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:07.331455 23517 raft_consensus.cc:2454] T 7660dfa2ef694927b8eda4100a2bcd9b P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 5.
I20260809 15:47:07.334609 23276 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53562 (request call id 1852) took 2877 ms. Trace:
I20260809 15:47:07.337303 23276 rpcz_store.cc:276] 0809 15:47:04.456795 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:04.456906 (+ 111us) service_pool.cc:227] Handling call
0809 15:47:04.457167 (+ 261us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:47:04.546067 (+ 88900us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:47:04.546094 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 15:47:04.546128 (+ 34us) raft_consensus.cc:1631] Marking committed up to 3
0809 15:47:04.546155 (+ 27us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:47:07.334386 (+2788231us) spinlock_profiling.cc:243] Waited 88.7 ms on lock 0x7b5400367254. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e9220f51 00007f34e921f99e 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772 00007f34e764a94c 00007f34e7649d86
0809 15:47:07.334428 (+ 42us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:47:07.334549 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":204058496}
I20260809 15:47:07.339726 23276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5712f6b6fffc4cc5a8ddfbe63e4097df" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:07.348529 23276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3be1e356b254d22b995be32f1c12f6f" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:07.349160 23276 raft_consensus.cc:2454] T d3be1e356b254d22b995be32f1c12f6f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 8.
W20260809 15:47:07.351994 23404 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55332 (request call id 1416) took 4422 ms (client timeout 1851 ms). Trace:
W20260809 15:47:07.352366 23404 rpcz_store.cc:269] 0809 15:47:02.929258 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:02.929392 (+ 134us) service_pool.cc:227] Handling call
0809 15:47:07.351756 (+4422364us) spinlock_profiling.cc:243] Waited 2.46 s on lock 0x7b5400200654. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e92242ae 00007f34ee161c27 00007f34e7849c04 00007f34e7849b92 00007f34e7849ac1 00007f34e7849a3c 00007f34e7848cf2 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:47:07.351888 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5666236800}
I20260809 15:47:07.353744 23276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d880de657e544468c82157769622a26" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:07.354388 23276 raft_consensus.cc:2454] T 3d880de657e544468c82157769622a26 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 3.
W20260809 15:47:07.387346 23468 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.495s user 0.023s sys 0.032s
W20260809 15:47:07.387734 23468 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.502s user 0.023s sys 0.033s
W20260809 15:47:07.394454 23282 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.319s user 0.015s sys 0.017s
I20260809 15:47:07.402515 23283 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53554 (request call id 1702) took 2942 ms. Trace:
I20260809 15:47:07.402873 23283 rpcz_store.cc:276] 0809 15:47:04.460553 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:04.461974 (+ 1421us) service_pool.cc:227] Handling call
0809 15:47:04.462195 (+ 221us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:47:04.496285 (+ 34090us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:47:07.393613 (+2897328us) spinlock_profiling.cc:243] Waited 25.9 ms on lock 0x7b54001536d4. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e9220f51 00007f34e921f99e 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772 00007f34e764a94c 00007f34e7649d86
0809 15:47:07.402452 (+ 8839us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":59646720}
I20260809 15:47:07.403952 23283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0741ab44d99439abc3aa3bdd27ee160" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:07.404511 23283 raft_consensus.cc:2454] T c0741ab44d99439abc3aa3bdd27ee160 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 6.
I20260809 15:47:07.363462 27507 leader_election.cc:288] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
W20260809 15:47:07.394819 23282 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.320s user 0.015s sys 0.017s
I20260809 15:47:07.426015 23283 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53562 (request call id 1859) took 2399 ms. Trace:
I20260809 15:47:07.426327 23283 rpcz_store.cc:276] 0809 15:47:05.026246 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:07.425328 (+2399082us) service_pool.cc:227] Handling call
0809 15:47:07.425597 (+ 269us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:47:07.425752 (+ 155us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:47:07.425776 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 15:47:07.425812 (+ 36us) raft_consensus.cc:1631] Marking committed up to 3
0809 15:47:07.425840 (+ 28us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:47:07.425886 (+ 46us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:47:07.425970 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:47:07.427817 23283 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53562 (request call id 1861) took 1930 ms. Trace:
I20260809 15:47:07.428736 23283 rpcz_store.cc:276] 0809 15:47:05.497096 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:07.427097 (+1930001us) service_pool.cc:227] Handling call
0809 15:47:07.427372 (+ 275us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:47:07.427534 (+ 162us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:47:07.427560 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 15:47:07.427598 (+ 38us) raft_consensus.cc:1631] Marking committed up to 3
0809 15:47:07.427627 (+ 29us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:47:07.427677 (+ 50us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:47:07.427770 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:47:07.431550 23392 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55332 (request call id 1428) took 2409 ms. Trace:
I20260809 15:47:07.431917 23392 rpcz_store.cc:276] 0809 15:47:05.022385 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:05.753056 (+730671us) service_pool.cc:227] Handling call
0809 15:47:05.753344 (+ 288us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:47:05.999397 (+246053us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:47:05.999425 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 15:47:05.999461 (+ 36us) raft_consensus.cc:1631] Marking committed up to 5
0809 15:47:05.999489 (+ 28us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:47:07.431324 (+1431835us) spinlock_profiling.cc:243] Waited 246 ms on lock 0x7b540001ccd4. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e9220f51 00007f34e921f99e 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772 00007f34e764a94c 00007f34e7649d86
0809 15:47:07.431368 (+ 44us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:47:07.431491 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":565433216}
W20260809 15:47:07.433918 24472 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38722 (request call id 506) took 2339 ms (client timeout 1947 ms). Trace:
W20260809 15:47:07.437270 24472 rpcz_store.cc:269] 0809 15:47:05.094762 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:05.517011 (+422249us) service_pool.cc:227] Handling call
0809 15:47:07.433750 (+1916739us) spinlock_profiling.cc:243] Waited 285 ms on lock 0x7b540040f454. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e92242ae 00007f34ee161c27 00007f34e7849c04 00007f34e7849b92 00007f34e7849ac1 00007f34e7849a3c 00007f34e7848cf2 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:47:07.433845 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":655354112}
I20260809 15:47:07.462678 23283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee02fff2c9c48a19f43eb358a89e7b0" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:47:07.463788 23276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e14d118b2eaa42aa8a7cd4025aba7128" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
W20260809 15:47:07.469573 23470 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.526s user 0.009s sys 0.029s
W20260809 15:47:07.470041 23470 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.527s user 0.009s sys 0.029s
I20260809 15:47:07.469887 23276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16ca412a0dae4feebe85dd01bf1a9173" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
W20260809 15:47:07.471040 23472 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.641s user 0.029s sys 0.063s
W20260809 15:47:07.474436 23472 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.645s user 0.029s sys 0.063s
I20260809 15:47:07.476086 23522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "294d4bcc1e7a477cb56455581277a60f" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:07.487884 23283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0acc47ce21c3422698d1ef7277c3ed63" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:07.488498 23283 raft_consensus.cc:2454] T 0acc47ce21c3422698d1ef7277c3ed63 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 7.
W20260809 15:47:07.471134 23351 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.634s user 0.043s sys 0.042s
W20260809 15:47:07.531009 23351 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.694s user 0.043s sys 0.042s
W20260809 15:47:07.531247 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 1.69447e+06us
W20260809 15:47:07.535097 23472 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 60266.2us
I20260809 15:47:07.548285 23276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "294d4bcc1e7a477cb56455581277a60f" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:07.549640 23351 leader_election.cc:302] T df20671a7b874dbb98c0ef28371c1308 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:47:07.550937 23522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73f6f44c259547f1b05a3a434616fa54" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
W20260809 15:47:07.554741 23470 connection.cc:232] RPC call timeout handler was delayed by -3.2049791070166975s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2002s for a response
I20260809 15:47:07.554991 23522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3e94b7e50754a81b3d72a07476cb7b9" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
W20260809 15:47:07.557515 23470 connection.cc:232] RPC call timeout handler was delayed by -2.9818432140164077s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15410000000000001s for a response
W20260809 15:47:07.558224 23470 connection.cc:232] RPC call timeout handler was delayed by -2.9145897170528769s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1522s for a response
I20260809 15:47:07.560034 23351 leader_election.cc:302] T 8e4ffcf83c234ae7910ddef3ec0b4bda P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 61e69695258f48baa984c9d395e8ce3f, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:47:07.561108 23351 leader_election.cc:302] T 73f6f44c259547f1b05a3a434616fa54 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:47:07.561780 23283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7baa2492df0340c4a5ac241959ce1ac3" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:07.562376 23351 leader_election.cc:302] T 563daf42d7784db195ea99c3cb21b370 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters: 61e69695258f48baa984c9d395e8ce3f
I20260809 15:47:07.563256 23276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee02fff2c9c48a19f43eb358a89e7b0" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:07.568219 23518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73f6f44c259547f1b05a3a434616fa54" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:07.471179 27984 raft_consensus.cc:482] T d3be1e356b254d22b995be32f1c12f6f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:07.572291 23351 leader_election.cc:302] T 998c6cee72cf40228c6dba064c0db2da P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:47:07.471091 27961 leader_election.cc:288] T cef3e7277d7d4760824a3436ed4a847d P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 7 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:07.548185 27954 leader_election.cc:288] T 7660dfa2ef694927b8eda4100a2bcd9b P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:07.577203 23283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99bdb93503db4de4a1cfb0161ea0c07c" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:07.571182 23522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3924b648461e463b9d7925e5671d68ae" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:07.597685 23518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998c6cee72cf40228c6dba064c0db2da" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:07.602571 23522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ced718c1a5c4da79cb8c1ebd1435b87" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:07.603609 23521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bcac4b24e7a4448abdc934bba5a624e" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:07.567528 27893 raft_consensus.cc:2726] T df20671a7b874dbb98c0ef28371c1308 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260809 15:47:07.608814 23283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae6d2c08a7a94f58ac8dd698e3fd456c" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:07.609411 23283 raft_consensus.cc:2454] T ae6d2c08a7a94f58ac8dd698e3fd456c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 7.
I20260809 15:47:07.574563 27945 raft_consensus.cc:2726] T 8e4ffcf83c234ae7910ddef3ec0b4bda P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260809 15:47:07.561493 27363 consensus_queue.cc:1035] T 297c82ed906d4b48b184985eed3de601 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.044s
I20260809 15:47:07.617395 23283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4564dfc6d4094c1b9240b367ccecf3b1" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:07.574905 27988 consensus_queue.cc:1035] T 538a228ddfb54bc6a36528d5ff07597b P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.038s
I20260809 15:47:07.608155 27518 leader_election.cc:288] T c0741ab44d99439abc3aa3bdd27ee160 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:07.613399 27961 raft_consensus.cc:482] T ba56725e42f74a1baa4c04d9660727f9 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:07.569120 28029 consensus_queue.cc:1035] T 07fcabb569414bddbd39df6cedd24e64 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.066s
I20260809 15:47:07.612776 27798 raft_consensus.cc:482] T a88421dba80d4d3388aa5eb62e23853b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:47:07.628100 23521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05ad7d8fdca64e3da527280ca5be9280" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:07.628703 23521 raft_consensus.cc:2454] T 05ad7d8fdca64e3da527280ca5be9280 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 6.
I20260809 15:47:07.583354 27984 raft_consensus.cc:504] T d3be1e356b254d22b995be32f1c12f6f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:07.614398 24448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5712f6b6fffc4cc5a8ddfbe63e4097df" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:47:07.616698 27893 raft_consensus.cc:2781] T 73f6f44c259547f1b05a3a434616fa54 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260809 15:47:07.631942 23283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df20671a7b874dbb98c0ef28371c1308" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:07.633101 23522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472816994d814cb198b69e5248697440" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:07.624590 27886 raft_consensus.cc:2781] T 998c6cee72cf40228c6dba064c0db2da P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260809 15:47:07.624460 28032 raft_consensus.cc:2781] T 563daf42d7784db195ea99c3cb21b370 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260809 15:47:07.637966 23283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b30f28311964a0b9149a50495e83d8f" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
W20260809 15:47:07.646369 23347 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.707s user 0.030s sys 0.044s
W20260809 15:47:07.647326 23347 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.708s user 0.030s sys 0.044s
I20260809 15:47:07.637235 23276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e14d118b2eaa42aa8a7cd4025aba7128" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:07.637209 23522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07fcabb569414bddbd39df6cedd24e64" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:07.647346 23283 raft_consensus.cc:2454] T 4b30f28311964a0b9149a50495e83d8f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 3.
I20260809 15:47:07.648792 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1a134ba33ea4076bbaac0ae460c17ba" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
W20260809 15:47:07.638044 27997 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.256s user 0.017s sys 0.027s
I20260809 15:47:07.650357 23518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "538a228ddfb54bc6a36528d5ff07597b" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:07.650920 23518 raft_consensus.cc:2454] T 538a228ddfb54bc6a36528d5ff07597b P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 6.
I20260809 15:47:07.637696 24406 leader_election.cc:302] T d3be1e356b254d22b995be32f1c12f6f P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:47:07.639567 23521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4564dfc6d4094c1b9240b367ccecf3b1" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
W20260809 15:47:07.638067 23345 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.666s user 0.015s sys 0.033s
W20260809 15:47:07.653324 23345 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.681s user 0.015s sys 0.033s
I20260809 15:47:07.653378 24406 leader_election.cc:302] T 3d880de657e544468c82157769622a26 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:47:07.653136 23276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00a1881267a14ba8b78c81040f17a014" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:07.657886 23518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16ca412a0dae4feebe85dd01bf1a9173" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:07.660014 24406 leader_election.cc:302] T c0741ab44d99439abc3aa3bdd27ee160 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c
I20260809 15:47:07.661162 23392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3be1e356b254d22b995be32f1c12f6f" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:07.651880 27792 tablet_copy_client.cc:531] T 4b30f28311964a0b9149a50495e83d8f P 61e69695258f48baa984c9d395e8ce3f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260809 15:47:07.668538 27953 raft_consensus.cc:482] T d0721a4403a440f383318a3cea9d1442 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:47:07.669026 27953 raft_consensus.cc:504] T d0721a4403a440f383318a3cea9d1442 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:07.669348 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07fcabb569414bddbd39df6cedd24e64" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:07.670782 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3622f6e3e3641cea1960e3a31498bb7" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:07.634447 27893 raft_consensus.cc:482] T 73f6f44c259547f1b05a3a434616fa54 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Starting leader election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:47:07.671504 27893 raft_consensus.cc:3037] T 73f6f44c259547f1b05a3a434616fa54 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Advancing to term 8
I20260809 15:47:07.671626 23404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7660dfa2ef694927b8eda4100a2bcd9b" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:07.672120 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472816994d814cb198b69e5248697440" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:47:07.662743 24471 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38722 (request call id 507) took 2172 ms. Trace:
I20260809 15:47:07.673282 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "763e58d6ba644eea830a6c1664d39c9b" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:07.661746 23399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3aa6a9a52804df0ba9b0ca3c82f38c3" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:07.664516 23522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "297c82ed906d4b48b184985eed3de601" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:07.674337 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee02fff2c9c48a19f43eb358a89e7b0" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:07.674897 23527 raft_consensus.cc:2454] T cee02fff2c9c48a19f43eb358a89e7b0 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 7.
I20260809 15:47:07.665457 23283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cef3e7277d7d4760824a3436ed4a847d" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:07.676349 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0da4d5283c8a46878bd39b85e5c71f66" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:07.666435 23276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "538a228ddfb54bc6a36528d5ff07597b" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:07.677249 23276 raft_consensus.cc:2454] T 538a228ddfb54bc6a36528d5ff07597b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 6.
I20260809 15:47:07.678072 23276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30c1740ef6874df0a332fbd6fd4b03ce" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
W20260809 15:47:07.649585 27997 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.300s user 0.017s sys 0.028s
I20260809 15:47:07.675726 23283 raft_consensus.cc:2454] T cef3e7277d7d4760824a3436ed4a847d P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 7.
I20260809 15:47:07.679123 24452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3296ef50be254fc29cfc77d22a8f7a23" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:47:07.682677 23522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2d4f749fcfd4fb3bfb8115af3471598" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:07.673664 24471 rpcz_store.cc:276] 0809 15:47:05.490017 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:05.584171 (+ 94154us) service_pool.cc:227] Handling call
0809 15:47:05.647463 (+ 63292us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:47:05.883767 (+236304us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:47:05.883797 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 15:47:05.883838 (+ 41us) raft_consensus.cc:1631] Marking committed up to 9
0809 15:47:05.883868 (+ 30us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:47:07.662520 (+1778652us) spinlock_profiling.cc:243] Waited 236 ms on lock 0x7b540049f554. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e9220f51 00007f34e921f99e 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772 00007f34e764a94c 00007f34e7649d86
0809 15:47:07.662571 (+ 51us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:47:07.662687 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":543006464}
I20260809 15:47:07.671063 27953 leader_election.cc:288] T d0721a4403a440f383318a3cea9d1442 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:07.685050 24475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e71a05ba1d4d42bd95c3f85f9313399e" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:47:07.685658 24475 raft_consensus.cc:2454] T e71a05ba1d4d42bd95c3f85f9313399e P 61e69695258f48baa984c9d395e8ce3f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 6.
I20260809 15:47:07.639887 27334 raft_consensus.cc:482] T 0c48a3049897452383257aa757834fb5 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:07.647111 27886 raft_consensus.cc:482] T 998c6cee72cf40228c6dba064c0db2da P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Starting leader election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:47:07.689774 23395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bcac4b24e7a4448abdc934bba5a624e" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:07.696880 23395 raft_consensus.cc:2454] T 4bcac4b24e7a4448abdc934bba5a624e P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 3.
I20260809 15:47:07.688604 23521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2d4f749fcfd4fb3bfb8115af3471598" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:07.651937 28032 raft_consensus.cc:482] T 563daf42d7784db195ea99c3cb21b370 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Starting leader election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
I20260809 15:47:07.697225 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99bdb93503db4de4a1cfb0161ea0c07c" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
W20260809 15:47:07.697947 23223 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 1.666s user 0.027s sys 0.060s
I20260809 15:47:07.698061 23283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "953eea268e17491ca76f3a555860a512" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:07.699340 23470 leader_election.cc:302] T c5c2e425a74b4e64ae70b5def750434c P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d
W20260809 15:47:07.703204 23223 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 1.671s user 0.027s sys 0.060s
W20260809 15:47:07.703508 23223 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.68:34689, user_credentials={real_user=slave}} blocked reactor thread for 1.69426e+06us
I20260809 15:47:07.704104 23223 leader_election.cc:302] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:47:07.663800 27688 raft_consensus.cc:2781] T d3be1e356b254d22b995be32f1c12f6f P 61e69695258f48baa984c9d395e8ce3f [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260809 15:47:07.634335 27798 raft_consensus.cc:504] T a88421dba80d4d3388aa5eb62e23853b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:07.705339 23223 leader_election.cc:302] T 294d4bcc1e7a477cb56455581277a60f P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:47:07.638746 24448 raft_consensus.cc:2454] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 4.
I20260809 15:47:07.634563 27961 raft_consensus.cc:504] T ba56725e42f74a1baa4c04d9660727f9 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:07.706439 24452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "907bacfa79d04a55a68c8dbe60056462" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:47:07.675804 27787 raft_consensus.cc:2781] T 3d880de657e544468c82157769622a26 P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260809 15:47:07.695101 27953 raft_consensus.cc:482] T a88421dba80d4d3388aa5eb62e23853b P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:47:07.708382 23399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "538a228ddfb54bc6a36528d5ff07597b" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:07.724609 23399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b30f28311964a0b9149a50495e83d8f" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:07.710148 24475 raft_consensus.cc:1261] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 61e69695258f48baa984c9d395e8ce3f [term 6 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
W20260809 15:47:07.658519 24455 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.775s user 0.023s sys 0.056s
I20260809 15:47:07.708112 23223 leader_election.cc:302] T 16ca412a0dae4feebe85dd01bf1a9173 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:47:07.711599 23392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "953eea268e17491ca76f3a555860a512" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:07.725945 23521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "213543cdbe814c89a17a963469922c1b" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:07.726666 23392 raft_consensus.cc:2396] T 953eea268e17491ca76f3a555860a512 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c for term 8 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260809 15:47:07.725224 23399 raft_consensus.cc:2454] T 4b30f28311964a0b9149a50495e83d8f P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 4.
I20260809 15:47:07.721078 24452 raft_consensus.cc:2396] T 907bacfa79d04a55a68c8dbe60056462 P 61e69695258f48baa984c9d395e8ce3f [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d for term 8 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20260809 15:47:07.728453 23392 raft_consensus.cc:1261] T 0c29937a80724ddb9d07a27534a146f3 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 5 index: 2. Preceding OpId from leader: term: 7 index: 3. (index mismatch)
I20260809 15:47:07.711858 23397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99bdb93503db4de4a1cfb0161ea0c07c" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:07.710939 23518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5c2e425a74b4e64ae70b5def750434c" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
W20260809 15:47:07.727582 24455 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.844s user 0.023s sys 0.056s
I20260809 15:47:07.729470 23518 raft_consensus.cc:2454] T c5c2e425a74b4e64ae70b5def750434c P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 6.
I20260809 15:47:07.730185 23404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0acc47ce21c3422698d1ef7277c3ed63" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:07.731899 23283 raft_consensus.cc:1261] T 0c29937a80724ddb9d07a27534a146f3 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 5 index: 2. Preceding OpId from leader: term: 7 index: 3. (index mismatch)
I20260809 15:47:07.705816 28032 raft_consensus.cc:3037] T 563daf42d7784db195ea99c3cb21b370 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Advancing to term 6
I20260809 15:47:07.736217 23276 raft_consensus.cc:1261] T ba56725e42f74a1baa4c04d9660727f9 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
W20260809 15:47:07.736606 23270 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.421s user 0.014s sys 0.056s
W20260809 15:47:07.736940 23270 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.422s user 0.015s sys 0.056s
I20260809 15:47:07.708750 27518 raft_consensus.cc:2781] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260809 15:47:07.730394 23223 leader_election.cc:302] T 3d880de657e544468c82157769622a26 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:47:07.681423 27790 consensus_queue.cc:1035] T 0a51c0baaa6e432997c9d3552af4f254 P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.039s
I20260809 15:47:07.662242 27661 raft_consensus.cc:504] T 998c6cee72cf40228c6dba064c0db2da P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:07.704591 27544 raft_consensus.cc:2781] T c0741ab44d99439abc3aa3bdd27ee160 P 61e69695258f48baa984c9d395e8ce3f [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260809 15:47:07.707355 27688 raft_consensus.cc:482] T d3be1e356b254d22b995be32f1c12f6f P 61e69695258f48baa984c9d395e8ce3f [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:47:07.702396 27988 raft_consensus.cc:482] T 0a51c0baaa6e432997c9d3552af4f254 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:07.703385 27886 raft_consensus.cc:3037] T 998c6cee72cf40228c6dba064c0db2da P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Advancing to term 8
I20260809 15:47:07.724143 27787 raft_consensus.cc:482] T 3d880de657e544468c82157769622a26 P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Starting leader election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:47:07.756421 23522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0721a4403a440f383318a3cea9d1442" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
W20260809 15:47:07.744973 23274 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55434 (request call id 273) took 3239 ms (client timeout 1802 ms). Trace:
W20260809 15:47:07.758239 23274 rpcz_store.cc:269] 0809 15:47:04.505544 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:05.491735 (+986191us) service_pool.cc:227] Handling call
0809 15:47:07.744803 (+2253068us) spinlock_profiling.cc:243] Waited 245 ms on lock 0x7b54002f58d4. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e92242ae 00007f34ee161c27 00007f34e7849c04 00007f34e7849b92 00007f34e7849ac1 00007f34e7849a3c 00007f34e7848cf2 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:47:07.744914 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":564519168}
I20260809 15:47:07.702711 27334 raft_consensus.cc:504] T 0c48a3049897452383257aa757834fb5 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:07.759182 23522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c29937a80724ddb9d07a27534a146f3" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 8 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:07.759377 23521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2db057fd10e149a9895dff6ef8c9f038" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
W20260809 15:47:07.729144 24448 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38722 (request call id 501) took 3270 ms (client timeout 2187 ms). Trace:
I20260809 15:47:07.780711 23397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cef3e7277d7d4760824a3436ed4a847d" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
W20260809 15:47:07.783147 23277 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55434 (request call id 274) took 3260 ms (client timeout 1731 ms). Trace:
I20260809 15:47:07.744163 23520 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56770 (request call id 1404) took 3219 ms. Trace:
I20260809 15:47:07.764264 23518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b30f28311964a0b9149a50495e83d8f" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:07.784018 23392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7baa2492df0340c4a5ac241959ce1ac3" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:07.782478 23528 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56766 (request call id 2029) took 4205 ms. Trace:
I20260809 15:47:07.783601 23397 raft_consensus.cc:2379] T cef3e7277d7d4760824a3436ed4a847d P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c in current term 8: Already voted for candidate c0f38c1908414566aa9115bb2cbdd85d in this term.
I20260809 15:47:07.784883 23528 rpcz_store.cc:276] 0809 15:47:03.554536 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:03.555020 (+ 484us) service_pool.cc:227] Handling call
0809 15:47:03.555287 (+ 267us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:47:05.647406 (+2092119us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:47:07.745215 (+2097809us) spinlock_profiling.cc:243] Waited 1.93 s on lock 0x7b540016ec54. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e9220f51 00007f34e921f99e 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772 00007f34e764a94c 00007f34e7649d86
0809 15:47:07.759898 (+ 14683us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4432680320}
W20260809 15:47:07.783516 23277 rpcz_store.cc:269] 0809 15:47:04.522813 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:04.953852 (+431039us) service_pool.cc:227] Handling call
0809 15:47:07.782992 (+2829140us) spinlock_profiling.cc:243] Waited 496 ms on lock 0x7b54002409d4. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e92242ae 00007f34ee161c27 00007f34e7849c04 00007f34e7849b92 00007f34e7849ac1 00007f34e7849a3c 00007f34e7848cf2 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:47:07.783087 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1140102528}
I20260809 15:47:07.787025 23404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30c1740ef6874df0a332fbd6fd4b03ce" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:07.744632 23394 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55320 (request call id 1545) took 2237 ms. Trace:
I20260809 15:47:07.789204 23522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cef3e7277d7d4760824a3436ed4a847d" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:47:07.788921 23394 rpcz_store.cc:276] 0809 15:47:05.506768 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:06.058541 (+551773us) service_pool.cc:227] Handling call
0809 15:47:06.073776 (+ 15235us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:47:06.315593 (+241817us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:47:06.315627 (+ 34us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 15:47:06.315669 (+ 42us) raft_consensus.cc:1631] Marking committed up to 3
0809 15:47:06.315703 (+ 34us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:47:07.744428 (+1428725us) spinlock_profiling.cc:243] Waited 242 ms on lock 0x7b5400193554. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e9220f51 00007f34e921f99e 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772 00007f34e764a94c 00007f34e7649d86
0809 15:47:07.744469 (+ 41us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:47:07.744580 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":555658496}
I20260809 15:47:07.794775 23280 raft_consensus.cc:1226] T 998c6cee72cf40228c6dba064c0db2da P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 7. Current term is 8. Ops: []
I20260809 15:47:07.784104 23520 rpcz_store.cc:276] 0809 15:47:04.524352 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:04.526176 (+ 1824us) service_pool.cc:227] Handling call
0809 15:47:04.526465 (+ 289us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:47:05.153792 (+627327us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:47:05.153820 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 15:47:05.153855 (+ 35us) raft_consensus.cc:1631] Marking committed up to 9
0809 15:47:05.153886 (+ 31us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:47:07.743958 (+2590072us) spinlock_profiling.cc:243] Waited 627 ms on lock 0x7b5400075dd4. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e9220f51 00007f34e921f99e 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772 00007f34e764a94c 00007f34e7649d86
0809 15:47:07.743998 (+ 40us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:47:07.744107 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1442296192}
I20260809 15:47:07.787657 23404 raft_consensus.cc:2454] T 30c1740ef6874df0a332fbd6fd4b03ce P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 7.
I20260809 15:47:07.741074 27518 raft_consensus.cc:482] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
I20260809 15:47:07.797067 23283 raft_consensus.cc:1261] T 297c82ed906d4b48b184985eed3de601 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260809 15:47:07.797811 23518 raft_consensus.cc:2379] T 4b30f28311964a0b9149a50495e83d8f P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 in current term 4: Already voted for candidate 65c36d164b0540f099d592f38307ee6c in this term.
I20260809 15:47:07.733323 27953 raft_consensus.cc:504] T a88421dba80d4d3388aa5eb62e23853b P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:07.746243 27940 raft_consensus.cc:2726] T c5c2e425a74b4e64ae70b5def750434c P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260809 15:47:07.759989 27544 raft_consensus.cc:482] T c0741ab44d99439abc3aa3bdd27ee160 P 61e69695258f48baa984c9d395e8ce3f [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:47:07.758944 27787 raft_consensus.cc:3037] T 3d880de657e544468c82157769622a26 P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Advancing to term 4
W20260809 15:47:07.782955 24448 rpcz_store.cc:269] 0809 15:47:04.455452 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:04.494686 (+ 39234us) service_pool.cc:227] Handling call
0809 15:47:07.606917 (+3112231us) spinlock_profiling.cc:243] Waited 17.9 ms on lock 0x7b5800005498. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e613cf11 00007f34e62c0851 00007f34e62c0702 00007f34ef1b0b09 00007f34ee0ec730 00007f34ee1b279b 00007f34ee1b2821 00007f34ee1621b0 00007f34ee161a94 00007f34e7849c04 00007f34e7849b92
0809 15:47:07.725380 (+118463us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":41115904}
I20260809 15:47:07.815929 23394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3924b648461e463b9d7925e5671d68ae" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:07.757364 27688 raft_consensus.cc:3037] T d3be1e356b254d22b995be32f1c12f6f P 61e69695258f48baa984c9d395e8ce3f [term 8 FOLLOWER]: Advancing to term 9
I20260809 15:47:07.823035 23528 raft_consensus.cc:1261] T 538a228ddfb54bc6a36528d5ff07597b P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260809 15:47:07.838634 27544 raft_consensus.cc:3037] T c0741ab44d99439abc3aa3bdd27ee160 P 61e69695258f48baa984c9d395e8ce3f [term 6 FOLLOWER]: Advancing to term 7
I20260809 15:47:07.842914 27661 leader_election.cc:288] T 998c6cee72cf40228c6dba064c0db2da P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 8 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:07.810111 27518 raft_consensus.cc:3037] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Advancing to term 7
I20260809 15:47:07.845821 23404 raft_consensus.cc:1261] T 2db057fd10e149a9895dff6ef8c9f038 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260809 15:47:07.811589 27954 raft_consensus.cc:482] T 28e17dcb5efe45b8b6a8c42da03b4819 P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:47:07.764873 27988 raft_consensus.cc:504] T 0a51c0baaa6e432997c9d3552af4f254 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:07.846140 23393 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55332 (request call id 1422) took 3446 ms. Trace:
I20260809 15:47:07.852239 23393 rpcz_store.cc:276] 0809 15:47:04.399804 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:04.400523 (+ 719us) service_pool.cc:227] Handling call
0809 15:47:04.400798 (+ 275us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:47:05.946519 (+1545721us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:47:07.845924 (+1899405us) spinlock_profiling.cc:243] Waited 1.46 s on lock 0x7b540009fbd4. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e9220f51 00007f34e921f99e 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772 00007f34e764a94c 00007f34e7649d86
0809 15:47:07.846089 (+ 165us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3357276416}
W20260809 15:47:07.853250 24403 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.668s user 0.038s sys 0.047s
I20260809 15:47:07.824628 27984 leader_election.cc:288] T d3be1e356b254d22b995be32f1c12f6f P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
W20260809 15:47:07.853863 23285 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.649s user 0.058s sys 0.083s
I20260809 15:47:07.853490 27982 raft_consensus.cc:482] T a3622f6e3e3641cea1960e3a31498bb7 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:47:07.861964 23518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0da4d5283c8a46878bd39b85e5c71f66" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
W20260809 15:47:07.853646 24403 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.677s user 0.038s sys 0.047s
I20260809 15:47:07.852836 27661 raft_consensus.cc:2781] T 294d4bcc1e7a477cb56455581277a60f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260809 15:47:07.888321 27982 raft_consensus.cc:504] T a3622f6e3e3641cea1960e3a31498bb7 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:47:07.893193 23226 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.638s user 0.010s sys 0.075s
W20260809 15:47:07.893503 23226 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.638s user 0.010s sys 0.075s
I20260809 15:47:07.892768 23528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cef3e7277d7d4760824a3436ed4a847d" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:47:07.894814 23404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df20671a7b874dbb98c0ef28371c1308" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:07.920853 23404 raft_consensus.cc:2454] T df20671a7b874dbb98c0ef28371c1308 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 6.
W20260809 15:47:07.886765 23285 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.682s user 0.058s sys 0.086s
I20260809 15:47:07.855155 27954 raft_consensus.cc:504] T 28e17dcb5efe45b8b6a8c42da03b4819 P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:07.891990 27661 raft_consensus.cc:482] T 294d4bcc1e7a477cb56455581277a60f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:47:07.894321 27984 raft_consensus.cc:2781] T 16ca412a0dae4feebe85dd01bf1a9173 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260809 15:47:07.958395 23404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae6d2c08a7a94f58ac8dd698e3fd456c" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:07.963954 23397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00a1881267a14ba8b78c81040f17a014" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:07.967307 23394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4564dfc6d4094c1b9240b367ccecf3b1" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:07.968010 23394 raft_consensus.cc:2379] T 4564dfc6d4094c1b9240b367ccecf3b1 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c in current term 7: Already voted for candidate c0f38c1908414566aa9115bb2cbdd85d in this term.
W20260809 15:47:07.929432 27892 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.892s user 0.037s sys 0.055s
I20260809 15:47:07.891134 27688 raft_consensus.cc:504] T d3be1e356b254d22b995be32f1c12f6f P 61e69695258f48baa984c9d395e8ce3f [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:07.929891 27961 leader_election.cc:288] T ba56725e42f74a1baa4c04d9660727f9 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:07.958784 27493 leader_election.cc:288] T 2424fd9bd7f14e508fd106cbfda786a4 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:07.958559 27661 raft_consensus.cc:3037] T 294d4bcc1e7a477cb56455581277a60f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Advancing to term 6
I20260809 15:47:07.959797 27984 raft_consensus.cc:482] T 16ca412a0dae4feebe85dd01bf1a9173 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:47:07.992533 23392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7660dfa2ef694927b8eda4100a2bcd9b" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:07.995047 23404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3aa6a9a52804df0ba9b0ca3c82f38c3" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:07.999804 23392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2db057fd10e149a9895dff6ef8c9f038" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:08.002209 27334 leader_election.cc:288] T 0c48a3049897452383257aa757834fb5 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:08.002939 27982 leader_election.cc:288] T a3622f6e3e3641cea1960e3a31498bb7 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:08.006903 27953 leader_election.cc:288] T a88421dba80d4d3388aa5eb62e23853b P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:08.008885 27792 tablet_bootstrap.cc:492] T 4b30f28311964a0b9149a50495e83d8f P 61e69695258f48baa984c9d395e8ce3f: Bootstrap starting.
W20260809 15:47:07.971405 27892 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.934s user 0.037s sys 0.055s
W20260809 15:47:08.011034 28035 log.cc:924] Time spent T d1a134ba33ea4076bbaac0ae460c17ba P c0f38c1908414566aa9115bb2cbdd85d: Append to log took a long time: real 0.117s user 0.000s sys 0.000s
I20260809 15:47:08.051501 23404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae6d2c08a7a94f58ac8dd698e3fd456c" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:08.000444 23392 raft_consensus.cc:2454] T 2db057fd10e149a9895dff6ef8c9f038 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 6.
I20260809 15:47:07.979792 27751 leader_election.cc:288] T ba56725e42f74a1baa4c04d9660727f9 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:08.050480 23394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "563daf42d7784db195ea99c3cb21b370" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:08.050658 27988 leader_election.cc:288] T 0a51c0baaa6e432997c9d3552af4f254 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:07.996343 27961 raft_consensus.cc:2781] T 3d880de657e544468c82157769622a26 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260809 15:47:07.997459 27984 raft_consensus.cc:3037] T 16ca412a0dae4feebe85dd01bf1a9173 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Advancing to term 6
I20260809 15:47:08.064457 27480 leader_election.cc:288] T 907bacfa79d04a55a68c8dbe60056462 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:47:08.048477 27694 leader_election.cc:288] T 0c48a3049897452383257aa757834fb5 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:08.047147 27928 leader_election.cc:288] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:08.053158 27918 leader_election.cc:288] T d3e94b7e50754a81b3d72a07476cb7b9 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:08.073237 23392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "297c82ed906d4b48b184985eed3de601" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:08.073947 23392 raft_consensus.cc:2454] T 297c82ed906d4b48b184985eed3de601 P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 8.
I20260809 15:47:08.071350 23522 raft_consensus.cc:2454] T cef3e7277d7d4760824a3436ed4a847d P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 7.
I20260809 15:47:08.075927 23400 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55320 (request call id 1544) took 2569 ms. Trace:
W20260809 15:47:08.054970 23282 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.584s user 0.022s sys 0.007s
W20260809 15:47:08.079545 23282 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.609s user 0.022s sys 0.007s
I20260809 15:47:08.076299 23400 rpcz_store.cc:276] 0809 15:47:05.506422 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:05.992527 (+486105us) service_pool.cc:227] Handling call
0809 15:47:05.992781 (+ 254us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:47:06.032966 (+ 40185us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:47:06.032994 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 15:47:06.033030 (+ 36us) raft_consensus.cc:1631] Marking committed up to 3
0809 15:47:06.033059 (+ 29us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:47:08.075709 (+2042650us) spinlock_profiling.cc:243] Waited 40 ms on lock 0x7b5400192b54. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e9220f51 00007f34e921f99e 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772 00007f34e764a94c 00007f34e7649d86
0809 15:47:08.075756 (+ 47us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:47:08.075872 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":92067200}
I20260809 15:47:08.011838 27735 consensus_queue.cc:1035] T c0741ab44d99439abc3aa3bdd27ee160 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.054s
I20260809 15:47:08.123199 23394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0acc47ce21c3422698d1ef7277c3ed63" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:08.124244 23404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3296ef50be254fc29cfc77d22a8f7a23" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:08.125571 23282 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53562 (request call id 1848) took 5261 ms. Trace:
W20260809 15:47:08.129274 23472 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.561s user 0.003s sys 0.016s
I20260809 15:47:08.128890 23282 rpcz_store.cc:276] 0809 15:47:02.818592 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:02.818685 (+ 93us) service_pool.cc:227] Handling call
0809 15:47:02.818986 (+ 301us) raft_consensus.cc:1457] Updating replica for 3 ops
0809 15:47:02.819451 (+ 465us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:47:02.819479 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 3 ops
0809 15:47:02.820204 (+ 725us) log.cc:842] Serialized 78 byte log entry
0809 15:47:03.135540 (+315336us) raft_consensus.cc:1631] Marking committed up to 5
0809 15:47:03.158317 (+ 22777us) log.cc:842] Serialized 14 byte log entry
0809 15:47:05.658493 (+2500176us) spinlock_profiling.cc:243] Waited 3.84 ms on lock 0x7b180031e640. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34ef1978b1 00007f34ef1a4661 00007f34e62f8638 00007f34e62fbb11 00007f34e62fbac9 00007f34e62fba59 00007f34e62fba21 00007f34e62fad1d 00007f34ef55c204 00007f34ef55c039 00007f34e62dadc6
0809 15:47:06.074606 (+416113us) spinlock_profiling.cc:243] Waited 2.04 ms on lock 0x7b180031e640. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34ef1978b1 00007f34ef1a4661 00007f34e626f4ca 00007f34ef950f99 00007f34e6133990 00007f34e62f5864 00007f34e62f5155 00007f34ecbf5167 00007f34e922a807 00007f34e9239f0f 00007f34e9239e88
0809 15:47:06.074814 (+ 208us) log.cc:842] Serialized 14 byte log entry
0809 15:47:08.079934 (+2005120us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:47:08.080043 (+ 109us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 15:47:08.080134 (+ 91us) raft_consensus.cc:1671] finished
0809 15:47:08.080147 (+ 13us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:47:08.080312 (+ 165us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2942451,"prepare.queue_time_us":2943726,"prepare.run_cpu_time_us":93,"prepare.run_wall_time_us":4720,"raft.queue_time_us":190466,"raft.run_cpu_time_us":393,"raft.run_wall_time_us":102087,"spinlock_wait_cycles":13532800,"thread_start_us":2261193,"threads_started":3,"wal-append.queue_time_us":2387575,"wal-append.run_cpu_time_us":3212,"wal-append.run_wall_time_us":1551380}
W20260809 15:47:08.127905 24406 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 446971us
W20260809 15:47:08.131004 23472 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.564s user 0.007s sys 0.016s
I20260809 15:47:08.060966 27961 raft_consensus.cc:482] T 3d880de657e544468c82157769622a26 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
W20260809 15:47:08.135802 23351 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.560s user 0.014s sys 0.006s
W20260809 15:47:08.136266 23351 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.560s user 0.014s sys 0.006s
W20260809 15:47:08.136512 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 564258us
I20260809 15:47:08.137151 23351 leader_election.cc:302] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 61e69695258f48baa984c9d395e8ce3f, 67b3aeadfa7a432abe8af4d292e04ab9
W20260809 15:47:08.137996 23353 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.558s user 0.008s sys 0.012s
W20260809 15:47:08.138335 23353 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.559s user 0.008s sys 0.012s
W20260809 15:47:08.132032 23472 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 564980us
I20260809 15:47:08.136710 23270 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53554 (request call id 1700) took 3676 ms. Trace:
I20260809 15:47:08.129163 27762 raft_consensus.cc:482] T d1a134ba33ea4076bbaac0ae460c17ba P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:47:08.144793 23270 rpcz_store.cc:276] 0809 15:47:04.460081 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:04.460419 (+ 338us) service_pool.cc:227] Handling call
0809 15:47:04.501419 (+ 41000us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 15:47:05.592601 (+1091182us) raft_consensus.cc:1520] Early marking committed up to index 3
0809 15:47:07.737349 (+2144748us) log.cc:842] Serialized 14 byte log entry
0809 15:47:08.128412 (+391063us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 15:47:08.128846 (+ 434us) log.cc:842] Serialized 26 byte log entry
0809 15:47:08.129012 (+ 166us) raft_consensus.cc:1631] Marking committed up to 3
0809 15:47:08.129052 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:47:08.136103 (+ 7051us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 15:47:08.136520 (+ 417us) raft_consensus.cc:1671] finished
0809 15:47:08.136540 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:47:08.136657 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":517199,"prepare.queue_time_us":765640,"prepare.run_cpu_time_us":54,"prepare.run_wall_time_us":8264,"raft.queue_time_us":531992,"raft.run_cpu_time_us":102,"raft.run_wall_time_us":103,"thread_start_us":2826861,"threads_started":3,"wal-append.queue_time_us":368537}
I20260809 15:47:08.122045 27761 raft_consensus.cc:482] T 00a1881267a14ba8b78c81040f17a014 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
W20260809 15:47:08.171195 23223 connection.cc:232] RPC call timeout handler was delayed by -1.381552335107699s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2187s for a response
W20260809 15:47:08.170152 23228 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.596s user 0.007s sys 0.015s
W20260809 15:47:08.138764 23353 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 583834us
I20260809 15:47:08.171293 24475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3aa6a9a52804df0ba9b0ca3c82f38c3" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:47:08.174435 23402 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55320 (request call id 1543) took 2668 ms. Trace:
I20260809 15:47:08.174823 24475 raft_consensus.cc:2454] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 61e69695258f48baa984c9d395e8ce3f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 6.
I20260809 15:47:08.137734 27961 raft_consensus.cc:3037] T 3d880de657e544468c82157769622a26 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:47:08.134186 28051 raft_consensus.cc:482] T 0c29937a80724ddb9d07a27534a146f3 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:08.174868 23402 rpcz_store.cc:276] 0809 15:47:05.506092 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:05.990409 (+484317us) service_pool.cc:227] Handling call
0809 15:47:05.990753 (+ 344us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 15:47:05.991048 (+ 295us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:47:05.991073 (+ 25us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 15:47:05.991466 (+ 393us) log.cc:842] Serialized 26 byte log entry
0809 15:47:07.297698 (+1306232us) raft_consensus.cc:1631] Marking committed up to 3
0809 15:47:07.297733 (+ 35us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:47:07.297798 (+ 65us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 15:47:08.174245 (+876447us) raft_consensus.cc:1671] finished
0809 15:47:08.174263 (+ 18us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:47:08.174373 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":12503,"raft.run_cpu_time_us":119,"raft.run_wall_time_us":8695,"thread_start_us":1305971,"threads_started":1,"wal-append.queue_time_us":1863909}
I20260809 15:47:08.181041 23468 leader_election.cc:302] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c; no voters:
I20260809 15:47:08.187772 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c48a3049897452383257aa757834fb5" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:47:08.188649 24472 raft_consensus.cc:2396] T 0c48a3049897452383257aa757834fb5 P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c for term 6 because replica has last-logged OpId of term: 4 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
W20260809 15:47:08.175874 23470 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 471644us
W20260809 15:47:08.189970 24410 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.510s user 0.000s sys 0.016s
W20260809 15:47:08.192888 24410 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.512s user 0.000s sys 0.017s
I20260809 15:47:08.184297 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7660dfa2ef694927b8eda4100a2bcd9b" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:47:08.194169 24471 raft_consensus.cc:2454] T 7660dfa2ef694927b8eda4100a2bcd9b P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 5.
I20260809 15:47:08.140002 27988 raft_consensus.cc:2726] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260809 15:47:08.170014 23529 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.622s user 0.011s sys 0.050s
W20260809 15:47:08.174661 23228 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.601s user 0.008s sys 0.015s
W20260809 15:47:08.195583 23529 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.666s user 0.012s sys 0.050s
I20260809 15:47:08.196447 28086 raft_consensus.cc:482] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
I20260809 15:47:08.196962 28086 raft_consensus.cc:504] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:08.199913 28086 leader_election.cc:288] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:08.205786 23282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ced718c1a5c4da79cb8c1ebd1435b87" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:08.207921 23286 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55434 (request call id 272) took 3712 ms. Trace:
I20260809 15:47:08.208308 23286 rpcz_store.cc:276] 0809 15:47:04.495228 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:06.449003 (+1953775us) service_pool.cc:227] Handling call
0809 15:47:06.449274 (+ 271us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:47:07.191194 (+741920us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:47:08.207685 (+1016491us) spinlock_profiling.cc:243] Waited 736 ms on lock 0x7b5400089b54. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e9220f51 00007f34e921f99e 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772 00007f34e764a94c 00007f34e7649d86
0809 15:47:08.207860 (+ 175us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1693646080}
I20260809 15:47:08.214346 23402 raft_consensus.cc:1261] T f9e7cadcd24d45fcbfa84209225e8f2d P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Refusing update from remote peer 67b3aeadfa7a432abe8af4d292e04ab9: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260809 15:47:08.215626 24452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a51c0baaa6e432997c9d3552af4f254" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:47:08.222344 23282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4564dfc6d4094c1b9240b367ccecf3b1" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:47:08.181802 27761 raft_consensus.cc:504] T 00a1881267a14ba8b78c81040f17a014 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:08.225240 23470 leader_election.cc:302] T 0acc47ce21c3422698d1ef7277c3ed63 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
W20260809 15:47:08.225764 28050 log.cc:924] Time spent T 953eea268e17491ca76f3a555860a512 P 67b3aeadfa7a432abe8af4d292e04ab9: Append to log took a long time: real 0.052s user 0.000s sys 0.000s
I20260809 15:47:08.203552 27493 raft_consensus.cc:482] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
I20260809 15:47:08.226300 27493 raft_consensus.cc:504] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:08.172595 27762 raft_consensus.cc:504] T d1a134ba33ea4076bbaac0ae460c17ba P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:08.227864 27518 raft_consensus.cc:504] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:08.228375 27493 leader_election.cc:288] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:08.235622 23528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2424fd9bd7f14e508fd106cbfda786a4" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:08.193133 28051 raft_consensus.cc:504] T 0c29937a80724ddb9d07a27534a146f3 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:08.236099 27892 raft_consensus.cc:482] T 28e17dcb5efe45b8b6a8c42da03b4819 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:47:08.279675 23283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1a134ba33ea4076bbaac0ae460c17ba" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:08.280922 27892 raft_consensus.cc:504] T 28e17dcb5efe45b8b6a8c42da03b4819 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:08.279755 23274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3296ef50be254fc29cfc77d22a8f7a23" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:08.281533 23283 raft_consensus.cc:2396] T d1a134ba33ea4076bbaac0ae460c17ba P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d for term 7 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260809 15:47:08.282397 23274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2d4f749fcfd4fb3bfb8115af3471598" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:08.282982 23274 raft_consensus.cc:2454] T d2d4f749fcfd4fb3bfb8115af3471598 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 6.
I20260809 15:47:08.282469 23283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "213543cdbe814c89a17a963469922c1b" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:08.275352 23277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07fcabb569414bddbd39df6cedd24e64" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:08.283707 23283 raft_consensus.cc:2454] T 213543cdbe814c89a17a963469922c1b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 3.
I20260809 15:47:08.284019 23277 raft_consensus.cc:2396] T 07fcabb569414bddbd39df6cedd24e64 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d for term 8 because replica has last-logged OpId of term: 7 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20260809 15:47:08.283911 23274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "297c82ed906d4b48b184985eed3de601" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:08.265087 23282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e71a05ba1d4d42bd95c3f85f9313399e" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:08.284582 23274 raft_consensus.cc:2454] T 297c82ed906d4b48b184985eed3de601 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 8.
I20260809 15:47:08.284739 23404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f9d6565aa4c45a3b3f01210d0eb71f7" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:08.285339 23404 raft_consensus.cc:2454] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 4.
I20260809 15:47:08.204133 27956 raft_consensus.cc:2781] T 0a51c0baaa6e432997c9d3552af4f254 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260809 15:47:08.202328 28036 raft_consensus.cc:2781] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260809 15:47:08.238071 23529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba56725e42f74a1baa4c04d9660727f9" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:08.281028 23270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2db057fd10e149a9895dff6ef8c9f038" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:08.287143 23285 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53554 (request call id 1712) took 3262 ms. Trace:
I20260809 15:47:08.239161 23400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73f6f44c259547f1b05a3a434616fa54" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:08.286617 23286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05ad7d8fdca64e3da527280ca5be9280" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
W20260809 15:47:08.429909 23468 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.68:34689, user_credentials={real_user=slave}} blocked reactor thread for 191895us
W20260809 15:47:08.460080 23517 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44018 (request call id 300) took 1660 ms (client timeout 1721 ms). Trace:
I20260809 15:47:08.413038 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e4ffcf83c234ae7910ddef3ec0b4bda" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:47:08.356590 23528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0721a4403a440f383318a3cea9d1442" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:08.427294 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998c6cee72cf40228c6dba064c0db2da" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:47:08.459308 23276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c29937a80724ddb9d07a27534a146f3" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 8 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
W20260809 15:47:08.475910 23517 rpcz_store.cc:269] 0809 15:47:06.799198 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:06.799334 (+ 136us) service_pool.cc:227] Handling call
0809 15:47:08.459906 (+1660572us) spinlock_profiling.cc:243] Waited 203 ms on lock 0x7b54000f87d4. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e92242ae 00007f34ee161c27 00007f34e7849c04 00007f34e7849b92 00007f34e7849ac1 00007f34e7849a3c 00007f34e7848cf2 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:47:08.460023 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":467888384}
I20260809 15:47:08.391201 23351 leader_election.cc:302] T d2d4f749fcfd4fb3bfb8115af3471598 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:47:08.358649 23274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "907bacfa79d04a55a68c8dbe60056462" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:08.474901 23270 raft_consensus.cc:2454] T 2db057fd10e149a9895dff6ef8c9f038 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 6.
W20260809 15:47:08.461280 23353 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 264528us
I20260809 15:47:08.357054 23392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "538a228ddfb54bc6a36528d5ff07597b" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:08.290184 23282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c48a3049897452383257aa757834fb5" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:08.251645 23345 leader_election.cc:302] T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d; no voters:
W20260809 15:47:08.479439 23470 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 254254us
I20260809 15:47:08.361462 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "907bacfa79d04a55a68c8dbe60056462" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:08.484937 23286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0da4d5283c8a46878bd39b85e5c71f66" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:08.485553 23402 raft_consensus.cc:2379] T 907bacfa79d04a55a68c8dbe60056462 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 in current term 7: Already voted for candidate c0f38c1908414566aa9115bb2cbdd85d in this term.
W20260809 15:47:08.386521 27544 consensus_meta.cc:287] T c0741ab44d99439abc3aa3bdd27ee160 P 61e69695258f48baa984c9d395e8ce3f: Time spent flushing consensus metadata: real 0.534s user 0.004s sys 0.004s
I20260809 15:47:08.356628 27735 consensus_queue.cc:1035] T c0741ab44d99439abc3aa3bdd27ee160 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.124s
I20260809 15:47:08.480513 23520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a88421dba80d4d3388aa5eb62e23853b" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:08.539240 23520 raft_consensus.cc:1261] T 953eea268e17491ca76f3a555860a512 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Refusing update from remote peer 67b3aeadfa7a432abe8af4d292e04ab9: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260809 15:47:08.542514 23280 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53554 (request call id 1707) took 3875 ms. Trace:
I20260809 15:47:08.542881 23280 rpcz_store.cc:276] 0809 15:47:04.667385 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:06.721507 (+2054122us) service_pool.cc:227] Handling call
0809 15:47:06.721755 (+ 248us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:47:07.805156 (+1083401us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:47:08.542282 (+737126us) spinlock_profiling.cc:243] Waited 1.07 s on lock 0x7b54000415d4. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e9220f51 00007f34e921f99e 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772 00007f34e764a94c 00007f34e7649d86
0809 15:47:08.542455 (+ 173us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2451038336}
W20260809 15:47:08.228492 28033 log.cc:924] Time spent T 4bcac4b24e7a4448abdc934bba5a624e P 61e69695258f48baa984c9d395e8ce3f: Append to log took a long time: real 0.169s user 0.000s sys 0.000s
I20260809 15:47:08.481094 23528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3622f6e3e3641cea1960e3a31498bb7" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:08.412503 23404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0721a4403a440f383318a3cea9d1442" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
W20260809 15:47:08.557083 24410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 877955us
I20260809 15:47:08.480513 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba56725e42f74a1baa4c04d9660727f9" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:08.412798 23396 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55320 (request call id 1539) took 3469 ms. Trace:
I20260809 15:47:08.479549 27492 raft_consensus.cc:2781] T 0acc47ce21c3422698d1ef7277c3ed63 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260809 15:47:08.243628 23395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "763e58d6ba644eea830a6c1664d39c9b" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:08.483726 23274 raft_consensus.cc:2454] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 7.
I20260809 15:47:08.484505 23517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0741ab44d99439abc3aa3bdd27ee160" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:08.485894 23283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cef3e7277d7d4760824a3436ed4a847d" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:47:08.485908 23470 leader_election.cc:302] T ae6d2c08a7a94f58ac8dd698e3fd456c P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
W20260809 15:47:08.255030 27893 consensus_meta.cc:287] T 73f6f44c259547f1b05a3a434616fa54 P c0f38c1908414566aa9115bb2cbdd85d: Time spent flushing consensus metadata: real 0.559s user 0.001s sys 0.007s
I20260809 15:47:08.357884 28000 leader_election.cc:288] T 3924b648461e463b9d7925e5671d68ae P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:08.215880 27688 leader_election.cc:288] T d3be1e356b254d22b995be32f1c12f6f P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 9 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:47:08.485572 23286 raft_consensus.cc:2454] T 0da4d5283c8a46878bd39b85e5c71f66 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 5.
I20260809 15:47:08.513612 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0da4d5283c8a46878bd39b85e5c71f66" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
W20260809 15:47:08.514413 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 123275us
I20260809 15:47:08.483249 23276 raft_consensus.cc:2454] T 0c29937a80724ddb9d07a27534a146f3 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 7.
I20260809 15:47:08.279580 27928 raft_consensus.cc:482] T f7715d71dbac447b9351fc5eb36c5e80 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:47:08.516474 23282 raft_consensus.cc:2454] T 0c48a3049897452383257aa757834fb5 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 5.
W20260809 15:47:08.279361 28032 consensus_meta.cc:287] T 563daf42d7784db195ea99c3cb21b370 P c0f38c1908414566aa9115bb2cbdd85d: Time spent flushing consensus metadata: real 0.536s user 0.002s sys 0.006s
I20260809 15:47:08.484757 23228 leader_election.cc:302] T 7baa2492df0340c4a5ac241959ce1ac3 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters: c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:08.384847 24406 leader_election.cc:302] T 294d4bcc1e7a477cb56455581277a60f P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
W20260809 15:47:08.651759 23472 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 57665.9us
W20260809 15:47:08.652505 24406 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 267714us
W20260809 15:47:08.457999 27886 consensus_meta.cc:287] T 998c6cee72cf40228c6dba064c0db2da P c0f38c1908414566aa9115bb2cbdd85d: Time spent flushing consensus metadata: real 0.653s user 0.008s sys 0.000s
I20260809 15:47:08.502149 23468 leader_election.cc:302] T 7660dfa2ef694927b8eda4100a2bcd9b P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c; no voters:
I20260809 15:47:08.384879 24448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5712f6b6fffc4cc5a8ddfbe63e4097df" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:47:08.240530 23399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2d4f749fcfd4fb3bfb8115af3471598" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:08.658353 23274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba56725e42f74a1baa4c04d9660727f9" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:08.477026 24452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a823d35f4f348d1b52ba9e6d80b00d2" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:47:08.658926 23274 raft_consensus.cc:2454] T ba56725e42f74a1baa4c04d9660727f9 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 5.
I20260809 15:47:08.362915 23277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3be1e356b254d22b995be32f1c12f6f" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:47:08.659473 23277 raft_consensus.cc:3037] T d3be1e356b254d22b995be32f1c12f6f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Advancing to term 9
I20260809 15:47:08.239827 23397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3622f6e3e3641cea1960e3a31498bb7" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:08.594985 23404 raft_consensus.cc:3037] T d0721a4403a440f383318a3cea9d1442 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Advancing to term 7
I20260809 15:47:08.596051 23396 rpcz_store.cc:276] 0809 15:47:04.943121 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:07.735066 (+2791945us) service_pool.cc:227] Handling call
0809 15:47:07.744794 (+ 9728us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 15:47:07.745079 (+ 285us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:47:07.745102 (+ 23us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 15:47:07.745479 (+ 377us) log.cc:842] Serialized 26 byte log entry
0809 15:47:08.302683 (+557204us) raft_consensus.cc:1631] Marking committed up to 5
0809 15:47:08.302722 (+ 39us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:47:08.302788 (+ 66us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 15:47:08.412602 (+109814us) raft_consensus.cc:1671] finished
0809 15:47:08.412623 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:47:08.412742 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":743,"raft.queue_time_us":183819,"raft.run_cpu_time_us":111,"raft.run_wall_time_us":113,"thread_start_us":490658,"threads_started":1,"wal-append.queue_time_us":496562}
I20260809 15:47:08.597337 23283 raft_consensus.cc:3037] T cef3e7277d7d4760824a3436ed4a847d P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Advancing to term 8
W20260809 15:47:08.628810 23353 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 113818us
I20260809 15:47:08.661475 23397 raft_consensus.cc:2454] T a3622f6e3e3641cea1960e3a31498bb7 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 5.
I20260809 15:47:08.662858 23353 leader_election.cc:302] T 6ced718c1a5c4da79cb8c1ebd1435b87 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:47:08.632476 23402 raft_consensus.cc:2454] T 0da4d5283c8a46878bd39b85e5c71f66 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 5.
I20260809 15:47:08.632112 23270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "907bacfa79d04a55a68c8dbe60056462" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:08.632807 23276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a51c0baaa6e432997c9d3552af4f254" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:08.515343 23529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ced718c1a5c4da79cb8c1ebd1435b87" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:08.633096 23280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2424fd9bd7f14e508fd106cbfda786a4" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:08.665458 23353 leader_election.cc:302] T 05ad7d8fdca64e3da527280ca5be9280 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:47:08.633904 23351 leader_election.cc:302] T 213543cdbe814c89a17a963469922c1b P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:47:08.635037 23392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cef3e7277d7d4760824a3436ed4a847d" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:47:08.596380 23395 raft_consensus.cc:2454] T 763e58d6ba644eea830a6c1664d39c9b P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 7.
I20260809 15:47:08.667373 23392 raft_consensus.cc:2354] T cef3e7277d7d4760824a3436ed4a847d P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 7. Current term is 8.
I20260809 15:47:08.412631 28061 raft_consensus.cc:482] T f7715d71dbac447b9351fc5eb36c5e80 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:47:08.474049 23285 rpcz_store.cc:276] 0809 15:47:05.024287 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:07.165966 (+2141679us) service_pool.cc:227] Handling call
0809 15:47:07.166277 (+ 311us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 15:47:07.166716 (+ 439us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:47:07.166745 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 15:47:07.167154 (+ 409us) log.cc:842] Serialized 26 byte log entry
0809 15:47:07.924418 (+757264us) raft_consensus.cc:1631] Marking committed up to 3
0809 15:47:07.971928 (+ 47510us) log.cc:842] Serialized 14 byte log entry
0809 15:47:07.972939 (+ 1011us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:47:07.973003 (+ 64us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 15:47:08.286951 (+313948us) raft_consensus.cc:1671] finished
0809 15:47:08.286969 (+ 18us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:47:08.287090 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":40076,"prepare.run_cpu_time_us":52,"prepare.run_wall_time_us":3474,"raft.queue_time_us":153419,"raft.run_cpu_time_us":266,"raft.run_wall_time_us":14327,"thread_start_us":719577,"threads_started":1,"wal-append.queue_time_us":880047}
I20260809 15:47:08.513775 27798 leader_election.cc:288] T a88421dba80d4d3388aa5eb62e23853b P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:08.645183 23286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0721a4403a440f383318a3cea9d1442" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:08.671124 23286 raft_consensus.cc:2379] T d0721a4403a440f383318a3cea9d1442 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d in current term 7: Already voted for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in this term.
I20260809 15:47:08.359184 27334 consensus_queue.cc:1035] T 2db057fd10e149a9895dff6ef8c9f038 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.287s
W20260809 15:47:08.676924 23468 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.68:34689, user_credentials={real_user=slave}} blocked reactor thread for 174852us
W20260809 15:47:08.646082 27661 consensus_meta.cc:287] T 294d4bcc1e7a477cb56455581277a60f P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent flushing consensus metadata: real 0.600s user 0.007s sys 0.001s
I20260809 15:47:08.593585 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee02fff2c9c48a19f43eb358a89e7b0" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:47:08.223364 27935 leader_election.cc:288] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:08.648149 23519 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44018 (request call id 305) took 1564 ms. Trace:
I20260809 15:47:08.648388 23282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3622f6e3e3641cea1960e3a31498bb7" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
W20260809 15:47:08.391759 27787 consensus_meta.cc:287] T 3d880de657e544468c82157769622a26 P 61e69695258f48baa984c9d395e8ce3f: Time spent flushing consensus metadata: real 0.537s user 0.008s sys 0.000s
I20260809 15:47:08.477175 28036 raft_consensus.cc:482] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:47:08.238857 23394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07fcabb569414bddbd39df6cedd24e64" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
W20260809 15:47:08.654968 23470 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 169126us
W20260809 15:47:08.654958 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 170266us
W20260809 15:47:08.654902 23345 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.68:34689, user_credentials={real_user=slave}} blocked reactor thread for 403304us
I20260809 15:47:08.223140 27954 leader_election.cc:288] T 28e17dcb5efe45b8b6a8c42da03b4819 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:47:08.675045 23397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5c2e425a74b4e64ae70b5def750434c" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:08.655611 27518 leader_election.cc:288] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 7 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:08.721495 23470 leader_election.cc:302] T 4b30f28311964a0b9149a50495e83d8f P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:47:08.678704 23270 raft_consensus.cc:2396] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d for term 8 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20260809 15:47:08.594206 28024 raft_consensus.cc:482] T f9e7cadcd24d45fcbfa84209225e8f2d P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:08.628790 27982 raft_consensus.cc:2781] T d2d4f749fcfd4fb3bfb8115af3471598 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260809 15:47:08.679766 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 45968.6us
W20260809 15:47:08.681341 28090 log.cc:924] Time spent T 0a51c0baaa6e432997c9d3552af4f254 P 67b3aeadfa7a432abe8af4d292e04ab9: Append to log took a long time: real 0.224s user 0.000s sys 0.001s
I20260809 15:47:08.631413 27492 raft_consensus.cc:482] T 0acc47ce21c3422698d1ef7277c3ed63 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Starting leader election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:47:08.683418 23282 raft_consensus.cc:2454] T a3622f6e3e3641cea1960e3a31498bb7 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 5.
I20260809 15:47:08.759678 23351 leader_election.cc:302] T 297c82ed906d4b48b184985eed3de601 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:47:08.683014 23519 rpcz_store.cc:276] 0809 15:47:07.084027 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:07.084860 (+ 833us) service_pool.cc:227] Handling call
0809 15:47:08.647993 (+1563133us) spinlock_profiling.cc:243] Waited 66.2 ms on lock 0x7b54001fc054. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e92242ae 00007f34ee161c27 00007f34e7849c04 00007f34e7849b92 00007f34e7849ac1 00007f34e7849a3c 00007f34e7848cf2 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:47:08.648094 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":152287104}
I20260809 15:47:08.679647 28039 raft_consensus.cc:2781] T 05ad7d8fdca64e3da527280ca5be9280 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260809 15:47:08.698487 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99bdb93503db4de4a1cfb0161ea0c07c" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:08.716176 23394 raft_consensus.cc:2454] T 07fcabb569414bddbd39df6cedd24e64 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 7.
W20260809 15:47:08.717290 24410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 86564us
I20260809 15:47:08.717475 23345 leader_election.cc:302] T 472816994d814cb198b69e5248697440 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c
W20260809 15:47:08.773388 23345 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.68:34689, user_credentials={real_user=slave}} blocked reactor thread for 56018us
I20260809 15:47:08.774087 23345 leader_election.cc:302] T 3296ef50be254fc29cfc77d22a8f7a23 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 61e69695258f48baa984c9d395e8ce3f, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:47:08.675849 23399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b30f28311964a0b9149a50495e83d8f" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:08.663970 27493 raft_consensus.cc:2781] T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260809 15:47:08.676538 23353 leader_election.cc:302] T d1a134ba33ea4076bbaac0ae460c17ba P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:47:08.653331 27893 raft_consensus.cc:504] T 73f6f44c259547f1b05a3a434616fa54 P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:08.665165 28059 raft_consensus.cc:2781] T 7baa2492df0340c4a5ac241959ce1ac3 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260809 15:47:08.675597 27751 raft_consensus.cc:2726] T 6ced718c1a5c4da79cb8c1ebd1435b87 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260809 15:47:08.680501 28000 raft_consensus.cc:2781] T ae6d2c08a7a94f58ac8dd698e3fd456c P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260809 15:47:08.687597 27704 raft_consensus.cc:2781] T 213543cdbe814c89a17a963469922c1b P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260809 15:47:08.726640 27688 raft_consensus.cc:2726] T 294d4bcc1e7a477cb56455581277a60f P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260809 15:47:08.651355 27544 raft_consensus.cc:504] T c0741ab44d99439abc3aa3bdd27ee160 P 61e69695258f48baa984c9d395e8ce3f [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:08.779855 24410 leader_election.cc:302] T 4bcac4b24e7a4448abdc934bba5a624e P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c
I20260809 15:47:08.688020 27992 raft_consensus.cc:2781] T 7660dfa2ef694927b8eda4100a2bcd9b P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260809 15:47:08.654379 27928 raft_consensus.cc:504] T f7715d71dbac447b9351fc5eb36c5e80 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:08.689769 24468 raft_consensus.cc:2454] T cee02fff2c9c48a19f43eb358a89e7b0 P 61e69695258f48baa984c9d395e8ce3f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 7.
I20260809 15:47:08.657639 28032 raft_consensus.cc:504] T 563daf42d7784db195ea99c3cb21b370 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:08.718356 28036 raft_consensus.cc:3037] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Advancing to term 7
W20260809 15:47:08.720242 27984 consensus_meta.cc:287] T 16ca412a0dae4feebe85dd01bf1a9173 P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent flushing consensus metadata: real 0.598s user 0.008s sys 0.000s
I20260809 15:47:08.674822 27886 raft_consensus.cc:504] T 998c6cee72cf40228c6dba064c0db2da P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:08.759452 27982 raft_consensus.cc:482] T d2d4f749fcfd4fb3bfb8115af3471598 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Starting leader election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:47:08.771090 28039 raft_consensus.cc:482] T 05ad7d8fdca64e3da527280ca5be9280 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Starting leader election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
I20260809 15:47:08.779158 27363 raft_consensus.cc:2781] T 4b30f28311964a0b9149a50495e83d8f P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260809 15:47:08.716517 23228 leader_election.cc:302] T ae6d2c08a7a94f58ac8dd698e3fd456c P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:47:08.851609 23273 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55434 (request call id 269) took 4361 ms. Trace:
I20260809 15:47:08.689522 23274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a88421dba80d4d3388aa5eb62e23853b" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:08.889918 23274 raft_consensus.cc:2454] T a88421dba80d4d3388aa5eb62e23853b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 5.
I20260809 15:47:08.688650 23277 raft_consensus.cc:2454] T d3be1e356b254d22b995be32f1c12f6f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 9.
I20260809 15:47:08.892297 28086 raft_consensus.cc:482] T 2424fd9bd7f14e508fd106cbfda786a4 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:47:08.894443 27945 raft_consensus.cc:2781] T 297c82ed906d4b48b184985eed3de601 P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260809 15:47:08.893472 28086 raft_consensus.cc:504] T 2424fd9bd7f14e508fd106cbfda786a4 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:08.896342 27751 raft_consensus.cc:2726] T d1a134ba33ea4076bbaac0ae460c17ba P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260809 15:47:08.896911 23193 auto_rebalancer.cc:254] failed to send replica move request: Illegal state: Replica 65c36d164b0540f099d592f38307ee6c is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 7 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } }
I20260809 15:47:08.898015 23280 raft_consensus.cc:1226] T d3be1e356b254d22b995be32f1c12f6f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 9 FOLLOWER]: Rejecting Update request from peer c0f38c1908414566aa9115bb2cbdd85d for earlier term 8. Current term is 9. Ops: []
I20260809 15:47:08.901304 28086 leader_election.cc:288] T 2424fd9bd7f14e508fd106cbfda786a4 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:08.858341 23283 raft_consensus.cc:2454] T cef3e7277d7d4760824a3436ed4a847d P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 8.
I20260809 15:47:08.775138 23345 leader_election.cc:302] T e71a05ba1d4d42bd95c3f85f9313399e P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:47:08.781626 23404 raft_consensus.cc:2454] T d0721a4403a440f383318a3cea9d1442 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 7.
I20260809 15:47:08.864809 23393 raft_consensus.cc:1226] T 998c6cee72cf40228c6dba064c0db2da P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 7. Current term is 8. Ops: []
I20260809 15:47:08.909598 23393 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55320 (request call id 1558) took 1053 ms. Trace:
I20260809 15:47:08.909996 23393 rpcz_store.cc:276] 0809 15:47:07.856596 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:07.856773 (+ 177us) service_pool.cc:227] Handling call
0809 15:47:07.865631 (+ 8858us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:47:08.908428 (+1042797us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:47:08.909212 (+ 784us) spinlock_profiling.cc:243] Waited 998 ms on lock 0x7b54002ff654. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e9220f51 00007f34e921f99e 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772 00007f34e764a94c 00007f34e7649d86
0809 15:47:08.909519 (+ 307us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2296491392}
I20260809 15:47:08.782305 23399 raft_consensus.cc:2379] T 4b30f28311964a0b9149a50495e83d8f P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 in current term 4: Already voted for candidate 65c36d164b0540f099d592f38307ee6c in this term.
I20260809 15:47:08.866084 23400 raft_consensus.cc:2379] T 73f6f44c259547f1b05a3a434616fa54 P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 in current term 8: Already voted for candidate c0f38c1908414566aa9115bb2cbdd85d in this term.
I20260809 15:47:08.761381 23470 leader_election.cc:302] T e14d118b2eaa42aa8a7cd4025aba7128 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 61e69695258f48baa984c9d395e8ce3f, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:47:08.912902 24451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0741ab44d99439abc3aa3bdd27ee160" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
W20260809 15:47:08.914000 23470 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 152557us
W20260809 15:47:08.868826 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 109076us
I20260809 15:47:08.916440 28077 raft_consensus.cc:2726] T e14d118b2eaa42aa8a7cd4025aba7128 P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260809 15:47:08.866098 23353 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 189492us
W20260809 15:47:08.873456 24406 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 219306us
I20260809 15:47:08.794492 24455 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38734 (request call id 363) took 4381 ms. Trace:
I20260809 15:47:08.918888 23353 leader_election.cc:302] T 07fcabb569414bddbd39df6cedd24e64 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:47:08.920760 27762 leader_election.cc:288] T d1a134ba33ea4076bbaac0ae460c17ba P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:08.919550 24455 rpcz_store.cc:276] 0809 15:47:04.406092 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:04.407375 (+ 1283us) service_pool.cc:227] Handling call
0809 15:47:04.568512 (+161137us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 15:47:05.769192 (+1200680us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:47:05.769747 (+ 555us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 15:47:05.861134 (+ 91387us) log.cc:842] Serialized 26 byte log entry
0809 15:47:07.734864 (+1873730us) raft_consensus.cc:1631] Marking committed up to 5
0809 15:47:07.734928 (+ 64us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:47:07.734994 (+ 66us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 15:47:08.761524 (+1026530us) raft_consensus.cc:1671] finished
0809 15:47:08.761544 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:47:08.786959 (+ 25415us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":87295,"raft.run_cpu_time_us":90,"raft.run_wall_time_us":87,"thread_start_us":3033464,"threads_started":2,"wal-append.queue_time_us":2086830}
I20260809 15:47:08.921440 23353 leader_election.cc:302] T 4564dfc6d4094c1b9240b367ccecf3b1 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:47:08.875934 24472 raft_consensus.cc:1226] T c0741ab44d99439abc3aa3bdd27ee160 P 61e69695258f48baa984c9d395e8ce3f [term 7 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20260809 15:47:08.763465 27492 raft_consensus.cc:3037] T 0acc47ce21c3422698d1ef7277c3ed63 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Advancing to term 8
W20260809 15:47:08.905929 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 189024us
I20260809 15:47:08.895090 27945 raft_consensus.cc:482] T 297c82ed906d4b48b184985eed3de601 P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:47:08.924782 27945 raft_consensus.cc:3037] T 297c82ed906d4b48b184985eed3de601 P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Advancing to term 9
I20260809 15:47:08.884435 23273 rpcz_store.cc:276] 0809 15:47:04.489510 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:04.495566 (+ 6056us) service_pool.cc:227] Handling call
0809 15:47:04.495832 (+ 266us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 15:47:05.487756 (+991924us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:47:05.487787 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 15:47:07.715997 (+2228210us) spinlock_profiling.cc:243] Waited 2.33 ms on lock 0x7b2c0000f228. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e7c03e11 00007f34e7c13442 00007f34e925bc11 00007f34e922058a 00007f34e921f99e 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772
0809 15:47:07.728586 (+ 12589us) log.cc:842] Serialized 26 byte log entry
0809 15:47:08.193679 (+465093us) raft_consensus.cc:1631] Marking committed up to 5
0809 15:47:08.193813 (+ 134us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:47:08.194062 (+ 249us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 15:47:08.850756 (+656694us) raft_consensus.cc:1671] finished
0809 15:47:08.850827 (+ 71us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:47:08.851285 (+ 458us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":446828,"raft.run_cpu_time_us":137,"raft.run_wall_time_us":60051,"spinlock_wait_cycles":5357312,"thread_start_us":1424613,"threads_started":2,"wal-append.queue_time_us":718068}
I20260809 15:47:08.781279 27493 raft_consensus.cc:482] T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Starting leader election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:47:08.927760 27493 raft_consensus.cc:3037] T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Advancing to term 8
I20260809 15:47:08.930054 23223 leader_election.cc:302] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c
I20260809 15:47:08.683117 27661 raft_consensus.cc:504] T 294d4bcc1e7a477cb56455581277a60f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:08.893404 27892 leader_election.cc:288] T 28e17dcb5efe45b8b6a8c42da03b4819 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:08.793651 27704 raft_consensus.cc:482] T 213543cdbe814c89a17a963469922c1b P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Starting leader election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:47:08.804625 27893 leader_election.cc:288] T 73f6f44c259547f1b05a3a434616fa54 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 8 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:08.684015 27787 raft_consensus.cc:504] T 3d880de657e544468c82157769622a26 P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:08.934736 27953 raft_consensus.cc:2726] T 4564dfc6d4094c1b9240b367ccecf3b1 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260809 15:47:08.805042 27928 leader_election.cc:288] T f7715d71dbac447b9351fc5eb36c5e80 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:08.780810 27957 raft_consensus.cc:2726] T 472816994d814cb198b69e5248697440 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260809 15:47:08.816989 27961 consensus_meta.cc:287] T 3d880de657e544468c82157769622a26 P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent flushing consensus metadata: real 0.626s user 0.007s sys 0.000s
I20260809 15:47:08.936672 28086 raft_consensus.cc:2781] T ae6d2c08a7a94f58ac8dd698e3fd456c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260809 15:47:08.806310 27982 raft_consensus.cc:3037] T d2d4f749fcfd4fb3bfb8115af3471598 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Advancing to term 7
I20260809 15:47:08.937168 28086 raft_consensus.cc:482] T ae6d2c08a7a94f58ac8dd698e3fd456c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Starting leader election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:47:08.787460 27988 raft_consensus.cc:2726] T 3296ef50be254fc29cfc77d22a8f7a23 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 15:47:08.798349 27992 raft_consensus.cc:482] T 7660dfa2ef694927b8eda4100a2bcd9b P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Starting leader election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:47:08.938661 27992 raft_consensus.cc:3037] T 7660dfa2ef694927b8eda4100a2bcd9b P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Advancing to term 6
I20260809 15:47:08.939040 27544 leader_election.cc:288] T c0741ab44d99439abc3aa3bdd27ee160 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 7 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:08.940441 27661 leader_election.cc:288] T 294d4bcc1e7a477cb56455581277a60f P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 6 election: Requested vote from peers 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:08.792789 28000 raft_consensus.cc:482] T ae6d2c08a7a94f58ac8dd698e3fd456c P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Starting leader election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:47:08.763154 28024 raft_consensus.cc:504] T f9e7cadcd24d45fcbfa84209225e8f2d P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:08.814872 28032 leader_election.cc:288] T 563daf42d7784db195ea99c3cb21b370 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 6 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:08.807857 28039 raft_consensus.cc:3037] T 05ad7d8fdca64e3da527280ca5be9280 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Advancing to term 7
I20260809 15:47:08.902962 24453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3be1e356b254d22b995be32f1c12f6f" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:47:08.944602 24453 raft_consensus.cc:2379] T d3be1e356b254d22b995be32f1c12f6f P 61e69695258f48baa984c9d395e8ce3f [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 in current term 9: Already voted for candidate 61e69695258f48baa984c9d395e8ce3f in this term.
I20260809 15:47:08.937026 27961 raft_consensus.cc:504] T 3d880de657e544468c82157769622a26 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:08.801385 28051 leader_election.cc:288] T 0c29937a80724ddb9d07a27534a146f3 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:08.679945 28061 raft_consensus.cc:504] T f7715d71dbac447b9351fc5eb36c5e80 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:08.953081 24463 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38734 (request call id 365) took 3437 ms. Trace:
I20260809 15:47:08.953575 24463 rpcz_store.cc:276] 0809 15:47:05.515951 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:06.468754 (+952803us) service_pool.cc:227] Handling call
0809 15:47:07.441329 (+972575us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:47:07.462156 (+ 20827us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:47:07.462187 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 15:47:07.462231 (+ 44us) raft_consensus.cc:1631] Marking committed up to 5
0809 15:47:07.462266 (+ 35us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:47:07.462325 (+ 59us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:47:08.952681 (+1490356us) spinlock_profiling.cc:243] Waited 968 ms on lock 0x7b540040f450. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e921fa09 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:47:08.952866 (+ 185us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2225521920}
I20260809 15:47:08.784790 28059 raft_consensus.cc:482] T 7baa2492df0340c4a5ac241959ce1ac3 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting leader election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:47:08.956972 28061 leader_election.cc:288] T f7715d71dbac447b9351fc5eb36c5e80 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
W20260809 15:47:08.724936 28081 log.cc:924] Time spent T df20671a7b874dbb98c0ef28371c1308 P 67b3aeadfa7a432abe8af4d292e04ab9: Append to log took a long time: real 0.078s user 0.000s sys 0.001s
I20260809 15:47:08.888729 27886 leader_election.cc:288] T 998c6cee72cf40228c6dba064c0db2da P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 8 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:08.905915 27363 raft_consensus.cc:482] T 4b30f28311964a0b9149a50495e83d8f P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:47:08.958482 27363 raft_consensus.cc:3037] T 4b30f28311964a0b9149a50495e83d8f P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Advancing to term 5
I20260809 15:47:08.958122 23228 leader_election.cc:302] T 0acc47ce21c3422698d1ef7277c3ed63 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
W20260809 15:47:08.931494 24410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 102033us
I20260809 15:47:08.959065 27892 consensus_queue.cc:1035] T f9e7cadcd24d45fcbfa84209225e8f2d P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.002s
I20260809 15:47:08.937427 27787 leader_election.cc:288] T 3d880de657e544468c82157769622a26 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 4 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:08.961931 27892 raft_consensus.cc:2781] T 0acc47ce21c3422698d1ef7277c3ed63 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260809 15:47:08.962329 27892 raft_consensus.cc:482] T 0acc47ce21c3422698d1ef7277c3ed63 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Starting leader election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:47:08.962788 27892 raft_consensus.cc:3037] T 0acc47ce21c3422698d1ef7277c3ed63 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Advancing to term 8
W20260809 15:47:08.894394 28125 log.cc:924] Time spent T ba56725e42f74a1baa4c04d9660727f9 P 65c36d164b0540f099d592f38307ee6c: Append to log took a long time: real 0.073s user 0.001s sys 0.000s
I20260809 15:47:08.806160 27984 raft_consensus.cc:504] T 16ca412a0dae4feebe85dd01bf1a9173 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:08.965842 27984 leader_election.cc:288] T 16ca412a0dae4feebe85dd01bf1a9173 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 6 election: Requested vote from peers 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:08.959589 28059 raft_consensus.cc:3037] T 7baa2492df0340c4a5ac241959ce1ac3 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Advancing to term 5
I20260809 15:47:08.937877 27751 raft_consensus.cc:2726] T 07fcabb569414bddbd39df6cedd24e64 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260809 15:47:08.941325 28000 raft_consensus.cc:3037] T ae6d2c08a7a94f58ac8dd698e3fd456c P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Advancing to term 8
I20260809 15:47:08.946893 27961 leader_election.cc:288] T 3d880de657e544468c82157769622a26 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 election: Requested vote from peers 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:08.934695 27704 raft_consensus.cc:3037] T 213543cdbe814c89a17a963469922c1b P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:47:08.956653 28024 leader_election.cc:288] T f9e7cadcd24d45fcbfa84209225e8f2d P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:47:08.959690 27928 raft_consensus.cc:2726] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260809 15:47:08.935788 27953 raft_consensus.cc:2781] T e71a05ba1d4d42bd95c3f85f9313399e P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260809 15:47:08.965867 23470 leader_election.cc:302] T 538a228ddfb54bc6a36528d5ff07597b P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:47:08.983819 27953 raft_consensus.cc:482] T e71a05ba1d4d42bd95c3f85f9313399e P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Starting leader election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
I20260809 15:47:08.984288 27953 raft_consensus.cc:3037] T e71a05ba1d4d42bd95c3f85f9313399e P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Advancing to term 7
I20260809 15:47:08.985108 28029 raft_consensus.cc:2781] T 538a228ddfb54bc6a36528d5ff07597b P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260809 15:47:08.985522 28029 raft_consensus.cc:482] T 538a228ddfb54bc6a36528d5ff07597b P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:47:08.985942 28029 raft_consensus.cc:3037] T 538a228ddfb54bc6a36528d5ff07597b P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Advancing to term 7
I20260809 15:47:08.987244 23402 raft_consensus.cc:1226] T 563daf42d7784db195ea99c3cb21b370 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Rejecting Update request from peer 61e69695258f48baa984c9d395e8ce3f for earlier term 5. Current term is 6. Ops: []
I20260809 15:47:08.984612 23470 leader_election.cc:302] T cef3e7277d7d4760824a3436ed4a847d P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:47:08.941838 28086 raft_consensus.cc:3037] T ae6d2c08a7a94f58ac8dd698e3fd456c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Advancing to term 8
I20260809 15:47:08.971385 23393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3e94b7e50754a81b3d72a07476cb7b9" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:08.972970 23519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c48a3049897452383257aa757834fb5" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:08.989979 28114 raft_consensus.cc:2781] T 4bcac4b24e7a4448abdc934bba5a624e P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260809 15:47:08.919158 27761 leader_election.cc:288] T 00a1881267a14ba8b78c81040f17a014 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:08.990386 23522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d880de657e544468c82157769622a26" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:47:08.990433 23519 raft_consensus.cc:2454] T 0c48a3049897452383257aa757834fb5 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 5.
I20260809 15:47:08.976961 23529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3e94b7e50754a81b3d72a07476cb7b9" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:08.991503 23223 leader_election.cc:302] T cee02fff2c9c48a19f43eb358a89e7b0 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:47:08.978631 23404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "907bacfa79d04a55a68c8dbe60056462" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:08.993548 23404 raft_consensus.cc:2454] T 907bacfa79d04a55a68c8dbe60056462 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 7.
I20260809 15:47:08.994521 27761 raft_consensus.cc:2781] T cee02fff2c9c48a19f43eb358a89e7b0 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260809 15:47:08.994974 27761 raft_consensus.cc:482] T cee02fff2c9c48a19f43eb358a89e7b0 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:47:08.995391 27761 raft_consensus.cc:3037] T cee02fff2c9c48a19f43eb358a89e7b0 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Advancing to term 8
W20260809 15:47:08.990557 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 74664.9us
I20260809 15:47:08.995934 23529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0741ab44d99439abc3aa3bdd27ee160" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:47:08.982764 23395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3be1e356b254d22b995be32f1c12f6f" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:47:08.997644 23351 leader_election.cc:302] T 2db057fd10e149a9895dff6ef8c9f038 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:47:08.988620 28070 raft_consensus.cc:2781] T cef3e7277d7d4760824a3436ed4a847d P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260809 15:47:08.998436 28070 raft_consensus.cc:482] T cef3e7277d7d4760824a3436ed4a847d P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:47:08.999344 27956 raft_consensus.cc:482] T 0a51c0baaa6e432997c9d3552af4f254 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
I20260809 15:47:08.999495 27886 raft_consensus.cc:2781] T 2db057fd10e149a9895dff6ef8c9f038 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260809 15:47:08.998807 28070 raft_consensus.cc:3037] T cef3e7277d7d4760824a3436ed4a847d P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Advancing to term 8
I20260809 15:47:08.999993 27886 raft_consensus.cc:482] T 2db057fd10e149a9895dff6ef8c9f038 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:47:09.000437 27886 raft_consensus.cc:3037] T 2db057fd10e149a9895dff6ef8c9f038 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Advancing to term 7
W20260809 15:47:08.998703 24410 connection.cc:232] RPC call timeout handler was delayed by -1.2290308689698577s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18680000000000002s for a response
I20260809 15:47:08.999704 23351 leader_election.cc:302] T 0da4d5283c8a46878bd39b85e5c71f66 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters:
W20260809 15:47:09.001276 24410 connection.cc:232] RPC call timeout handler was delayed by -1.2246948850806803s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1857s for a response
W20260809 15:47:09.001838 24410 connection.cc:232] RPC call timeout handler was delayed by -1.2212925651110709s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19240000000000002s for a response
W20260809 15:47:09.006289 23353 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 85135us
I20260809 15:47:08.995422 28114 raft_consensus.cc:482] T 4bcac4b24e7a4448abdc934bba5a624e P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Starting leader election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:47:09.005265 27956 raft_consensus.cc:3037] T 0a51c0baaa6e432997c9d3552af4f254 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Advancing to term 7
I20260809 15:47:07.319680 21289 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20260809 15:47:09.030680 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 69879.8us
I20260809 15:47:09.032261 28030 raft_consensus.cc:482] T 6ae14a74a26f434db08c652e097385c8 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
I20260809 15:47:09.033030 28030 raft_consensus.cc:504] T 6ae14a74a26f434db08c652e097385c8 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:09.028579 27751 raft_consensus.cc:2781] T 0da4d5283c8a46878bd39b85e5c71f66 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260809 15:47:09.035470 23351 leader_election.cc:302] T 0c29937a80724ddb9d07a27534a146f3 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:47:09.023113 28114 raft_consensus.cc:3037] T 4bcac4b24e7a4448abdc934bba5a624e P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:47:09.046623 23351 leader_election.cc:302] T ba56725e42f74a1baa4c04d9660727f9 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:47:09.041052 27751 raft_consensus.cc:482] T 0da4d5283c8a46878bd39b85e5c71f66 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Starting leader election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:47:09.047608 28061 raft_consensus.cc:2781] T 0c29937a80724ddb9d07a27534a146f3 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260809 15:47:09.047690 27751 raft_consensus.cc:3037] T 0da4d5283c8a46878bd39b85e5c71f66 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Advancing to term 6
I20260809 15:47:09.048012 28061 raft_consensus.cc:482] T 0c29937a80724ddb9d07a27534a146f3 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:47:09.048365 28061 raft_consensus.cc:3037] T 0c29937a80724ddb9d07a27534a146f3 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Advancing to term 8
I20260809 15:47:09.052488 28032 raft_consensus.cc:2781] T ba56725e42f74a1baa4c04d9660727f9 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260809 15:47:09.052910 28032 raft_consensus.cc:482] T ba56725e42f74a1baa4c04d9660727f9 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Starting leader election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:47:09.053270 28032 raft_consensus.cc:3037] T ba56725e42f74a1baa4c04d9660727f9 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Advancing to term 6
I20260809 15:47:09.057461 28039 raft_consensus.cc:504] T 05ad7d8fdca64e3da527280ca5be9280 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:09.059247 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "907bacfa79d04a55a68c8dbe60056462" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f"
I20260809 15:47:09.059917 27792 tablet_bootstrap.cc:700] T 4b30f28311964a0b9149a50495e83d8f P 61e69695258f48baa984c9d395e8ce3f: Time spent opening tablet: real 0.117s user 0.000s sys 0.003s
I20260809 15:47:09.060293 23351 leader_election.cc:302] T a3622f6e3e3641cea1960e3a31498bb7 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters:
W20260809 15:47:09.063476 23115 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:56222 (request call id 245) took 17382 ms (client timeout 15000 ms). Trace:
I20260809 15:47:09.063302 23118 catalog_manager.cc:5495] T 16ca412a0dae4feebe85dd01bf1a9173 P 61e69695258f48baa984c9d395e8ce3f reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "65c36d164b0540f099d592f38307ee6c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:47:09.065806 23118 catalog_manager.cc:5495] T e71a05ba1d4d42bd95c3f85f9313399e P 61e69695258f48baa984c9d395e8ce3f reported cstate change: term changed from 2 to 6. New cstate: current_term: 6 leader_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:47:09.088579 24463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "294d4bcc1e7a477cb56455581277a60f" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f"
I20260809 15:47:09.086431 27988 consensus_queue.cc:1035] T 00a1881267a14ba8b78c81040f17a014 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.007s
I20260809 15:47:09.015100 28036 raft_consensus.cc:504] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
W20260809 15:47:09.087404 23345 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.68:34689, user_credentials={real_user=slave}} blocked reactor thread for 286480us
W20260809 15:47:09.096396 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 60281.9us
I20260809 15:47:09.122218 28036 leader_election.cc:288] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 7 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:09.068989 27992 raft_consensus.cc:504] T 7660dfa2ef694927b8eda4100a2bcd9b P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:09.123620 23518 raft_consensus.cc:1226] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Rejecting Update request from peer c0f38c1908414566aa9115bb2cbdd85d for earlier term 6. Current term is 7. Ops: []
I20260809 15:47:09.067191 23118 catalog_manager.cc:5495] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 61e69695258f48baa984c9d395e8ce3f reported cstate change: term changed from 5 to 9, leader changed from c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66) to 61e69695258f48baa984c9d395e8ce3f (127.20.202.68). New cstate: current_term: 9 leader_uuid: "61e69695258f48baa984c9d395e8ce3f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 15:47:09.125052 27992 leader_election.cc:288] T 7660dfa2ef694927b8eda4100a2bcd9b P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 6 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:09.125991 23518 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56766 (request call id 2055) took 1232 ms. Trace:
I20260809 15:47:09.126415 23518 rpcz_store.cc:276] 0809 15:47:07.894003 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:07.894192 (+ 189us) service_pool.cc:227] Handling call
0809 15:47:08.648117 (+753925us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:47:09.124128 (+476011us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:47:09.125175 (+ 1047us) spinlock_profiling.cc:243] Waited 475 ms on lock 0x7b54001fc054. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e9220f51 00007f34e921f99e 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772 00007f34e764a94c 00007f34e7649d86
0809 15:47:09.125777 (+ 602us) spinlock_profiling.cc:243] Waited 719 ms on lock 0x7b54001fc050. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e921fa09 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:47:09.125893 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2747309824}
I20260809 15:47:09.129976 28011 consensus_queue.cc:1035] T 297c82ed906d4b48b184985eed3de601 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260809 15:47:09.088423 24453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d880de657e544468c82157769622a26" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f"
I20260809 15:47:09.133222 24453 raft_consensus.cc:2379] T 3d880de657e544468c82157769622a26 P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 in current term 4: Already voted for candidate 61e69695258f48baa984c9d395e8ce3f in this term.
I20260809 15:47:09.064644 27893 raft_consensus.cc:2781] T a3622f6e3e3641cea1960e3a31498bb7 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260809 15:47:09.134078 27893 raft_consensus.cc:482] T a3622f6e3e3641cea1960e3a31498bb7 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Starting leader election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
W20260809 15:47:09.063984 23115 rpcz_store.cc:269] 0809 15:46:51.669989 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:51.670140 (+ 151us) service_pool.cc:227] Handling call
0809 15:47:09.046985 (+17376845us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0809 15:47:00.207633 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0809 15:47:00.461110 (+253477us) write_op.cc:432] Acquiring schema lock in shared mode
0809 15:47:00.469845 (+ 8735us) write_op.cc:435] Acquired schema lock
0809 15:47:00.477907 (+ 8062us) tablet.cc:660] Decoding operations
0809 15:47:00.756548 (+278641us) write_op.cc:620] Acquiring the partition lock for write op
0809 15:47:00.756640 (+ 92us) write_op.cc:641] Partition lock acquired for write op
0809 15:47:00.756671 (+ 31us) tablet.cc:683] Acquiring locks for 18 operations
0809 15:47:00.758007 (+ 1336us) tablet.cc:699] Row locks acquired
0809 15:47:00.758054 (+ 47us) write_op.cc:260] PREPARE: finished
0809 15:47:00.758200 (+ 146us) write_op.cc:270] Start()
0809 15:47:00.758338 (+ 138us) write_op.cc:276] Timestamp: P: 1786290420758140 usec, L: 0
0809 15:47:00.758377 (+ 39us) op_driver.cc:331] REPLICATION: starting
0809 15:47:00.856237 (+ 97860us) log.cc:842] Serialized 7124 byte log entry
0809 15:47:01.330284 (+474047us) spinlock_profiling.cc:243] Waited 2.94 ms on lock 0x7b44000232cc. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e613cf11 00007f34e620aa95 00007f34e620b4b2 00007f34e8eb3511 00007f34e8eaee1a 00007f34e62da5fe 00007f34ef111a39 00007f34e62f78bb 00007f34e62f5cbf 00007f34e62f7e22 00007f34e8ea15bb
0809 15:47:02.237765 (+907481us) spinlock_profiling.cc:243] Waited 5.67 ms on lock 0x7b44000232cc. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e613cf11 00007f34e620aa95 00007f34e620b4b2 00007f34e8eb3511 00007f34e8eaee1a 00007f34e8ea2bee 00007f34e8ea1f36 00007f34e8ea1b58 00007f34e8eab551 00007f34e8eab509 00007f34e8eab499
0809 15:47:02.668425 (+430660us) op_driver.cc:444] REPLICATION: finished
0809 15:47:04.014115 (+1345690us) write_op.cc:301] APPLY: starting
0809 15:47:04.027834 (+ 13719us) tablet.cc:1365] starting BulkCheckPresence
0809 15:47:04.028221 (+ 387us) tablet.cc:1368] finished BulkCheckPresence
0809 15:47:04.028242 (+ 21us) tablet.cc:1370] starting ApplyRowOperation cycle
0809 15:47:04.030165 (+ 1923us) tablet.cc:1381] finished ApplyRowOperation cycle
0809 15:47:04.030524 (+ 359us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=18
0809 15:47:04.030569 (+ 45us) write_op.cc:312] APPLY: finished
0809 15:47:04.031003 (+ 434us) log.cc:842] Serialized 125 byte log entry
0809 15:47:04.242279 (+211276us) write_op.cc:489] Releasing partition, row and schema locks
0809 15:47:04.256758 (+ 14479us) write_op.cc:454] Released schema lock
0809 15:47:04.265627 (+ 8869us) write_op.cc:341] FINISH: Updating metrics
0809 15:47:04.278106 (+ 12479us) write_op.cc:489] Releasing partition, row and schema locks
0809 15:47:04.278135 (+ 29us) write_op.cc:454] Released schema lock
0809 15:47:06.798645 (+2520510us) spinlock_profiling.cc:243] Waited 6.43 ms on lock 0x7b44000232cc. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e613cf11 00007f34e620aa95 00007f34e620b4b2 00007f34e8eb3511 00007f34e8eaee1a 00007f34e8ea2bee 00007f34e8ea1f36 00007f34e8ea1b58 00007f34e8eab551 00007f34e8eab509 00007f34e8eab499
Metrics: {"mutex_wait_us":1616083,"reported_tablets":35,"child_traces":[["op",{"apply.queue_time_us":1170869,"apply.run_cpu_time_us":82465,"apply.run_wall_time_us":286097,"num_ops":18,"prepare.queue_time_us":410322,"prepare.run_cpu_time_us":60142,"prepare.run_wall_time_us":1253649,"raft.queue_time_us":209846,"raft.run_cpu_time_us":47804,"raft.run_wall_time_us":1515936,"replication_time_us":1885173,"spinlock_wait_cycles":34606592,"thread_start_us":2316056,"threads_started":5,"wal-append.queue_time_us":957858,"wal-append.run_cpu_time_us":96706,"wal-append.run_wall_time_us":5910371}]]}
I20260809 15:47:09.159857 28084 consensus_queue.cc:1035] T ba56725e42f74a1baa4c04d9660727f9 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260809 15:47:09.160570 23527 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56770 (request call id 1432) took 1443 ms. Trace:
I20260809 15:47:09.162843 23527 rpcz_store.cc:276] 0809 15:47:07.717135 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:07.717283 (+ 148us) service_pool.cc:227] Handling call
0809 15:47:07.717606 (+ 323us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 15:47:08.945325 (+1227719us) spinlock_profiling.cc:243] Waited 5.07 ms on lock 0x7b2c00190e38. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e75c7301 00007f34e75e18ba 00007f34e921981d 00007f34e921ff6b 00007f34e921f99e 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772
0809 15:47:08.945560 (+ 235us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:47:08.945584 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0809 15:47:08.946605 (+ 1021us) log.cc:842] Serialized 52 byte log entry
0809 15:47:09.022513 (+ 75908us) raft_consensus.cc:1631] Marking committed up to 2
0809 15:47:09.022549 (+ 36us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:47:09.029189 (+ 6640us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 15:47:09.061859 (+ 32670us) raft_consensus.cc:1671] finished
0809 15:47:09.061890 (+ 31us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:47:09.160270 (+ 98380us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":28513,"raft.run_cpu_time_us":124,"raft.run_wall_time_us":24098,"spinlock_wait_cycles":11664768,"thread_start_us":526852,"threads_started":2,"wal-append.queue_time_us":98154}
I20260809 15:47:09.073341 27493 raft_consensus.cc:504] T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:09.165181 23393 raft_consensus.cc:2379] T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 61e69695258f48baa984c9d395e8ce3f in current term 8: Already voted for candidate c0f38c1908414566aa9115bb2cbdd85d in this term.
I20260809 15:47:09.163949 27749 consensus_queue.cc:1046] T 998c6cee72cf40228c6dba064c0db2da P 65c36d164b0540f099d592f38307ee6c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.003s
I20260809 15:47:09.125492 23118 catalog_manager.cc:5495] T e14d118b2eaa42aa8a7cd4025aba7128 P 61e69695258f48baa984c9d395e8ce3f reported cstate change: term changed from 5 to 8, leader changed from 65c36d164b0540f099d592f38307ee6c (127.20.202.67) to 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65). New cstate: current_term: 8 leader_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:47:09.202543 28000 raft_consensus.cc:504] T ae6d2c08a7a94f58ac8dd698e3fd456c P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:09.166780 27493 leader_election.cc:288] T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 8 election: Requested vote from peers 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:09.068022 28030 leader_election.cc:288] T 6ae14a74a26f434db08c652e097385c8 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:47:09.247367 28000 leader_election.cc:288] T ae6d2c08a7a94f58ac8dd698e3fd456c P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 8 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:47:09.248358 23351 leader_election.cc:302] T a88421dba80d4d3388aa5eb62e23853b P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters:
W20260809 15:47:09.248571 23353 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 160969us
I20260809 15:47:09.202754 24472 raft_consensus.cc:2454] T 907bacfa79d04a55a68c8dbe60056462 P 61e69695258f48baa984c9d395e8ce3f [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 7.
I20260809 15:47:09.163153 27893 raft_consensus.cc:3037] T a3622f6e3e3641cea1960e3a31498bb7 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Advancing to term 6
I20260809 15:47:09.245093 23118 catalog_manager.cc:5495] T 563daf42d7784db195ea99c3cb21b370 P 61e69695258f48baa984c9d395e8ce3f reported cstate change: term changed from 2 to 5. New cstate: current_term: 5 leader_uuid: "61e69695258f48baa984c9d395e8ce3f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 15:47:09.261075 28070 raft_consensus.cc:504] T cef3e7277d7d4760824a3436ed4a847d P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:09.262641 24448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7660dfa2ef694927b8eda4100a2bcd9b" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f"
I20260809 15:47:09.164004 28039 leader_election.cc:288] T 05ad7d8fdca64e3da527280ca5be9280 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 7 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:09.265478 24448 raft_consensus.cc:3037] T 7660dfa2ef694927b8eda4100a2bcd9b P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Advancing to term 6
I20260809 15:47:09.238613 27761 raft_consensus.cc:504] T cee02fff2c9c48a19f43eb358a89e7b0 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:09.207810 28059 raft_consensus.cc:504] T 7baa2492df0340c4a5ac241959ce1ac3 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:09.203193 28032 raft_consensus.cc:504] T ba56725e42f74a1baa4c04d9660727f9 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:09.267657 24452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3aa6a9a52804df0ba9b0ca3c82f38c3" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f"
I20260809 15:47:09.327580 24452 raft_consensus.cc:3037] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 61e69695258f48baa984c9d395e8ce3f [term 6 FOLLOWER]: Advancing to term 7
I20260809 15:47:09.327960 23351 leader_election.cc:302] T cef3e7277d7d4760824a3436ed4a847d P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:47:09.291793 24455 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3924b648461e463b9d7925e5671d68ae" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:47:09.306027 28086 raft_consensus.cc:504] T ae6d2c08a7a94f58ac8dd698e3fd456c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:09.296572 28114 raft_consensus.cc:504] T 4bcac4b24e7a4448abdc934bba5a624e P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:09.291386 27493 raft_consensus.cc:2781] T a88421dba80d4d3388aa5eb62e23853b P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260809 15:47:09.287427 26966 raft_consensus.cc:3032] T 998c6cee72cf40228c6dba064c0db2da P 65c36d164b0540f099d592f38307ee6c [term 7 LEADER]: Stepping down as leader of term 7
I20260809 15:47:09.299490 27892 raft_consensus.cc:504] T 0acc47ce21c3422698d1ef7277c3ed63 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:09.290891 27945 raft_consensus.cc:504] T 297c82ed906d4b48b184985eed3de601 P c0f38c1908414566aa9115bb2cbdd85d [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:09.292634 28084 consensus_queue.cc:1046] T c0741ab44d99439abc3aa3bdd27ee160 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.017s
I20260809 15:47:09.309036 27953 raft_consensus.cc:504] T e71a05ba1d4d42bd95c3f85f9313399e P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:09.309391 27956 raft_consensus.cc:504] T 0a51c0baaa6e432997c9d3552af4f254 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:09.309891 28070 leader_election.cc:288] T cef3e7277d7d4760824a3436ed4a847d P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 8 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:47:09.310678 27761 leader_election.cc:288] T cee02fff2c9c48a19f43eb358a89e7b0 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 8 election: Requested vote from peers 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:09.287796 23118 catalog_manager.cc:5495] T 3924b648461e463b9d7925e5671d68ae P 61e69695258f48baa984c9d395e8ce3f reported cstate change: term changed from 6 to 8. New cstate: current_term: 8 leader_uuid: "61e69695258f48baa984c9d395e8ce3f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 15:47:09.327172 28032 leader_election.cc:288] T ba56725e42f74a1baa4c04d9660727f9 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 6 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:09.307869 27751 raft_consensus.cc:504] T 0da4d5283c8a46878bd39b85e5c71f66 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:09.207367 27886 raft_consensus.cc:504] T 2db057fd10e149a9895dff6ef8c9f038 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:09.267875 24463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16ca412a0dae4feebe85dd01bf1a9173" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f"
I20260809 15:47:09.316740 24448 raft_consensus.cc:2454] T 7660dfa2ef694927b8eda4100a2bcd9b P 61e69695258f48baa984c9d395e8ce3f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 6.
I20260809 15:47:09.298103 27492 raft_consensus.cc:504] T 0acc47ce21c3422698d1ef7277c3ed63 P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:09.317713 27893 raft_consensus.cc:504] T a3622f6e3e3641cea1960e3a31498bb7 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:09.321026 27363 raft_consensus.cc:504] T 4b30f28311964a0b9149a50495e83d8f P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } attrs { replace: true } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } }
I20260809 15:47:09.204457 28061 raft_consensus.cc:504] T 0c29937a80724ddb9d07a27534a146f3 P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:09.322788 28039 consensus_queue.cc:1035] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260809 15:47:09.324790 27982 raft_consensus.cc:504] T d2d4f749fcfd4fb3bfb8115af3471598 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:47:09.291422 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 77112.8us
W20260809 15:47:09.335037 24410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 292386us
I20260809 15:47:09.334581 23345 leader_election.cc:302] T 907bacfa79d04a55a68c8dbe60056462 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 61e69695258f48baa984c9d395e8ce3f, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:47:09.172495 27704 raft_consensus.cc:504] T 213543cdbe814c89a17a963469922c1b P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:09.300568 27334 consensus_queue.cc:1035] T 0c29937a80724ddb9d07a27534a146f3 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.283s
I20260809 15:47:09.339643 27493 raft_consensus.cc:482] T a88421dba80d4d3388aa5eb62e23853b P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Starting leader election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:47:09.340072 26966 raft_consensus.cc:729] T 998c6cee72cf40228c6dba064c0db2da P 65c36d164b0540f099d592f38307ee6c [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
W20260809 15:47:09.389235 23223 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.68:34689, user_credentials={real_user=slave}} blocked reactor thread for 44998.2us
I20260809 15:47:09.357685 23276 raft_consensus.cc:1226] T 3d880de657e544468c82157769622a26 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 3. Current term is 4. Ops: []
I20260809 15:47:09.349017 23282 raft_consensus.cc:1226] T 7baa2492df0340c4a5ac241959ce1ac3 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Rejecting Update request from peer c0f38c1908414566aa9115bb2cbdd85d for earlier term 4. Current term is 5. Ops: []
I20260809 15:47:09.359892 24463 raft_consensus.cc:3037] T 16ca412a0dae4feebe85dd01bf1a9173 P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Advancing to term 6
I20260809 15:47:09.351534 23273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c48a3049897452383257aa757834fb5" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:09.340888 28029 raft_consensus.cc:504] T 538a228ddfb54bc6a36528d5ff07597b P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:09.361567 28084 raft_consensus.cc:482] T cee02fff2c9c48a19f43eb358a89e7b0 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:09.349184 27957 raft_consensus.cc:2781] T cef3e7277d7d4760824a3436ed4a847d P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Leader election won for term 8
W20260809 15:47:09.415448 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 85650.8us
I20260809 15:47:09.349182 27492 leader_election.cc:288] T 0acc47ce21c3422698d1ef7277c3ed63 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 8 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
W20260809 15:47:09.417357 23353 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 80396.1us
I20260809 15:47:09.412536 28011 raft_consensus.cc:3032] T c0741ab44d99439abc3aa3bdd27ee160 P 65c36d164b0540f099d592f38307ee6c [term 6 LEADER]: Stepping down as leader of term 6
W20260809 15:47:09.407969 24410 connection.cc:232] RPC call timeout handler was delayed by -1.1497118659317493s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17070000000000002s for a response
I20260809 15:47:09.362519 27945 leader_election.cc:288] T 297c82ed906d4b48b184985eed3de601 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 9 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:09.352857 27886 leader_election.cc:288] T 2db057fd10e149a9895dff6ef8c9f038 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 7 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:09.410882 27493 raft_consensus.cc:3037] T a88421dba80d4d3388aa5eb62e23853b P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Advancing to term 6
I20260809 15:47:09.426805 23273 raft_consensus.cc:2396] T 0c48a3049897452383257aa757834fb5 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c for term 6 because replica has last-logged OpId of term: 4 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20260809 15:47:09.361239 27751 leader_election.cc:288] T 0da4d5283c8a46878bd39b85e5c71f66 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 6 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:09.360908 27749 raft_consensus.cc:482] T 05ad7d8fdca64e3da527280ca5be9280 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
I20260809 15:47:09.351912 28114 leader_election.cc:288] T 4bcac4b24e7a4448abdc934bba5a624e P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 4 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:09.429225 23277 raft_consensus.cc:1226] T d0721a4403a440f383318a3cea9d1442 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 6. Current term is 7. Ops: []
I20260809 15:47:09.429803 27749 raft_consensus.cc:504] T 05ad7d8fdca64e3da527280ca5be9280 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:47:09.430260 23468 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.68:34689, user_credentials={real_user=slave}} blocked reactor thread for 63946.9us
I20260809 15:47:09.431941 27749 leader_election.cc:288] T 05ad7d8fdca64e3da527280ca5be9280 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:47:09.432143 28086 leader_election.cc:288] T ae6d2c08a7a94f58ac8dd698e3fd456c P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 8 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:09.416553 27992 raft_consensus.cc:482] T c5c2e425a74b4e64ae70b5def750434c P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:47:09.433310 27953 leader_election.cc:288] T e71a05ba1d4d42bd95c3f85f9313399e P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 7 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:09.433933 27956 leader_election.cc:288] T 0a51c0baaa6e432997c9d3552af4f254 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 7 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:09.437278 28029 leader_election.cc:288] T 538a228ddfb54bc6a36528d5ff07597b P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 7 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:47:09.433318 27992 raft_consensus.cc:504] T c5c2e425a74b4e64ae70b5def750434c P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:09.439909 27892 leader_election.cc:288] T 0acc47ce21c3422698d1ef7277c3ed63 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 8 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:09.440645 23468 leader_election.cc:302] T 7660dfa2ef694927b8eda4100a2bcd9b P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c; no voters:
I20260809 15:47:09.442106 27322 raft_consensus.cc:2781] T 7660dfa2ef694927b8eda4100a2bcd9b P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Leader election won for term 6
I20260809 15:47:09.421942 23118 catalog_manager.cc:5495] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 61e69695258f48baa984c9d395e8ce3f reported cstate change: term changed from 3 to 4, leader changed from c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66) to 61e69695258f48baa984c9d395e8ce3f (127.20.202.68). New cstate: current_term: 4 leader_uuid: "61e69695258f48baa984c9d395e8ce3f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260809 15:47:09.347652 23472 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 195581us
I20260809 15:47:09.358525 24451 raft_consensus.cc:1226] T 4bcac4b24e7a4448abdc934bba5a624e P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 3. Current term is 4. Ops: []
I20260809 15:47:09.422437 23228 leader_election.cc:302] T 538a228ddfb54bc6a36528d5ff07597b P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:47:09.426036 23223 leader_election.cc:302] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:47:09.426174 26966 consensus_queue.cc:260] T 998c6cee72cf40228c6dba064c0db2da P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:09.357347 27988 consensus_queue.cc:1035] T 538a228ddfb54bc6a36528d5ff07597b P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.071s
I20260809 15:47:09.422526 28011 raft_consensus.cc:729] T c0741ab44d99439abc3aa3bdd27ee160 P 65c36d164b0540f099d592f38307ee6c [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:47:09.349326 28131 consensus_queue.cc:1046] T d3be1e356b254d22b995be32f1c12f6f P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.103s
I20260809 15:47:09.439929 27992 leader_election.cc:288] T c5c2e425a74b4e64ae70b5def750434c P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
W20260809 15:47:09.452682 23345 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.68:34689, user_credentials={real_user=slave}} blocked reactor thread for 129076us
I20260809 15:47:09.442556 27322 raft_consensus.cc:686] T 7660dfa2ef694927b8eda4100a2bcd9b P 65c36d164b0540f099d592f38307ee6c [term 6 LEADER]: Becoming Leader. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:47:09.443732 27957 raft_consensus.cc:686] T cef3e7277d7d4760824a3436ed4a847d P c0f38c1908414566aa9115bb2cbdd85d [term 8 LEADER]: Becoming Leader. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:47:09.453786 27956 raft_consensus.cc:2781] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Leader election won for term 7
I20260809 15:47:09.454308 27982 leader_election.cc:288] T d2d4f749fcfd4fb3bfb8115af3471598 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 7 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:09.454612 24453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a51c0baaa6e432997c9d3552af4f254" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f"
I20260809 15:47:09.455732 23353 leader_election.cc:302] T 2424fd9bd7f14e508fd106cbfda786a4 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
W20260809 15:47:09.454941 23193 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 4b30f28311964a0b9149a50495e83d8f, TS 65c36d164b0540f099d592f38307ee6c -> TS 61e69695258f48baa984c9d395e8ce3f move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:09.446975 27886 raft_consensus.cc:2726] T 907bacfa79d04a55a68c8dbe60056462 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260809 15:47:09.449116 23276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e17dcb5efe45b8b6a8c42da03b4819" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
W20260809 15:47:09.463353 23193 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 4b30f28311964a0b9149a50495e83d8f, TS 65c36d164b0540f099d592f38307ee6c -> TS 61e69695258f48baa984c9d395e8ce3f move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:09.474824 24452 raft_consensus.cc:2454] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 61e69695258f48baa984c9d395e8ce3f [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 7.
I20260809 15:47:09.467208 26966 raft_consensus.cc:3037] T 998c6cee72cf40228c6dba064c0db2da P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Advancing to term 8
I20260809 15:47:09.463474 27761 raft_consensus.cc:2781] T 538a228ddfb54bc6a36528d5ff07597b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260809 15:47:09.477942 23468 leader_election.cc:302] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c; no voters:
I20260809 15:47:09.479089 27322 consensus_queue.cc:237] T 7660dfa2ef694927b8eda4100a2bcd9b P 65c36d164b0540f099d592f38307ee6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:09.469161 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee02fff2c9c48a19f43eb358a89e7b0" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f"
I20260809 15:47:09.480214 24468 raft_consensus.cc:3037] T cee02fff2c9c48a19f43eb358a89e7b0 P 61e69695258f48baa984c9d395e8ce3f [term 7 FOLLOWER]: Advancing to term 8
W20260809 15:47:09.488565 23193 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet cef3e7277d7d4760824a3436ed4a847d, TS c0f38c1908414566aa9115bb2cbdd85d -> TS 61e69695258f48baa984c9d395e8ce3f move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:47:09.488883 23193 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet cef3e7277d7d4760824a3436ed4a847d, TS c0f38c1908414566aa9115bb2cbdd85d -> TS 61e69695258f48baa984c9d395e8ce3f move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:09.471340 27956 raft_consensus.cc:686] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 LEADER]: Becoming Leader. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Running, Role: LEADER
I20260809 15:47:09.477432 27761 raft_consensus.cc:482] T 538a228ddfb54bc6a36528d5ff07597b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Starting leader election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:47:09.477758 28059 leader_election.cc:288] T 7baa2492df0340c4a5ac241959ce1ac3 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:09.498687 23277 raft_consensus.cc:1226] T 16ca412a0dae4feebe85dd01bf1a9173 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 5. Current term is 6. Ops: []
I20260809 15:47:09.486570 27992 raft_consensus.cc:2781] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Leader election won for term 7
I20260809 15:47:09.460879 28084 raft_consensus.cc:504] T cee02fff2c9c48a19f43eb358a89e7b0 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:09.489225 27886 raft_consensus.cc:2726] T 2424fd9bd7f14e508fd106cbfda786a4 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260809 15:47:09.504427 23353 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 53251.5us
I20260809 15:47:09.497074 27704 leader_election.cc:288] T 213543cdbe814c89a17a963469922c1b P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 4 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:09.499301 27761 raft_consensus.cc:3037] T 538a228ddfb54bc6a36528d5ff07597b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Advancing to term 7
I20260809 15:47:09.501957 27893 leader_election.cc:288] T a3622f6e3e3641cea1960e3a31498bb7 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 6 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:09.501832 27363 leader_election.cc:288] T 4b30f28311964a0b9149a50495e83d8f P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 5 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:47:09.482811 27957 consensus_queue.cc:237] T cef3e7277d7d4760824a3436ed4a847d P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:09.507704 27953 raft_consensus.cc:482] T f9e7cadcd24d45fcbfa84209225e8f2d P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:09.502521 28061 leader_election.cc:288] T 0c29937a80724ddb9d07a27534a146f3 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 8 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:09.516682 23277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae6d2c08a7a94f58ac8dd698e3fd456c" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:47:09.504222 27977 consensus_queue.cc:1046] T 4bcac4b24e7a4448abdc934bba5a624e P 65c36d164b0540f099d592f38307ee6c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.013s
I20260809 15:47:09.517477 23273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9e7cadcd24d45fcbfa84209225e8f2d" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 8 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:09.520591 23274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ae14a74a26f434db08c652e097385c8" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
W20260809 15:47:09.521960 23193 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 05ad7d8fdca64e3da527280ca5be9280, TS 67b3aeadfa7a432abe8af4d292e04ab9 -> TS 61e69695258f48baa984c9d395e8ce3f move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:09.522008 27751 raft_consensus.cc:3032] T d3be1e356b254d22b995be32f1c12f6f P c0f38c1908414566aa9115bb2cbdd85d [term 8 LEADER]: Stepping down as leader of term 8
W20260809 15:47:09.527797 23193 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 05ad7d8fdca64e3da527280ca5be9280, TS 67b3aeadfa7a432abe8af4d292e04ab9 -> TS 61e69695258f48baa984c9d395e8ce3f move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:09.501523 28011 consensus_queue.cc:260] T c0741ab44d99439abc3aa3bdd27ee160 P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:09.524329 23353 leader_election.cc:302] T d0721a4403a440f383318a3cea9d1442 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:47:09.509433 27992 raft_consensus.cc:686] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 65c36d164b0540f099d592f38307ee6c [term 7 LEADER]: Becoming Leader. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:47:09.526840 24410 leader_election.cc:302] T 16ca412a0dae4feebe85dd01bf1a9173 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:47:09.537197 23472 leader_election.cc:302] T 953eea268e17491ca76f3a555860a512 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:09.542475 28084 leader_election.cc:288] T cee02fff2c9c48a19f43eb358a89e7b0 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
W20260809 15:47:09.540264 28142 log.cc:924] Time spent T a3622f6e3e3641cea1960e3a31498bb7 P 67b3aeadfa7a432abe8af4d292e04ab9: Append to log took a long time: real 0.053s user 0.000s sys 0.000s
I20260809 15:47:09.541731 24453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee02fff2c9c48a19f43eb358a89e7b0" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:47:09.543848 27977 raft_consensus.cc:2726] T 953eea268e17491ca76f3a555860a512 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260809 15:47:09.548545 27956 consensus_queue.cc:237] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:09.539934 23228 leader_election.cc:302] T c5c2e425a74b4e64ae70b5def750434c P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:47:09.524732 28039 raft_consensus.cc:482] T 953eea268e17491ca76f3a555860a512 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:09.552006 28059 raft_consensus.cc:2781] T c5c2e425a74b4e64ae70b5def750434c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260809 15:47:09.529034 27751 raft_consensus.cc:729] T d3be1e356b254d22b995be32f1c12f6f P c0f38c1908414566aa9115bb2cbdd85d [term 8 LEADER]: Becoming Follower/Learner. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:47:09.569389 28039 raft_consensus.cc:504] T 953eea268e17491ca76f3a555860a512 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:09.551942 28011 raft_consensus.cc:3037] T c0741ab44d99439abc3aa3bdd27ee160 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Advancing to term 7
I20260809 15:47:09.573819 23273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cef3e7277d7d4760824a3436ed4a847d" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:47:09.577593 23273 raft_consensus.cc:2379] T cef3e7277d7d4760824a3436ed4a847d P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c in current term 8: Already voted for candidate c0f38c1908414566aa9115bb2cbdd85d in this term.
W20260809 15:47:09.582159 23353 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 57910.1us
I20260809 15:47:09.571966 28061 raft_consensus.cc:2726] T d0721a4403a440f383318a3cea9d1442 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260809 15:47:09.572819 28059 raft_consensus.cc:482] T c5c2e425a74b4e64ae70b5def750434c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Starting leader election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:47:09.588829 28039 leader_election.cc:288] T 953eea268e17491ca76f3a555860a512 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:09.552248 27953 raft_consensus.cc:504] T f9e7cadcd24d45fcbfa84209225e8f2d P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:09.592953 27953 leader_election.cc:288] T f9e7cadcd24d45fcbfa84209225e8f2d P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:09.596998 24463 raft_consensus.cc:2454] T 16ca412a0dae4feebe85dd01bf1a9173 P 61e69695258f48baa984c9d395e8ce3f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 6.
I20260809 15:47:09.604049 24448 raft_consensus.cc:1226] T 16ca412a0dae4feebe85dd01bf1a9173 P 61e69695258f48baa984c9d395e8ce3f [term 6 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 5. Current term is 6. Ops: []
W20260809 15:47:09.617959 23193 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 4564dfc6d4094c1b9240b367ccecf3b1, TS 67b3aeadfa7a432abe8af4d292e04ab9 -> TS 61e69695258f48baa984c9d395e8ce3f move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:47:09.618306 23193 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 4564dfc6d4094c1b9240b367ccecf3b1, TS 67b3aeadfa7a432abe8af4d292e04ab9 -> TS 61e69695258f48baa984c9d395e8ce3f move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:09.593555 28059 raft_consensus.cc:3037] T c5c2e425a74b4e64ae70b5def750434c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Advancing to term 7
I20260809 15:47:09.621806 23223 leader_election.cc:302] T 16ca412a0dae4feebe85dd01bf1a9173 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:47:09.628813 27751 consensus_queue.cc:260] T d3be1e356b254d22b995be32f1c12f6f P c0f38c1908414566aa9115bb2cbdd85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.7, Last appended by leader: 7, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:09.620148 28114 raft_consensus.cc:2726] T 16ca412a0dae4feebe85dd01bf1a9173 P 61e69695258f48baa984c9d395e8ce3f [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260809 15:47:09.604545 27992 consensus_queue.cc:237] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:09.661473 27762 raft_consensus.cc:2781] T 16ca412a0dae4feebe85dd01bf1a9173 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Leader election won for term 6
I20260809 15:47:09.662017 27762 raft_consensus.cc:686] T 16ca412a0dae4feebe85dd01bf1a9173 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 LEADER]: Becoming Leader. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Running, Role: LEADER
I20260809 15:47:09.662873 27762 consensus_queue.cc:237] T 16ca412a0dae4feebe85dd01bf1a9173 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:09.666864 24468 raft_consensus.cc:2454] T cee02fff2c9c48a19f43eb358a89e7b0 P 61e69695258f48baa984c9d395e8ce3f [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 8.
I20260809 15:47:09.670145 23223 leader_election.cc:302] T cee02fff2c9c48a19f43eb358a89e7b0 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:47:09.784570 27493 raft_consensus.cc:504] T a88421dba80d4d3388aa5eb62e23853b P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:47:09.835896 24410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 337040us
I20260809 15:47:09.861723 28061 raft_consensus.cc:482] T 30c1740ef6874df0a332fbd6fd4b03ce P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
W20260809 15:47:09.865360 28147 log.cc:924] Time spent T a88421dba80d4d3388aa5eb62e23853b P 67b3aeadfa7a432abe8af4d292e04ab9: Append to log took a long time: real 0.351s user 0.002s sys 0.001s
I20260809 15:47:09.863130 28061 raft_consensus.cc:504] T 30c1740ef6874df0a332fbd6fd4b03ce P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:09.869720 28000 raft_consensus.cc:3032] T 4bcac4b24e7a4448abdc934bba5a624e P 65c36d164b0540f099d592f38307ee6c [term 3 LEADER]: Stepping down as leader of term 3
I20260809 15:47:09.870646 28000 raft_consensus.cc:729] T 4bcac4b24e7a4448abdc934bba5a624e P 65c36d164b0540f099d592f38307ee6c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:47:09.872994 27761 raft_consensus.cc:504] T 538a228ddfb54bc6a36528d5ff07597b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:09.873495 28000 consensus_queue.cc:260] T 4bcac4b24e7a4448abdc934bba5a624e P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
W20260809 15:47:09.898397 23472 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 324973us
W20260809 15:47:09.900986 23353 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 303050us
W20260809 15:47:09.901460 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 302114us
W20260809 15:47:09.898084 23223 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.68:34689, user_credentials={real_user=slave}} blocked reactor thread for 228026us
I20260809 15:47:09.899669 23273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0acc47ce21c3422698d1ef7277c3ed63" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:47:09.902788 23273 raft_consensus.cc:2379] T 0acc47ce21c3422698d1ef7277c3ed63 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c in current term 8: Already voted for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in this term.
I20260809 15:47:09.906082 24448 raft_consensus.cc:1226] T 3d880de657e544468c82157769622a26 P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 3. Current term is 4. Ops: []
I20260809 15:47:09.906279 24410 leader_election.cc:302] T e14d118b2eaa42aa8a7cd4025aba7128 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:47:09.903033 28000 raft_consensus.cc:3037] T 4bcac4b24e7a4448abdc934bba5a624e P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:47:09.903651 27892 raft_consensus.cc:2781] T cee02fff2c9c48a19f43eb358a89e7b0 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Leader election won for term 8
I20260809 15:47:09.899164 28061 leader_election.cc:288] T 30c1740ef6874df0a332fbd6fd4b03ce P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:09.902321 27751 raft_consensus.cc:3037] T d3be1e356b254d22b995be32f1c12f6f P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Advancing to term 9
I20260809 15:47:09.914139 27322 consensus_queue.cc:1046] T 16ca412a0dae4feebe85dd01bf1a9173 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260809 15:47:09.921293 27322 raft_consensus.cc:3032] T 16ca412a0dae4feebe85dd01bf1a9173 P 65c36d164b0540f099d592f38307ee6c [term 5 LEADER]: Stepping down as leader of term 5
I20260809 15:47:09.921458 23521 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "73f6f44c259547f1b05a3a434616fa54"
dest_uuid: "65c36d164b0540f099d592f38307ee6c"
mode: GRACEFUL
new_leader_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
from {username='slave'} at 127.0.0.1:48888
I20260809 15:47:09.922191 23521 raft_consensus.cc:595] T 73f6f44c259547f1b05a3a434616fa54 P 65c36d164b0540f099d592f38307ee6c [term 7 LEADER]: Received request to transfer leadership to TS c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:09.922626 27761 leader_election.cc:288] T 538a228ddfb54bc6a36528d5ff07597b P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 7 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:09.923898 24410 leader_election.cc:302] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:47:09.912443 28114 raft_consensus.cc:2726] T e14d118b2eaa42aa8a7cd4025aba7128 P 61e69695258f48baa984c9d395e8ce3f [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260809 15:47:09.923290 27322 raft_consensus.cc:729] T 16ca412a0dae4feebe85dd01bf1a9173 P 65c36d164b0540f099d592f38307ee6c [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:47:09.929435 23276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05ad7d8fdca64e3da527280ca5be9280" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:09.932055 23521 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "953eea268e17491ca76f3a555860a512"
dest_uuid: "65c36d164b0540f099d592f38307ee6c"
mode: GRACEFUL
new_leader_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
from {username='slave'} at 127.0.0.1:48888
I20260809 15:47:09.935523 23228 leader_election.cc:302] T cef3e7277d7d4760824a3436ed4a847d P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:47:09.937168 23521 raft_consensus.cc:595] T 953eea268e17491ca76f3a555860a512 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Received request to transfer leadership to TS c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:09.937569 23521 raft_consensus.cc:603] T 953eea268e17491ca76f3a555860a512 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260809 15:47:09.935467 27322 consensus_queue.cc:260] T 16ca412a0dae4feebe85dd01bf1a9173 P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:09.942332 27322 raft_consensus.cc:3037] T 16ca412a0dae4feebe85dd01bf1a9173 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Advancing to term 6
I20260809 15:47:09.932147 28114 raft_consensus.cc:2726] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260809 15:47:09.937484 28059 raft_consensus.cc:504] T c5c2e425a74b4e64ae70b5def750434c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:09.944963 28059 leader_election.cc:288] T c5c2e425a74b4e64ae70b5def750434c P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 7 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:09.916870 28011 consensus_queue.cc:1046] T 3d880de657e544468c82157769622a26 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.021s
I20260809 15:47:09.959941 24448 raft_consensus.cc:1261] T 7660dfa2ef694927b8eda4100a2bcd9b P 61e69695258f48baa984c9d395e8ce3f [term 6 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260809 15:47:09.965396 23518 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "998c6cee72cf40228c6dba064c0db2da"
dest_uuid: "65c36d164b0540f099d592f38307ee6c"
mode: GRACEFUL
new_leader_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
from {username='slave'} at 127.0.0.1:48888
I20260809 15:47:09.965982 23518 raft_consensus.cc:595] T 998c6cee72cf40228c6dba064c0db2da P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Received request to transfer leadership to TS c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:09.966357 23518 raft_consensus.cc:603] T 998c6cee72cf40228c6dba064c0db2da P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260809 15:47:09.967751 27493 leader_election.cc:288] T a88421dba80d4d3388aa5eb62e23853b P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 6 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:09.939254 28164 raft_consensus.cc:2741] T cef3e7277d7d4760824a3436ed4a847d P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Leader election decision vote started in defunct term 7: won
I20260809 15:47:09.973510 23521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad8a3b25d7b49e5a2142d5709edd0c7" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:10.016778 27492 raft_consensus.cc:3032] T 3d880de657e544468c82157769622a26 P 65c36d164b0540f099d592f38307ee6c [term 3 LEADER]: Stepping down as leader of term 3
I20260809 15:47:10.017194 27492 raft_consensus.cc:729] T 3d880de657e544468c82157769622a26 P 65c36d164b0540f099d592f38307ee6c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:47:10.017992 27492 consensus_queue.cc:260] T 3d880de657e544468c82157769622a26 P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:10.019487 23518 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "cef3e7277d7d4760824a3436ed4a847d"
dest_uuid: "65c36d164b0540f099d592f38307ee6c"
mode: GRACEFUL
new_leader_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
from {username='slave'} at 127.0.0.1:48888
I20260809 15:47:10.020030 23518 raft_consensus.cc:595] T cef3e7277d7d4760824a3436ed4a847d P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Received request to transfer leadership to TS 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:47:10.020376 23518 raft_consensus.cc:603] T cef3e7277d7d4760824a3436ed4a847d P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260809 15:47:10.020977 28164 consensus_queue.cc:1035] T 953eea268e17491ca76f3a555860a512 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20260809 15:47:10.021396 23228 leader_election.cc:302] T c0741ab44d99439abc3aa3bdd27ee160 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c
I20260809 15:47:10.023399 28086 raft_consensus.cc:2726] T c0741ab44d99439abc3aa3bdd27ee160 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260809 15:47:10.026281 23518 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d2d4f749fcfd4fb3bfb8115af3471598"
dest_uuid: "65c36d164b0540f099d592f38307ee6c"
mode: GRACEFUL
new_leader_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
from {username='slave'} at 127.0.0.1:48888
I20260809 15:47:10.006705 26966 consensus_queue.cc:1035] T 7660dfa2ef694927b8eda4100a2bcd9b P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.021s
I20260809 15:47:10.027896 23518 raft_consensus.cc:595] T d2d4f749fcfd4fb3bfb8115af3471598 P 65c36d164b0540f099d592f38307ee6c [term 6 LEADER]: Received request to transfer leadership to TS 67b3aeadfa7a432abe8af4d292e04ab9
W20260809 15:47:10.027273 28162 log.cc:924] Time spent T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9: Append to log took a long time: real 0.103s user 0.001s sys 0.000s
I20260809 15:47:09.971164 28061 consensus_queue.cc:1046] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.064s
I20260809 15:47:10.036110 28061 raft_consensus.cc:3032] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P c0f38c1908414566aa9115bb2cbdd85d [term 6 LEADER]: Stepping down as leader of term 6
I20260809 15:47:10.036463 28061 raft_consensus.cc:729] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P c0f38c1908414566aa9115bb2cbdd85d [term 6 LEADER]: Becoming Follower/Learner. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:47:10.037369 28061 consensus_queue.cc:260] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P c0f38c1908414566aa9115bb2cbdd85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:10.038861 28061 raft_consensus.cc:3037] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Advancing to term 7
I20260809 15:47:10.040060 27492 raft_consensus.cc:3037] T 3d880de657e544468c82157769622a26 P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:47:10.040630 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73f6f44c259547f1b05a3a434616fa54" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:10.041266 23519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3e94b7e50754a81b3d72a07476cb7b9" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:10.043640 23529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3924b648461e463b9d7925e5671d68ae" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:10.044327 23529 raft_consensus.cc:2454] T 3924b648461e463b9d7925e5671d68ae P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 8.
I20260809 15:47:10.045123 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7715d71dbac447b9351fc5eb36c5e80" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:10.045249 23522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bcac4b24e7a4448abdc934bba5a624e" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:10.046454 23519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05ad7d8fdca64e3da527280ca5be9280" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:47:10.047027 23519 raft_consensus.cc:3037] T 05ad7d8fdca64e3da527280ca5be9280 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Advancing to term 7
I20260809 15:47:10.019405 23521 raft_consensus.cc:2454] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 4.
I20260809 15:47:10.051110 23276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "538a228ddfb54bc6a36528d5ff07597b" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:47:10.052578 21289 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260809 15:47:10.066226 23518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73f6f44c259547f1b05a3a434616fa54" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:47:10.066442 23529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba56725e42f74a1baa4c04d9660727f9" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:47:10.067493 23518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c29937a80724ddb9d07a27534a146f3" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 8 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:47:10.067646 23529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "213543cdbe814c89a17a963469922c1b" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:47:10.068568 23518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30c1740ef6874df0a332fbd6fd4b03ce" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:10.069173 23518 raft_consensus.cc:2454] T 30c1740ef6874df0a332fbd6fd4b03ce P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 7.
I20260809 15:47:10.069511 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2db057fd10e149a9895dff6ef8c9f038" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:47:10.069959 23518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2d4f749fcfd4fb3bfb8115af3471598" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:47:10.070729 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "953eea268e17491ca76f3a555860a512" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:10.071975 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3e94b7e50754a81b3d72a07476cb7b9" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:47:10.046396 23517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998c6cee72cf40228c6dba064c0db2da" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:47:10.068850 23529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3622f6e3e3641cea1960e3a31498bb7" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:47:10.071071 23518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "297c82ed906d4b48b184985eed3de601" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:47:10.054426 21289 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 15:47:10.082429 23519 raft_consensus.cc:2454] T 05ad7d8fdca64e3da527280ca5be9280 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 7.
I20260809 15:47:10.084115 23522 raft_consensus.cc:2396] T 4bcac4b24e7a4448abdc934bba5a624e P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d for term 4 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260809 15:47:10.083654 23273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5c2e425a74b4e64ae70b5def750434c" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:10.083552 23526 raft_consensus.cc:2454] T 953eea268e17491ca76f3a555860a512 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 7.
I20260809 15:47:10.089253 23519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9e7cadcd24d45fcbfa84209225e8f2d" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 8 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:10.090489 24463 raft_consensus.cc:1261] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 61e69695258f48baa984c9d395e8ce3f [term 7 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260809 15:47:10.092849 23519 raft_consensus.cc:2454] T f9e7cadcd24d45fcbfa84209225e8f2d P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 7.
I20260809 15:47:10.094666 23276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b30f28311964a0b9149a50495e83d8f" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:47:10.096405 23529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a88421dba80d4d3388aa5eb62e23853b" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:47:10.106325 23276 raft_consensus.cc:3037] T 4b30f28311964a0b9149a50495e83d8f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Advancing to term 5
I20260809 15:47:10.108587 23521 raft_consensus.cc:1226] T 538a228ddfb54bc6a36528d5ff07597b P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Rejecting Update request from peer c0f38c1908414566aa9115bb2cbdd85d for earlier term 6. Current term is 7. Ops: [6.4-6.4]
W20260809 15:47:10.126551 23223 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.68:34689, user_credentials={real_user=slave}} blocked reactor thread for 215848us
I20260809 15:47:10.128430 23273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee02fff2c9c48a19f43eb358a89e7b0" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:10.127810 23517 raft_consensus.cc:2454] T 998c6cee72cf40228c6dba064c0db2da P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 8.
W20260809 15:47:10.131130 23472 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 222671us
I20260809 15:47:10.106941 27892 raft_consensus.cc:686] T cee02fff2c9c48a19f43eb358a89e7b0 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 LEADER]: Becoming Leader. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Running, Role: LEADER
I20260809 15:47:10.128293 23520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0da4d5283c8a46878bd39b85e5c71f66" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
W20260809 15:47:10.135095 28153 log.cc:924] Time spent T 763e58d6ba644eea830a6c1664d39c9b P 67b3aeadfa7a432abe8af4d292e04ab9: Append to log took a long time: real 0.179s user 0.001s sys 0.001s
W20260809 15:47:10.142068 24406 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 56690.6us
W20260809 15:47:10.143119 24410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 219351us
I20260809 15:47:10.148695 27977 consensus_queue.cc:1035] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260809 15:47:10.151320 23470 leader_election.cc:302] T 0c48a3049897452383257aa757834fb5 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 61e69695258f48baa984c9d395e8ce3f, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:47:10.152885 27493 consensus_queue.cc:1046] T 7baa2492df0340c4a5ac241959ce1ac3 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260809 15:47:10.153970 27493 raft_consensus.cc:3032] T 7baa2492df0340c4a5ac241959ce1ac3 P c0f38c1908414566aa9115bb2cbdd85d [term 4 LEADER]: Stepping down as leader of term 4
I20260809 15:47:10.154271 27493 raft_consensus.cc:729] T 7baa2492df0340c4a5ac241959ce1ac3 P c0f38c1908414566aa9115bb2cbdd85d [term 4 LEADER]: Becoming Follower/Learner. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:47:10.155836 23280 raft_consensus.cc:1226] T 294d4bcc1e7a477cb56455581277a60f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 5. Current term is 6. Ops: []
I20260809 15:47:10.170094 27992 raft_consensus.cc:2726] T 0c48a3049897452383257aa757834fb5 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260809 15:47:10.170398 23277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05ad7d8fdca64e3da527280ca5be9280" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:47:10.170643 27322 consensus_queue.cc:1046] T d0721a4403a440f383318a3cea9d1442 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260809 15:47:10.171803 23286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73f6f44c259547f1b05a3a434616fa54" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:47:10.172870 23286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0da4d5283c8a46878bd39b85e5c71f66" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:47:10.173825 23286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba56725e42f74a1baa4c04d9660727f9" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:47:10.174726 23286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "297c82ed906d4b48b184985eed3de601" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:47:10.176403 23274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2db057fd10e149a9895dff6ef8c9f038" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:47:10.177614 23274 raft_consensus.cc:1226] T c5c2e425a74b4e64ae70b5def750434c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Rejecting Update request from peer c0f38c1908414566aa9115bb2cbdd85d for earlier term 6. Current term is 7. Ops: []
I20260809 15:47:10.181797 23285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7715d71dbac447b9351fc5eb36c5e80" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:10.183632 23282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998c6cee72cf40228c6dba064c0db2da" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:47:10.184441 23282 raft_consensus.cc:2379] T 998c6cee72cf40228c6dba064c0db2da P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d in current term 8: Already voted for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in this term.
I20260809 15:47:10.185292 23283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "563daf42d7784db195ea99c3cb21b370" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:47:10.185913 23283 raft_consensus.cc:3037] T 563daf42d7784db195ea99c3cb21b370 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Advancing to term 6
W20260809 15:47:10.170287 23353 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 65650.1us
I20260809 15:47:10.190038 23353 leader_election.cc:302] T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters: 61e69695258f48baa984c9d395e8ce3f
I20260809 15:47:10.191449 27988 raft_consensus.cc:2781] T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260809 15:47:10.191778 27988 raft_consensus.cc:482] T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Starting leader election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
I20260809 15:47:10.192116 27988 raft_consensus.cc:3037] T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Advancing to term 9
I20260809 15:47:10.188812 23520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c29937a80724ddb9d07a27534a146f3" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 8 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:10.188741 23522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a88421dba80d4d3388aa5eb62e23853b" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:10.205919 23353 leader_election.cc:302] T 30c1740ef6874df0a332fbd6fd4b03ce P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:47:10.206679 23276 raft_consensus.cc:2454] T 4b30f28311964a0b9149a50495e83d8f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 5.
I20260809 15:47:10.209100 23285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "213543cdbe814c89a17a963469922c1b" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:47:10.211511 23277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e71a05ba1d4d42bd95c3f85f9313399e" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:47:10.207778 23280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2d4f749fcfd4fb3bfb8115af3471598" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:47:10.208561 23282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3622f6e3e3641cea1960e3a31498bb7" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:47:10.155053 27493 consensus_queue.cc:260] T 7baa2492df0340c4a5ac241959ce1ac3 P c0f38c1908414566aa9115bb2cbdd85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:10.214980 23285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c29937a80724ddb9d07a27534a146f3" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 8 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:47:10.213737 24406 leader_election.cc:302] T 907bacfa79d04a55a68c8dbe60056462 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:47:10.216495 23520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7715d71dbac447b9351fc5eb36c5e80" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:10.218870 23276 raft_consensus.cc:1226] T 538a228ddfb54bc6a36528d5ff07597b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Rejecting Update request from peer c0f38c1908414566aa9115bb2cbdd85d for earlier term 6. Current term is 7. Ops: []
W20260809 15:47:10.219427 24410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 51923.2us
I20260809 15:47:10.220268 24410 leader_election.cc:302] T cee02fff2c9c48a19f43eb358a89e7b0 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c; no voters: 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:47:10.228047 27493 raft_consensus.cc:3037] T 7baa2492df0340c4a5ac241959ce1ac3 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Advancing to term 5
I20260809 15:47:10.229972 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a51c0baaa6e432997c9d3552af4f254" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:47:10.215457 28061 raft_consensus.cc:2781] T 30c1740ef6874df0a332fbd6fd4b03ce P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260809 15:47:10.231596 28061 raft_consensus.cc:482] T 30c1740ef6874df0a332fbd6fd4b03ce P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Starting leader election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
I20260809 15:47:10.231940 28061 raft_consensus.cc:3037] T 30c1740ef6874df0a332fbd6fd4b03ce P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Advancing to term 8
I20260809 15:47:10.235857 23353 leader_election.cc:302] T 05ad7d8fdca64e3da527280ca5be9280 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
W20260809 15:47:10.215323 23472 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 45440us
I20260809 15:47:10.240784 24406 leader_election.cc:302] T d3be1e356b254d22b995be32f1c12f6f P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:47:10.243038 23520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e17dcb5efe45b8b6a8c42da03b4819" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:10.245805 23522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2424fd9bd7f14e508fd106cbfda786a4" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:10.246934 23353 leader_election.cc:302] T 953eea268e17491ca76f3a555860a512 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
W20260809 15:47:10.249241 23228 connection.cc:232] RPC call timeout handler was delayed by -1.7992380310315639s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1835s for a response
I20260809 15:47:10.251065 23276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "953eea268e17491ca76f3a555860a512" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:10.251449 23517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1a134ba33ea4076bbaac0ae460c17ba" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
W20260809 15:47:10.249747 23228 connection.cc:232] RPC call timeout handler was delayed by -1.7929380310233682s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18420000000000003s for a response
W20260809 15:47:10.251992 23228 connection.cc:232] RPC call timeout handler was delayed by -1.7911380310542881s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1844s for a response
W20260809 15:47:10.252310 23228 connection.cc:232] RPC call timeout handler was delayed by -1.7326380310114473s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19090000000000001s for a response
W20260809 15:47:10.252583 23228 connection.cc:232] RPC call timeout handler was delayed by -1.7164380310568959s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1927s for a response
W20260809 15:47:10.252825 23228 connection.cc:232] RPC call timeout handler was delayed by -1.7038380310405046s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19410000000000002s for a response
I20260809 15:47:10.254380 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00a1881267a14ba8b78c81040f17a014" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:10.258186 23277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9e7cadcd24d45fcbfa84209225e8f2d" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 8 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:10.260607 23353 leader_election.cc:302] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:47:10.242563 28171 raft_consensus.cc:2781] T 907bacfa79d04a55a68c8dbe60056462 P 61e69695258f48baa984c9d395e8ce3f [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260809 15:47:10.263546 23276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a88421dba80d4d3388aa5eb62e23853b" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:47:10.250272 28131 raft_consensus.cc:2781] T 05ad7d8fdca64e3da527280ca5be9280 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Leader election won for term 7
I20260809 15:47:10.250427 28114 raft_consensus.cc:2741] T cee02fff2c9c48a19f43eb358a89e7b0 P 61e69695258f48baa984c9d395e8ce3f [term 8 FOLLOWER]: Leader election decision vote started in defunct term 7: won
I20260809 15:47:10.257733 27957 raft_consensus.cc:2781] T 953eea268e17491ca76f3a555860a512 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260809 15:47:10.272372 23285 raft_consensus.cc:1261] T cef3e7277d7d4760824a3436ed4a847d P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
I20260809 15:47:10.273473 23277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30c1740ef6874df0a332fbd6fd4b03ce" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:10.273047 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "294d4bcc1e7a477cb56455581277a60f" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:47:10.272526 23351 leader_election.cc:302] T 73f6f44c259547f1b05a3a434616fa54 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:47:10.264087 28171 raft_consensus.cc:482] T 907bacfa79d04a55a68c8dbe60056462 P 61e69695258f48baa984c9d395e8ce3f [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260809 15:47:10.279772 23470 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 126649us
W20260809 15:47:10.283308 24410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 63168.9us
I20260809 15:47:10.284403 26966 raft_consensus.cc:979] T d2d4f749fcfd4fb3bfb8115af3471598 P 65c36d164b0540f099d592f38307ee6c: : Instructing follower 67b3aeadfa7a432abe8af4d292e04ab9 to start an election
I20260809 15:47:10.284895 26966 raft_consensus.cc:1067] T d2d4f749fcfd4fb3bfb8115af3471598 P 65c36d164b0540f099d592f38307ee6c [term 6 LEADER]: Signalling peer 67b3aeadfa7a432abe8af4d292e04ab9 to start an election
I20260809 15:47:10.285579 28114 raft_consensus.cc:2781] T d3be1e356b254d22b995be32f1c12f6f P 61e69695258f48baa984c9d395e8ce3f [term 9 FOLLOWER]: Leader election won for term 9
W20260809 15:47:10.279592 23223 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.68:34689, user_credentials={real_user=slave}} blocked reactor thread for 111698us
I20260809 15:47:10.275770 27957 raft_consensus.cc:482] T 953eea268e17491ca76f3a555860a512 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:47:10.289990 27957 raft_consensus.cc:3037] T 953eea268e17491ca76f3a555860a512 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Advancing to term 8
I20260809 15:47:10.284839 27953 raft_consensus.cc:2726] T 73f6f44c259547f1b05a3a434616fa54 P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260809 15:47:10.288236 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad8a3b25d7b49e5a2142d5709edd0c7" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:47:10.288343 23517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d880de657e544468c82157769622a26" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:47:10.327960 21289 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 15:47:10.254048 27892 consensus_queue.cc:237] T cee02fff2c9c48a19f43eb358a89e7b0 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
W20260809 15:47:10.334100 23472 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 45242.4us
I20260809 15:47:10.349412 23521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee02fff2c9c48a19f43eb358a89e7b0" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:47:10.349992 23521 raft_consensus.cc:3037] T cee02fff2c9c48a19f43eb358a89e7b0 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Advancing to term 8
I20260809 15:47:10.350526 23518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7baa2492df0340c4a5ac241959ce1ac3" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:47:10.275635 28131 raft_consensus.cc:686] T 05ad7d8fdca64e3da527280ca5be9280 P c0f38c1908414566aa9115bb2cbdd85d [term 7 LEADER]: Becoming Leader. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:47:10.352885 23518 raft_consensus.cc:3037] T 7baa2492df0340c4a5ac241959ce1ac3 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Advancing to term 5
I20260809 15:47:10.276685 28168 raft_consensus.cc:482] T 0c48a3049897452383257aa757834fb5 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:10.353077 23519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0acc47ce21c3422698d1ef7277c3ed63" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
W20260809 15:47:10.339388 28193 log.cc:924] Time spent T 00000000000000000000000000000000 P a86a2513b8024a869d80303dcbf44163: Append to log took a long time: real 0.058s user 0.000s sys 0.002s
I20260809 15:47:10.354375 23522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16ca412a0dae4feebe85dd01bf1a9173" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:47:10.356076 23519 raft_consensus.cc:2379] T 0acc47ce21c3422698d1ef7277c3ed63 P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 in current term 8: Already voted for candidate 65c36d164b0540f099d592f38307ee6c in this term.
W20260809 15:47:10.352169 23353 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 95993.6us
I20260809 15:47:10.286535 28171 raft_consensus.cc:3037] T 907bacfa79d04a55a68c8dbe60056462 P 61e69695258f48baa984c9d395e8ce3f [term 7 FOLLOWER]: Advancing to term 8
I20260809 15:47:10.366766 27953 raft_consensus.cc:2781] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260809 15:47:10.367208 27953 raft_consensus.cc:482] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Starting leader election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
I20260809 15:47:10.367585 27953 raft_consensus.cc:3037] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Advancing to term 5
I20260809 15:47:10.375536 23529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae6d2c08a7a94f58ac8dd698e3fd456c" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:47:10.376237 23529 raft_consensus.cc:2379] T ae6d2c08a7a94f58ac8dd698e3fd456c P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 in current term 8: Already voted for candidate 65c36d164b0540f099d592f38307ee6c in this term.
I20260809 15:47:10.377182 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "538a228ddfb54bc6a36528d5ff07597b" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:47:10.377897 23527 raft_consensus.cc:2379] T 538a228ddfb54bc6a36528d5ff07597b P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 in current term 7: Already voted for candidate 65c36d164b0540f099d592f38307ee6c in this term.
I20260809 15:47:10.346915 28061 raft_consensus.cc:504] T 30c1740ef6874df0a332fbd6fd4b03ce P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:10.379166 23353 leader_election.cc:302] T f9e7cadcd24d45fcbfa84209225e8f2d P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:47:10.394497 23519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5c2e425a74b4e64ae70b5def750434c" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:47:10.393967 23283 raft_consensus.cc:2454] T 563daf42d7784db195ea99c3cb21b370 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 6.
I20260809 15:47:10.396042 27957 raft_consensus.cc:504] T 953eea268e17491ca76f3a555860a512 P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:10.405854 23353 leader_election.cc:302] T 998c6cee72cf40228c6dba064c0db2da P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:47:10.396420 27751 raft_consensus.cc:2781] T f9e7cadcd24d45fcbfa84209225e8f2d P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260809 15:47:10.414265 24410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 130065us
I20260809 15:47:10.417580 27751 raft_consensus.cc:482] T f9e7cadcd24d45fcbfa84209225e8f2d P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:47:10.419616 27751 raft_consensus.cc:3037] T f9e7cadcd24d45fcbfa84209225e8f2d P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Advancing to term 8
W20260809 15:47:10.424031 28189 log.cc:924] Time spent T 6ae14a74a26f434db08c652e097385c8 P 67b3aeadfa7a432abe8af4d292e04ab9: Append to log took a long time: real 0.068s user 0.000s sys 0.000s
I20260809 15:47:10.405591 28011 raft_consensus.cc:3032] T d0721a4403a440f383318a3cea9d1442 P 65c36d164b0540f099d592f38307ee6c [term 6 LEADER]: Stepping down as leader of term 6
I20260809 15:47:10.433091 23518 raft_consensus.cc:2454] T 7baa2492df0340c4a5ac241959ce1ac3 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 5.
I20260809 15:47:10.416079 28114 raft_consensus.cc:686] T d3be1e356b254d22b995be32f1c12f6f P 61e69695258f48baa984c9d395e8ce3f [term 9 LEADER]: Becoming Leader. State: Replica: 61e69695258f48baa984c9d395e8ce3f, State: Running, Role: LEADER
I20260809 15:47:10.444793 23521 raft_consensus.cc:2454] T cee02fff2c9c48a19f43eb358a89e7b0 P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 8.
I20260809 15:47:10.435145 28011 raft_consensus.cc:729] T d0721a4403a440f383318a3cea9d1442 P 65c36d164b0540f099d592f38307ee6c [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:47:10.432637 27957 leader_election.cc:288] T 953eea268e17491ca76f3a555860a512 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 8 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:10.397593 28168 raft_consensus.cc:504] T 0c48a3049897452383257aa757834fb5 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:10.432701 27493 consensus_queue.cc:1046] T 538a228ddfb54bc6a36528d5ff07597b P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.035s
W20260809 15:47:10.470211 24406 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 229498us
I20260809 15:47:10.470345 24410 leader_election.cc:302] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c; no voters: c0f38c1908414566aa9115bb2cbdd85d
W20260809 15:47:10.472045 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 199567us
W20260809 15:47:10.476516 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 223298us
I20260809 15:47:10.476583 23351 leader_election.cc:302] T ba56725e42f74a1baa4c04d9660727f9 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:47:10.485108 23273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "953eea268e17491ca76f3a555860a512" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
W20260809 15:47:10.494998 23470 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 59186.3us
I20260809 15:47:10.498530 27953 raft_consensus.cc:504] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:10.471969 28210 raft_consensus.cc:2741] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 61e69695258f48baa984c9d395e8ce3f [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
I20260809 15:47:10.485411 27493 raft_consensus.cc:3032] T 538a228ddfb54bc6a36528d5ff07597b P c0f38c1908414566aa9115bb2cbdd85d [term 6 LEADER]: Stepping down as leader of term 6
I20260809 15:47:10.499716 27493 raft_consensus.cc:729] T 538a228ddfb54bc6a36528d5ff07597b P c0f38c1908414566aa9115bb2cbdd85d [term 6 LEADER]: Becoming Follower/Learner. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:47:10.500499 27493 consensus_queue.cc:260] T 538a228ddfb54bc6a36528d5ff07597b P c0f38c1908414566aa9115bb2cbdd85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:10.501407 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "563daf42d7784db195ea99c3cb21b370" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f"
W20260809 15:47:10.501999 23472 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 50244us
I20260809 15:47:10.503238 23522 raft_consensus.cc:2454] T 16ca412a0dae4feebe85dd01bf1a9173 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 6.
I20260809 15:47:10.504268 27493 raft_consensus.cc:3037] T 538a228ddfb54bc6a36528d5ff07597b P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Advancing to term 7
I20260809 15:47:10.504901 28208 raft_consensus.cc:2726] T ba56725e42f74a1baa4c04d9660727f9 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260809 15:47:10.505479 28208 raft_consensus.cc:482] T 8e4ffcf83c234ae7910ddef3ec0b4bda P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
I20260809 15:47:10.505945 28208 raft_consensus.cc:504] T 8e4ffcf83c234ae7910ddef3ec0b4bda P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:10.506326 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3e94b7e50754a81b3d72a07476cb7b9" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f"
I20260809 15:47:10.505831 24455 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e71a05ba1d4d42bd95c3f85f9313399e" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f"
I20260809 15:47:10.507680 28061 leader_election.cc:288] T 30c1740ef6874df0a332fbd6fd4b03ce P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 8 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:10.507583 28011 consensus_queue.cc:260] T d0721a4403a440f383318a3cea9d1442 P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:10.452602 28131 consensus_queue.cc:237] T 05ad7d8fdca64e3da527280ca5be9280 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:10.477476 27957 raft_consensus.cc:2781] T 998c6cee72cf40228c6dba064c0db2da P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Leader election won for term 8
I20260809 15:47:10.508338 27792 tablet_bootstrap.cc:492] T 4b30f28311964a0b9149a50495e83d8f P 61e69695258f48baa984c9d395e8ce3f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
W20260809 15:47:10.518225 24410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 56690.7us
I20260809 15:47:10.500541 27953 leader_election.cc:288] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 election: Requested vote from peers 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:10.530805 27988 raft_consensus.cc:504] T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:10.531776 23393 raft_consensus.cc:1226] T 297c82ed906d4b48b184985eed3de601 P c0f38c1908414566aa9115bb2cbdd85d [term 9 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 8. Current term is 9. Ops: [8.5-8.5]
I20260809 15:47:10.532754 27988 leader_election.cc:288] T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 9 election: Requested vote from peers 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:10.534723 27751 raft_consensus.cc:504] T f9e7cadcd24d45fcbfa84209225e8f2d P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:10.540649 23393 raft_consensus.cc:1226] T 73f6f44c259547f1b05a3a434616fa54 P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 7. Current term is 8. Ops: []
I20260809 15:47:10.541076 24410 leader_election.cc:302] T 0c48a3049897452383257aa757834fb5 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c; no voters:
I20260809 15:47:10.527940 27957 raft_consensus.cc:686] T 998c6cee72cf40228c6dba064c0db2da P c0f38c1908414566aa9115bb2cbdd85d [term 8 LEADER]: Becoming Leader. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:47:10.541551 23517 raft_consensus.cc:2454] T 3d880de657e544468c82157769622a26 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 4.
I20260809 15:47:10.546499 23395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e17dcb5efe45b8b6a8c42da03b4819" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
W20260809 15:47:10.547175 28201 log.cc:924] Time spent T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 61e69695258f48baa984c9d395e8ce3f: Append to log took a long time: real 0.061s user 0.000s sys 0.000s
I20260809 15:47:10.523367 28061 raft_consensus.cc:482] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
I20260809 15:47:10.536583 27751 leader_election.cc:288] T f9e7cadcd24d45fcbfa84209225e8f2d P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 8 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:10.495705 28114 consensus_queue.cc:237] T d3be1e356b254d22b995be32f1c12f6f P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:10.538771 28011 raft_consensus.cc:3037] T d0721a4403a440f383318a3cea9d1442 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Advancing to term 7
W20260809 15:47:10.563381 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 86873us
I20260809 15:47:10.565706 23351 leader_election.cc:302] T 2db057fd10e149a9895dff6ef8c9f038 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:47:10.565260 23393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b30f28311964a0b9149a50495e83d8f" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:47:10.569087 23351 leader_election.cc:302] T 297c82ed906d4b48b184985eed3de601 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:47:10.570827 23351 leader_election.cc:302] T f7715d71dbac447b9351fc5eb36c5e80 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
W20260809 15:47:10.571869 23228 connection.cc:232] RPC call timeout handler was delayed by -1.6678380309604108s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19810000000000003s for a response
W20260809 15:47:10.572314 23228 connection.cc:232] RPC call timeout handler was delayed by -1.6543380308430642s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1996s for a response
I20260809 15:47:10.573616 23399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a823d35f4f348d1b52ba9e6d80b00d2" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:10.575321 23393 raft_consensus.cc:2362] T 4b30f28311964a0b9149a50495e83d8f P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 4. Re-sending same reply.
I20260809 15:47:10.576370 23472 leader_election.cc:302] T 30c1740ef6874df0a332fbd6fd4b03ce P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters: 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:47:10.548367 27953 raft_consensus.cc:482] T 472816994d814cb198b69e5248697440 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
I20260809 15:47:10.574105 28220 consensus_queue.cc:1035] T 0c29937a80724ddb9d07a27534a146f3 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 15:47:10.574266 28168 leader_election.cc:288] T 0c48a3049897452383257aa757834fb5 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
W20260809 15:47:10.583583 23228 connection.cc:232] RPC call timeout handler was delayed by -1.637238031020388s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2015s for a response
W20260809 15:47:10.584619 23228 connection.cc:232] RPC call timeout handler was delayed by -1.5832380310166627s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20750000000000002s for a response
W20260809 15:47:10.585035 23228 connection.cc:232] RPC call timeout handler was delayed by -1.5634380308911204s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2097s for a response
W20260809 15:47:10.585466 23228 connection.cc:232] RPC call timeout handler was delayed by -1.4653380310628563s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22060000000000002s for a response
W20260809 15:47:10.585777 23228 connection.cc:232] RPC call timeout handler was delayed by -1.4626380309928209s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2209s for a response
W20260809 15:47:10.586055 23228 connection.cc:232] RPC call timeout handler was delayed by -1.1817827890627086s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15700000000000003s for a response
W20260809 15:47:10.586198 23470 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 62381.6us
I20260809 15:47:10.564541 27977 consensus_queue.cc:1046] T 294d4bcc1e7a477cb56455581277a60f P 65c36d164b0540f099d592f38307ee6c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.027s
I20260809 15:47:10.587245 23273 raft_consensus.cc:1226] T 563daf42d7784db195ea99c3cb21b370 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Rejecting Update request from peer 61e69695258f48baa984c9d395e8ce3f for earlier term 5. Current term is 6. Ops: []
W20260809 15:47:10.583623 23353 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 177837us
I20260809 15:47:10.630961 23400 raft_consensus.cc:1226] T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 7. Current term is 8. Ops: []
I20260809 15:47:10.628011 24463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c48a3049897452383257aa757834fb5" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:47:10.543354 27957 consensus_queue.cc:237] T 998c6cee72cf40228c6dba064c0db2da P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:10.664047 23404 raft_consensus.cc:1226] T ba56725e42f74a1baa4c04d9660727f9 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 5. Current term is 6. Ops: []
W20260809 15:47:10.641079 23468 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.68:34689, user_credentials={real_user=slave}} blocked reactor thread for 71033.6us
I20260809 15:47:10.670509 23353 leader_election.cc:302] T 0da4d5283c8a46878bd39b85e5c71f66 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:47:10.587396 23283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0741ab44d99439abc3aa3bdd27ee160" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:47:10.602464 23395 raft_consensus.cc:1226] T a88421dba80d4d3388aa5eb62e23853b P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 5. Current term is 6. Ops: []
I20260809 15:47:10.587697 23285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d880de657e544468c82157769622a26" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:47:10.601814 23393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0acc47ce21c3422698d1ef7277c3ed63" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:47:10.594944 23394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9e7cadcd24d45fcbfa84209225e8f2d" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 8 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:10.708593 23394 raft_consensus.cc:2379] T f9e7cadcd24d45fcbfa84209225e8f2d P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c in current term 8: Already voted for candidate c0f38c1908414566aa9115bb2cbdd85d in this term.
I20260809 15:47:10.628218 28208 leader_election.cc:288] T 8e4ffcf83c234ae7910ddef3ec0b4bda P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:10.577525 27751 raft_consensus.cc:2726] T 2db057fd10e149a9895dff6ef8c9f038 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260809 15:47:10.628942 27988 raft_consensus.cc:2726] T f7715d71dbac447b9351fc5eb36c5e80 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260809 15:47:10.768543 23285 raft_consensus.cc:2379] T 3d880de657e544468c82157769622a26 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 61e69695258f48baa984c9d395e8ce3f in current term 4: Already voted for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in this term.
I20260809 15:47:10.587330 28131 consensus_queue.cc:1046] T c5c2e425a74b4e64ae70b5def750434c P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.189s
I20260809 15:47:10.575609 28061 raft_consensus.cc:504] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:10.788576 23395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3924b648461e463b9d7925e5671d68ae" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:10.791327 23395 raft_consensus.cc:2379] T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c in current term 9: Already voted for candidate c0f38c1908414566aa9115bb2cbdd85d in this term.
W20260809 15:47:10.799952 28225 log.cc:924] Time spent T 953eea268e17491ca76f3a555860a512 P 65c36d164b0540f099d592f38307ee6c: Append to log took a long time: real 0.091s user 0.000s sys 0.000s
I20260809 15:47:10.790402 28208 raft_consensus.cc:482] T 3296ef50be254fc29cfc77d22a8f7a23 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
I20260809 15:47:10.803766 28208 raft_consensus.cc:504] T 3296ef50be254fc29cfc77d22a8f7a23 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:10.805706 28208 leader_election.cc:288] T 3296ef50be254fc29cfc77d22a8f7a23 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:10.808063 28061 leader_election.cc:288] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:10.808663 23399 raft_consensus.cc:2454] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 9.
W20260809 15:47:10.809520 28227 log.cc:924] Time spent T 297c82ed906d4b48b184985eed3de601 P 65c36d164b0540f099d592f38307ee6c: Append to log took a long time: real 0.219s user 0.001s sys 0.000s
I20260809 15:47:10.790725 28229 raft_consensus.cc:2726] T 0da4d5283c8a46878bd39b85e5c71f66 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260809 15:47:10.640435 28209 raft_consensus.cc:2781] T 0c48a3049897452383257aa757834fb5 P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260809 15:47:11.093086 24448 raft_consensus.cc:1226] T d3be1e356b254d22b995be32f1c12f6f P 61e69695258f48baa984c9d395e8ce3f [term 9 LEADER]: Rejecting Update request from peer c0f38c1908414566aa9115bb2cbdd85d for earlier term 8. Current term is 9. Ops: []
I20260809 15:47:11.064332 23118 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:37326 (request call id 150) took 10580 ms. Trace:
I20260809 15:47:10.823374 23396 raft_consensus.cc:1226] T a3622f6e3e3641cea1960e3a31498bb7 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 5. Current term is 6. Ops: []
W20260809 15:47:10.897014 23470 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 310012us
W20260809 15:47:10.931111 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 360331us
I20260809 15:47:11.100164 24463 raft_consensus.cc:2454] T 0c48a3049897452383257aa757834fb5 P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 5.
I20260809 15:47:11.101549 23353 leader_election.cc:302] T 4bcac4b24e7a4448abdc934bba5a624e P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c
I20260809 15:47:10.817914 23392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ae14a74a26f434db08c652e097385c8" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:11.113802 23351 leader_election.cc:302] T 213543cdbe814c89a17a963469922c1b P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
W20260809 15:47:10.859094 24410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 318059us
W20260809 15:47:10.879865 24406 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.20.202.65:35459 timed out after 1.604s (SENT) [suppressed 28 similar messages]
I20260809 15:47:10.590948 27493 raft_consensus.cc:2726] T 297c82ed906d4b48b184985eed3de601 P c0f38c1908414566aa9115bb2cbdd85d [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
W20260809 15:47:11.120766 23472 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 555614us
I20260809 15:47:11.114480 23392 raft_consensus.cc:2454] T 6ae14a74a26f434db08c652e097385c8 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 7.
I20260809 15:47:11.088343 27792 tablet_bootstrap.cc:492] T 4b30f28311964a0b9149a50495e83d8f P 61e69695258f48baa984c9d395e8ce3f: Bootstrap complete.
I20260809 15:47:11.123772 23470 leader_election.cc:302] T 4b30f28311964a0b9149a50495e83d8f P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
W20260809 15:47:11.082329 28011 consensus_meta.cc:287] T d0721a4403a440f383318a3cea9d1442 P 65c36d164b0540f099d592f38307ee6c: Time spent flushing consensus metadata: real 0.506s user 0.006s sys 0.000s
I20260809 15:47:10.710346 27492 raft_consensus.cc:2781] T 30c1740ef6874df0a332fbd6fd4b03ce P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260809 15:47:11.129091 23273 raft_consensus.cc:1226] T 0a51c0baaa6e432997c9d3552af4f254 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Rejecting Update request from peer 61e69695258f48baa984c9d395e8ce3f for earlier term 6. Current term is 7. Ops: []
I20260809 15:47:11.128041 23285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c48a3049897452383257aa757834fb5" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:10.640594 28171 raft_consensus.cc:504] T 907bacfa79d04a55a68c8dbe60056462 P 61e69695258f48baa984c9d395e8ce3f [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:11.152132 23399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b30f28311964a0b9149a50495e83d8f" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:47:11.130895 23285 raft_consensus.cc:2454] T 0c48a3049897452383257aa757834fb5 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 5.
I20260809 15:47:11.151422 23400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7660dfa2ef694927b8eda4100a2bcd9b" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:47:11.100196 23118 rpcz_store.cc:276] 0809 15:47:00.483817 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:00.484046 (+ 229us) service_pool.cc:227] Handling call
0809 15:47:11.064228 (+10580182us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0809 15:47:09.975941 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0809 15:47:10.031946 (+ 56005us) write_op.cc:432] Acquiring schema lock in shared mode
0809 15:47:10.032002 (+ 56us) write_op.cc:435] Acquired schema lock
0809 15:47:10.032015 (+ 13us) tablet.cc:660] Decoding operations
0809 15:47:10.032981 (+ 966us) write_op.cc:620] Acquiring the partition lock for write op
0809 15:47:10.033076 (+ 95us) write_op.cc:641] Partition lock acquired for write op
0809 15:47:10.033105 (+ 29us) tablet.cc:683] Acquiring locks for 7 operations
0809 15:47:10.033797 (+ 692us) tablet.cc:699] Row locks acquired
0809 15:47:10.033827 (+ 30us) write_op.cc:260] PREPARE: finished
0809 15:47:10.034022 (+ 195us) write_op.cc:270] Start()
0809 15:47:10.034183 (+ 161us) write_op.cc:276] Timestamp: P: 1786290430033957 usec, L: 0
0809 15:47:10.034217 (+ 34us) op_driver.cc:331] REPLICATION: starting
0809 15:47:10.034912 (+ 695us) log.cc:842] Serialized 2835 byte log entry
0809 15:47:10.543213 (+508301us) op_driver.cc:444] REPLICATION: finished
0809 15:47:10.840931 (+297718us) write_op.cc:301] APPLY: starting
0809 15:47:10.843282 (+ 2351us) tablet.cc:1365] starting BulkCheckPresence
0809 15:47:10.843530 (+ 248us) tablet.cc:1368] finished BulkCheckPresence
0809 15:47:10.843667 (+ 137us) tablet.cc:1370] starting ApplyRowOperation cycle
0809 15:47:10.929940 (+ 86273us) tablet.cc:1381] finished ApplyRowOperation cycle
0809 15:47:10.973355 (+ 43415us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=7
0809 15:47:10.973407 (+ 52us) write_op.cc:312] APPLY: finished
0809 15:47:10.973763 (+ 356us) log.cc:842] Serialized 59 byte log entry
0809 15:47:10.973945 (+ 182us) write_op.cc:489] Releasing partition, row and schema locks
0809 15:47:10.974061 (+ 116us) write_op.cc:454] Released schema lock
0809 15:47:10.974125 (+ 64us) write_op.cc:341] FINISH: Updating metrics
0809 15:47:10.974471 (+ 346us) write_op.cc:489] Releasing partition, row and schema locks
0809 15:47:10.974507 (+ 36us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":19,"child_traces":[["op",{"apply.queue_time_us":234264,"apply.run_cpu_time_us":11539,"apply.run_wall_time_us":318056,"num_ops":7,"prepare.queue_time_us":400063,"prepare.run_cpu_time_us":56017,"prepare.run_wall_time_us":276037,"raft.queue_time_us":117648,"raft.run_cpu_time_us":18524,"raft.run_wall_time_us":131616,"replication_time_us":508896,"thread_start_us":481485,"threads_started":4,"wal-append.queue_time_us":240445}]]}
I20260809 15:47:11.125855 28229 raft_consensus.cc:3037] T 4bcac4b24e7a4448abdc934bba5a624e P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:47:10.601619 27953 raft_consensus.cc:504] T 472816994d814cb198b69e5248697440 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:11.119643 28209 raft_consensus.cc:482] T 0c48a3049897452383257aa757834fb5 P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:47:11.173261 28209 raft_consensus.cc:3037] T 0c48a3049897452383257aa757834fb5 P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Advancing to term 6
I20260809 15:47:11.160529 23392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3aa6a9a52804df0ba9b0ca3c82f38c3" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:47:10.932459 27988 raft_consensus.cc:3032] T c5c2e425a74b4e64ae70b5def750434c P c0f38c1908414566aa9115bb2cbdd85d [term 6 LEADER]: Stepping down as leader of term 6
I20260809 15:47:11.175465 27988 raft_consensus.cc:729] T c5c2e425a74b4e64ae70b5def750434c P c0f38c1908414566aa9115bb2cbdd85d [term 6 LEADER]: Becoming Follower/Learner. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:47:11.176234 27988 consensus_queue.cc:260] T c5c2e425a74b4e64ae70b5def750434c P c0f38c1908414566aa9115bb2cbdd85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:11.129029 23351 leader_election.cc:302] T d2d4f749fcfd4fb3bfb8115af3471598 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:47:11.127900 27492 raft_consensus.cc:482] T 30c1740ef6874df0a332fbd6fd4b03ce P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Starting leader election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
I20260809 15:47:11.163563 23393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "538a228ddfb54bc6a36528d5ff07597b" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:47:11.032217 28168 raft_consensus.cc:482] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:47:11.167727 23399 raft_consensus.cc:3037] T 4b30f28311964a0b9149a50495e83d8f P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Advancing to term 5
I20260809 15:47:11.157851 28061 raft_consensus.cc:2726] T 213543cdbe814c89a17a963469922c1b P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260809 15:47:11.180886 23393 raft_consensus.cc:2396] T 538a228ddfb54bc6a36528d5ff07597b P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c for term 7 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260809 15:47:11.182809 24468 raft_consensus.cc:1226] T 907bacfa79d04a55a68c8dbe60056462 P 61e69695258f48baa984c9d395e8ce3f [term 8 FOLLOWER]: Rejecting Update request from peer 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 7. Current term is 8. Ops: []
I20260809 15:47:11.183454 23116 catalog_manager.cc:5495] T f7715d71dbac447b9351fc5eb36c5e80 P 67b3aeadfa7a432abe8af4d292e04ab9 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "65c36d164b0540f099d592f38307ee6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } }
I20260809 15:47:11.184377 27953 leader_election.cc:288] T 472816994d814cb198b69e5248697440 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:11.158385 23396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5c2e425a74b4e64ae70b5def750434c" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:11.180859 23395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05ad7d8fdca64e3da527280ca5be9280" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:11.187798 23404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae6d2c08a7a94f58ac8dd698e3fd456c" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:47:11.192392 23394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cef3e7277d7d4760824a3436ed4a847d" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
W20260809 15:47:11.200693 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 71744.4us
I20260809 15:47:11.204427 23351 leader_election.cc:302] T a3622f6e3e3641cea1960e3a31498bb7 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
W20260809 15:47:11.205333 23472 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 73960.8us
I20260809 15:47:11.205495 23351 leader_election.cc:302] T 0c29937a80724ddb9d07a27534a146f3 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
W20260809 15:47:11.203593 23470 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 83545.5us
I20260809 15:47:11.168100 28131 raft_consensus.cc:482] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:11.207542 23395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae6d2c08a7a94f58ac8dd698e3fd456c" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:11.188508 27492 raft_consensus.cc:3037] T 30c1740ef6874df0a332fbd6fd4b03ce P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Advancing to term 8
I20260809 15:47:11.204576 23400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7660dfa2ef694927b8eda4100a2bcd9b" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:11.217789 23395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4564dfc6d4094c1b9240b367ccecf3b1" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:11.215466 23470 leader_election.cc:302] T cee02fff2c9c48a19f43eb358a89e7b0 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 61e69695258f48baa984c9d395e8ce3f, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:47:11.218390 23351 leader_election.cc:302] T a88421dba80d4d3388aa5eb62e23853b P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:47:11.228289 23393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0acc47ce21c3422698d1ef7277c3ed63" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:11.212529 27988 raft_consensus.cc:3037] T c5c2e425a74b4e64ae70b5def750434c P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Advancing to term 7
I20260809 15:47:11.216135 28114 raft_consensus.cc:482] T 16ca412a0dae4feebe85dd01bf1a9173 P 61e69695258f48baa984c9d395e8ce3f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:11.193586 27792 ts_tablet_manager.cc:1397] T 4b30f28311964a0b9149a50495e83d8f P 61e69695258f48baa984c9d395e8ce3f: Time spent bootstrapping tablet: real 3.185s user 0.026s sys 0.085s
W20260809 15:47:11.261795 23472 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 51859us
I20260809 15:47:11.263808 23400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7baa2492df0340c4a5ac241959ce1ac3" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:11.264413 23400 raft_consensus.cc:2454] T 7baa2492df0340c4a5ac241959ce1ac3 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 5.
I20260809 15:47:11.265209 23400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bcac4b24e7a4448abdc934bba5a624e" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:47:11.227703 28061 raft_consensus.cc:2726] T a3622f6e3e3641cea1960e3a31498bb7 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260809 15:47:11.269368 23404 raft_consensus.cc:1226] T d0721a4403a440f383318a3cea9d1442 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 6. Current term is 7. Ops: []
I20260809 15:47:11.270325 23404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3be1e356b254d22b995be32f1c12f6f" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:11.271080 23404 raft_consensus.cc:2396] T d3be1e356b254d22b995be32f1c12f6f P c0f38c1908414566aa9115bb2cbdd85d [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 for term 9 because replica has last-logged OpId of term: 8 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 6.
I20260809 15:47:11.272239 26966 raft_consensus.cc:2781] T 4b30f28311964a0b9149a50495e83d8f P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Leader election won for term 5
I20260809 15:47:11.201402 23116 catalog_manager.cc:5495] T d0721a4403a440f383318a3cea9d1442 P 67b3aeadfa7a432abe8af4d292e04ab9 reported cstate change: term changed from 3 to 6, leader changed from c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66) to 65c36d164b0540f099d592f38307ee6c (127.20.202.67). New cstate: current_term: 6 leader_uuid: "65c36d164b0540f099d592f38307ee6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } }
I20260809 15:47:11.271915 28114 raft_consensus.cc:504] T 16ca412a0dae4feebe85dd01bf1a9173 P 61e69695258f48baa984c9d395e8ce3f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:11.206979 27953 raft_consensus.cc:2726] T d2d4f749fcfd4fb3bfb8115af3471598 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
W20260809 15:47:11.274132 24404 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.533s user 0.023s sys 0.031s
W20260809 15:47:11.280421 24404 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.622s user 0.024s sys 0.032s
I20260809 15:47:11.274545 28011 consensus_queue.cc:1035] T 2db057fd10e149a9895dff6ef8c9f038 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20260809 15:47:11.275466 23470 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 60085.9us
I20260809 15:47:11.210566 28131 raft_consensus.cc:504] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:11.276778 28247 raft_consensus.cc:2726] T cee02fff2c9c48a19f43eb358a89e7b0 P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260809 15:47:11.278085 23395 raft_consensus.cc:2379] T 4564dfc6d4094c1b9240b367ccecf3b1 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c in current term 7: Already voted for candidate c0f38c1908414566aa9115bb2cbdd85d in this term.
I20260809 15:47:11.278017 23393 raft_consensus.cc:1226] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Rejecting Update request from peer 61e69695258f48baa984c9d395e8ce3f for earlier term 4. Current term is 5. Ops: []
I20260809 15:47:11.272771 26966 raft_consensus.cc:686] T 4b30f28311964a0b9149a50495e83d8f P 65c36d164b0540f099d592f38307ee6c [term 5 LEADER]: Becoming Leader. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:47:11.289572 23273 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d2d4f749fcfd4fb3bfb8115af3471598"
dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
from {username='slave'} at 127.0.0.1:53554
I20260809 15:47:11.227567 27751 raft_consensus.cc:2726] T 0c29937a80724ddb9d07a27534a146f3 P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260809 15:47:11.292544 23273 raft_consensus.cc:482] T d2d4f749fcfd4fb3bfb8115af3471598 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 15:47:11.294412 23273 raft_consensus.cc:3037] T d2d4f749fcfd4fb3bfb8115af3471598 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Advancing to term 7
I20260809 15:47:11.201174 28168 raft_consensus.cc:504] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:11.266898 28209 raft_consensus.cc:504] T 0c48a3049897452383257aa757834fb5 P 61e69695258f48baa984c9d395e8ce3f [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
W20260809 15:47:11.340284 24406 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 460418us
I20260809 15:47:11.273865 27957 raft_consensus.cc:2726] T a88421dba80d4d3388aa5eb62e23853b P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260809 15:47:11.352092 23392 raft_consensus.cc:2396] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c for term 7 because replica has last-logged OpId of term: 6 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 5.
I20260809 15:47:11.353420 23393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998c6cee72cf40228c6dba064c0db2da" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
W20260809 15:47:11.354290 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 68362.7us
I20260809 15:47:11.300307 23116 catalog_manager.cc:5495] T 953eea268e17491ca76f3a555860a512 P 67b3aeadfa7a432abe8af4d292e04ab9 reported cstate change: term changed from 4 to 5, leader changed from 65c36d164b0540f099d592f38307ee6c (127.20.202.67) to 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65). New cstate: current_term: 5 leader_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } health_report { overall_health: UNKNOWN } } }
W20260809 15:47:11.356750 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 167203us
W20260809 15:47:11.355700 23470 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 59889.9us
W20260809 15:47:11.375245 23472 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 97093.3us
I20260809 15:47:11.377460 23351 leader_election.cc:302] T 563daf42d7784db195ea99c3cb21b370 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:47:11.373948 23116 catalog_manager.cc:5495] T 99bdb93503db4de4a1cfb0161ea0c07c P 67b3aeadfa7a432abe8af4d292e04ab9 reported cstate change: term changed from 1 to 4. New cstate: current_term: 4 leader_uuid: "c0f38c1908414566aa9115bb2cbdd85d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } }
I20260809 15:47:11.379832 23223 leader_election.cc:302] T 0c48a3049897452383257aa757834fb5 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:47:11.380406 24406 leader_election.cc:302] T c0741ab44d99439abc3aa3bdd27ee160 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:47:11.398520 23520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bcac4b24e7a4448abdc934bba5a624e" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:47:11.398744 23395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ced718c1a5c4da79cb8c1ebd1435b87" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:11.398845 23393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba56725e42f74a1baa4c04d9660727f9" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:11.402026 23529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7660dfa2ef694927b8eda4100a2bcd9b" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:11.324837 28246 raft_consensus.cc:482] T e14d118b2eaa42aa8a7cd4025aba7128 P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
I20260809 15:47:11.352191 28198 raft_consensus.cc:482] T 472816994d814cb198b69e5248697440 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
I20260809 15:47:11.404335 23395 raft_consensus.cc:2454] T 6ced718c1a5c4da79cb8c1ebd1435b87 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 4.
W20260809 15:47:11.404982 23228 leader_election.cc:334] T d3be1e356b254d22b995be32f1c12f6f P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263): Timed out: RequestConsensusVote RPC to 127.20.202.66:36263 timed out after 1.851s (SENT)
I20260809 15:47:11.405768 23393 raft_consensus.cc:2379] T ba56725e42f74a1baa4c04d9660727f9 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 in current term 6: Already voted for candidate c0f38c1908414566aa9115bb2cbdd85d in this term.
I20260809 15:47:11.406720 23518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c48a3049897452383257aa757834fb5" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:11.407090 23395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e4ffcf83c234ae7910ddef3ec0b4bda" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:11.407997 23395 raft_consensus.cc:2454] T 8e4ffcf83c234ae7910ddef3ec0b4bda P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 4.
I20260809 15:47:11.408047 23518 raft_consensus.cc:2454] T 0c48a3049897452383257aa757834fb5 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 5.
I20260809 15:47:11.411178 23521 raft_consensus.cc:1261] T 16ca412a0dae4feebe85dd01bf1a9173 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Refusing update from remote peer 67b3aeadfa7a432abe8af4d292e04ab9: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
W20260809 15:47:11.412333 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 57361.5us
I20260809 15:47:11.436127 23404 raft_consensus.cc:1261] T d3be1e356b254d22b995be32f1c12f6f P c0f38c1908414566aa9115bb2cbdd85d [term 9 FOLLOWER]: Refusing update from remote peer 61e69695258f48baa984c9d395e8ce3f: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 9 index: 7. (term mismatch)
I20260809 15:47:11.436478 23404 pending_rounds.cc:76] T d3be1e356b254d22b995be32f1c12f6f P c0f38c1908414566aa9115bb2cbdd85d: Aborting all ops after (but not including) 6
I20260809 15:47:11.443104 23526 raft_consensus.cc:1226] T 05ad7d8fdca64e3da527280ca5be9280 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Rejecting Update request from peer 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 6. Current term is 7. Ops: []
I20260809 15:47:11.443521 23393 raft_consensus.cc:1226] T 953eea268e17491ca76f3a555860a512 P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Rejecting Update request from peer 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 7. Current term is 8. Ops: []
I20260809 15:47:11.454421 23276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30c1740ef6874df0a332fbd6fd4b03ce" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:47:11.455282 23280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e4ffcf83c234ae7910ddef3ec0b4bda" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:11.436678 23404 pending_rounds.cc:97] T d3be1e356b254d22b995be32f1c12f6f P c0f38c1908414566aa9115bb2cbdd85d: Aborting uncommitted NO_OP operation due to leader change: 8.7
W20260809 15:47:11.462778 24410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 202070us
I20260809 15:47:11.461416 23274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9e7cadcd24d45fcbfa84209225e8f2d" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 8 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:47:11.431370 28230 raft_consensus.cc:2781] T 563daf42d7784db195ea99c3cb21b370 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Leader election won for term 6
I20260809 15:47:11.508131 21289 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 15:47:11.472118 23404 raft_consensus.cc:2866] T d3be1e356b254d22b995be32f1c12f6f P c0f38c1908414566aa9115bb2cbdd85d [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 15:47:11.508966 23395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "907bacfa79d04a55a68c8dbe60056462" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:47:11.505724 23393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3be1e356b254d22b995be32f1c12f6f" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:11.507521 23285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3296ef50be254fc29cfc77d22a8f7a23" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
W20260809 15:47:11.508059 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 54800.8us
I20260809 15:47:11.378208 28263 consensus_queue.cc:1046] T 563daf42d7784db195ea99c3cb21b370 P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.118s
W20260809 15:47:11.516572 24406 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 115142us
I20260809 15:47:11.399724 28244 consensus_queue.cc:1035] T cef3e7277d7d4760824a3436ed4a847d P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.137s
W20260809 15:47:11.515865 23472 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 140053us
I20260809 15:47:11.415907 27956 raft_consensus.cc:2781] T 0c48a3049897452383257aa757834fb5 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260809 15:47:11.519368 23116 catalog_manager.cc:5495] T 0a51c0baaa6e432997c9d3552af4f254 P 67b3aeadfa7a432abe8af4d292e04ab9 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260809 15:47:11.517848 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 112905us
W20260809 15:47:11.449554 28210 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.519s user 0.001s sys 0.023s
I20260809 15:47:11.504846 27957 raft_consensus.cc:482] T 907bacfa79d04a55a68c8dbe60056462 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:11.527125 23280 raft_consensus.cc:1226] T 0acc47ce21c3422698d1ef7277c3ed63 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Rejecting Update request from peer c0f38c1908414566aa9115bb2cbdd85d for earlier term 7. Current term is 8. Ops: []
I20260809 15:47:11.504213 27493 raft_consensus.cc:482] T df20671a7b874dbb98c0ef28371c1308 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
I20260809 15:47:11.410907 28198 raft_consensus.cc:504] T 472816994d814cb198b69e5248697440 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:11.504408 28171 leader_election.cc:288] T 907bacfa79d04a55a68c8dbe60056462 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 8 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:47:11.505043 28208 raft_consensus.cc:482] T 2424fd9bd7f14e508fd106cbfda786a4 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:47:11.530702 23393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a88421dba80d4d3388aa5eb62e23853b" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:11.533208 24455 raft_consensus.cc:1261] T cee02fff2c9c48a19f43eb358a89e7b0 P 61e69695258f48baa984c9d395e8ce3f [term 8 FOLLOWER]: Refusing update from remote peer 67b3aeadfa7a432abe8af4d292e04ab9: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260809 15:47:11.533388 23274 raft_consensus.cc:1226] T ae6d2c08a7a94f58ac8dd698e3fd456c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Rejecting Update request from peer c0f38c1908414566aa9115bb2cbdd85d for earlier term 7. Current term is 8. Ops: []
I20260809 15:47:11.531816 23116 catalog_manager.cc:5495] T f9e7cadcd24d45fcbfa84209225e8f2d P 67b3aeadfa7a432abe8af4d292e04ab9 reported cstate change: term changed from 4 to 5, leader changed from 65c36d164b0540f099d592f38307ee6c (127.20.202.67) to 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65). New cstate: current_term: 5 leader_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } health_report { overall_health: UNKNOWN } } }
I20260809 15:47:11.540570 27492 raft_consensus.cc:504] T 30c1740ef6874df0a332fbd6fd4b03ce P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:11.512176 27953 raft_consensus.cc:482] T 4564dfc6d4094c1b9240b367ccecf3b1 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:11.431252 27892 consensus_queue.cc:1046] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.087s
I20260809 15:47:11.454620 27751 raft_consensus.cc:482] T d0721a4403a440f383318a3cea9d1442 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:11.521018 28230 raft_consensus.cc:686] T 563daf42d7784db195ea99c3cb21b370 P c0f38c1908414566aa9115bb2cbdd85d [term 6 LEADER]: Becoming Leader. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
W20260809 15:47:11.557303 23470 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 176022us
I20260809 15:47:11.561008 23393 raft_consensus.cc:2379] T a88421dba80d4d3388aa5eb62e23853b P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 in current term 6: Already voted for candidate c0f38c1908414566aa9115bb2cbdd85d in this term.
I20260809 15:47:11.526968 27956 raft_consensus.cc:482] T 0c48a3049897452383257aa757834fb5 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:47:11.524425 28203 raft_consensus.cc:2726] T c0741ab44d99439abc3aa3bdd27ee160 P 61e69695258f48baa984c9d395e8ce3f [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
W20260809 15:47:11.564109 23468 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.68:34689, user_credentials={real_user=slave}} blocked reactor thread for 149907us
I20260809 15:47:11.602421 23518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30c1740ef6874df0a332fbd6fd4b03ce" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:47:11.564653 23404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c29937a80724ddb9d07a27534a146f3" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 8 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:11.409705 28246 raft_consensus.cc:504] T e14d118b2eaa42aa8a7cd4025aba7128 P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:11.617614 24468 raft_consensus.cc:1226] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 61e69695258f48baa984c9d395e8ce3f [term 7 FOLLOWER]: Rejecting Update request from peer c0f38c1908414566aa9115bb2cbdd85d for earlier term 6. Current term is 7. Ops: [6.6-6.6]
I20260809 15:47:11.617539 23517 raft_consensus.cc:1226] T ae6d2c08a7a94f58ac8dd698e3fd456c P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Rejecting Update request from peer c0f38c1908414566aa9115bb2cbdd85d for earlier term 7. Current term is 8. Ops: []
I20260809 15:47:11.571820 23116 catalog_manager.cc:5495] T 2db057fd10e149a9895dff6ef8c9f038 P 67b3aeadfa7a432abe8af4d292e04ab9 reported cstate change: term changed from 2 to 4, leader changed from 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65) to 65c36d164b0540f099d592f38307ee6c (127.20.202.67). New cstate: current_term: 4 leader_uuid: "65c36d164b0540f099d592f38307ee6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } }
W20260809 15:47:11.592835 24410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 64589.7us
I20260809 15:47:11.434481 26966 consensus_queue.cc:237] T 4b30f28311964a0b9149a50495e83d8f P 65c36d164b0540f099d592f38307ee6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } attrs { replace: true } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } }
I20260809 15:47:11.595032 28168 leader_election.cc:288] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:11.595813 28230 consensus_queue.cc:237] T 563daf42d7784db195ea99c3cb21b370 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
W20260809 15:47:11.592559 23472 connection.cc:232] RPC call timeout handler was delayed by -1.032827781047672s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20779999999999998s for a response
I20260809 15:47:11.593039 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e17dcb5efe45b8b6a8c42da03b4819" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:11.555644 27493 raft_consensus.cc:504] T df20671a7b874dbb98c0ef28371c1308 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:11.572577 28061 consensus_queue.cc:1046] T 0acc47ce21c3422698d1ef7277c3ed63 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: INVALID_TERM, Last received: 7.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.037s
I20260809 15:47:11.597020 27751 raft_consensus.cc:504] T d0721a4403a440f383318a3cea9d1442 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:11.615863 23518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472816994d814cb198b69e5248697440" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:11.599212 27956 raft_consensus.cc:3037] T 0c48a3049897452383257aa757834fb5 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Advancing to term 6
I20260809 15:47:11.616091 23393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7715d71dbac447b9351fc5eb36c5e80" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
W20260809 15:47:11.600952 28251 log.cc:924] Time spent T 07fcabb569414bddbd39df6cedd24e64 P c0f38c1908414566aa9115bb2cbdd85d: Append to log took a long time: real 0.373s user 0.001s sys 0.000s
I20260809 15:47:11.616894 23404 raft_consensus.cc:2379] T 0c29937a80724ddb9d07a27534a146f3 P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 in current term 8: Already voted for candidate c0f38c1908414566aa9115bb2cbdd85d in this term.
I20260809 15:47:11.616719 23528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "953eea268e17491ca76f3a555860a512" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:47:11.615952 23274 raft_consensus.cc:1226] T 294d4bcc1e7a477cb56455581277a60f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 5. Current term is 6. Ops: []
I20260809 15:47:11.594115 28114 leader_election.cc:288] T 16ca412a0dae4feebe85dd01bf1a9173 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:11.563704 28171 raft_consensus.cc:482] T 294d4bcc1e7a477cb56455581277a60f P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:47:11.641865 23116 catalog_manager.cc:5495] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 67b3aeadfa7a432abe8af4d292e04ab9 reported cstate change: term changed from 1 to 4, leader changed from c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66) to 65c36d164b0540f099d592f38307ee6c (127.20.202.67). New cstate: current_term: 4 leader_uuid: "65c36d164b0540f099d592f38307ee6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } }
I20260809 15:47:11.619680 24410 leader_election.cc:302] T d3e94b7e50754a81b3d72a07476cb7b9 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:11.624706 23353 leader_election.cc:302] T 30c1740ef6874df0a332fbd6fd4b03ce P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:47:11.624599 23517 raft_consensus.cc:1261] T 05ad7d8fdca64e3da527280ca5be9280 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260809 15:47:11.632827 23518 raft_consensus.cc:2454] T 472816994d814cb198b69e5248697440 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 5.
I20260809 15:47:11.589749 27953 raft_consensus.cc:504] T 4564dfc6d4094c1b9240b367ccecf3b1 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:11.570556 28208 raft_consensus.cc:504] T 2424fd9bd7f14e508fd106cbfda786a4 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:11.632642 23521 raft_consensus.cc:1261] T cef3e7277d7d4760824a3436ed4a847d P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
W20260809 15:47:11.633651 24406 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 111544us
I20260809 15:47:11.630679 23520 raft_consensus.cc:2454] T 4bcac4b24e7a4448abdc934bba5a624e P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 4.
I20260809 15:47:11.616466 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a823d35f4f348d1b52ba9e6d80b00d2" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:11.636205 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad8a3b25d7b49e5a2142d5709edd0c7" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f"
I20260809 15:47:11.636865 24463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3924b648461e463b9d7925e5671d68ae" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f"
I20260809 15:47:11.597571 28131 leader_election.cc:288] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:11.676446 23393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0acc47ce21c3422698d1ef7277c3ed63" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:11.617271 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad8a3b25d7b49e5a2142d5709edd0c7" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
W20260809 15:47:11.625172 28257 log.cc:924] Time spent T 0c29937a80724ddb9d07a27534a146f3 P 65c36d164b0540f099d592f38307ee6c: Append to log took a long time: real 0.347s user 0.000s sys 0.001s
I20260809 15:47:11.640291 23529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3924b648461e463b9d7925e5671d68ae" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:47:11.640036 24448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e4ffcf83c234ae7910ddef3ec0b4bda" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:47:11.620729 23519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9e7cadcd24d45fcbfa84209225e8f2d" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 8 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
W20260809 15:47:11.642107 23472 leader_election.cc:334] T 4b30f28311964a0b9149a50495e83d8f P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263): Timed out: RequestConsensusVote RPC to 127.20.202.66:36263 timed out after 1.522s (SENT)
I20260809 15:47:11.559518 27957 raft_consensus.cc:504] T 907bacfa79d04a55a68c8dbe60056462 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:11.644234 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0acc47ce21c3422698d1ef7277c3ed63" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:47:11.592613 28248 consensus_queue.cc:1046] T ae6d2c08a7a94f58ac8dd698e3fd456c P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.051s
I20260809 15:47:11.677599 23526 raft_consensus.cc:3037] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Advancing to term 5
I20260809 15:47:11.678889 23404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2db057fd10e149a9895dff6ef8c9f038" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
W20260809 15:47:11.684859 23353 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 60219.4us
I20260809 15:47:11.687167 23116 catalog_manager.cc:5495] T d1a134ba33ea4076bbaac0ae460c17ba P 67b3aeadfa7a432abe8af4d292e04ab9 reported cstate change: term changed from 1 to 4, leader changed from c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66) to 65c36d164b0540f099d592f38307ee6c (127.20.202.67). New cstate: current_term: 4 leader_uuid: "65c36d164b0540f099d592f38307ee6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } }
I20260809 15:47:11.675534 27493 leader_election.cc:288] T df20671a7b874dbb98c0ef28371c1308 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:11.683405 23393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7baa2492df0340c4a5ac241959ce1ac3" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:47:11.683740 23528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3aa6a9a52804df0ba9b0ca3c82f38c3" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:11.684401 24452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a823d35f4f348d1b52ba9e6d80b00d2" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:47:11.685757 23472 leader_election.cc:302] T 4b30f28311964a0b9149a50495e83d8f P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:11.686031 23519 raft_consensus.cc:1226] T 7660dfa2ef694927b8eda4100a2bcd9b P 65c36d164b0540f099d592f38307ee6c [term 6 LEADER]: Rejecting Update request from peer c0f38c1908414566aa9115bb2cbdd85d for earlier term 5. Current term is 6. Ops: []
I20260809 15:47:11.682235 23396 raft_consensus.cc:2454] T c5c2e425a74b4e64ae70b5def750434c P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 7.
I20260809 15:47:11.687211 23404 raft_consensus.cc:2379] T 2db057fd10e149a9895dff6ef8c9f038 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 in current term 7: Already voted for candidate c0f38c1908414566aa9115bb2cbdd85d in this term.
I20260809 15:47:11.642617 28294 raft_consensus.cc:482] T c0741ab44d99439abc3aa3bdd27ee160 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:11.698004 23353 leader_election.cc:302] T 953eea268e17491ca76f3a555860a512 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:47:11.691222 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472816994d814cb198b69e5248697440" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:47:11.700763 23395 raft_consensus.cc:2379] T 907bacfa79d04a55a68c8dbe60056462 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 in current term 7: Already voted for candidate c0f38c1908414566aa9115bb2cbdd85d in this term.
W20260809 15:47:11.711074 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 138719us
I20260809 15:47:11.707640 23519 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "538a228ddfb54bc6a36528d5ff07597b"
dest_uuid: "65c36d164b0540f099d592f38307ee6c"
mode: GRACEFUL
new_leader_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
from {username='slave'} at 127.0.0.1:48888
I20260809 15:47:11.683377 28220 raft_consensus.cc:3032] T 294d4bcc1e7a477cb56455581277a60f P 65c36d164b0540f099d592f38307ee6c [term 5 LEADER]: Stepping down as leader of term 5
I20260809 15:47:11.715588 23519 raft_consensus.cc:595] T 538a228ddfb54bc6a36528d5ff07597b P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Received request to transfer leadership to TS c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:11.716063 23519 raft_consensus.cc:603] T 538a228ddfb54bc6a36528d5ff07597b P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260809 15:47:11.701817 23526 raft_consensus.cc:2454] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 5.
W20260809 15:47:11.692312 28276 log.cc:924] Time spent T 763e58d6ba644eea830a6c1664d39c9b P c0f38c1908414566aa9115bb2cbdd85d: Append to log took a long time: real 0.095s user 0.000s sys 0.000s
W20260809 15:47:11.694895 28281 log.cc:924] Time spent T 0a51c0baaa6e432997c9d3552af4f254 P 61e69695258f48baa984c9d395e8ce3f: Append to log took a long time: real 0.126s user 0.000s sys 0.000s
I20260809 15:47:11.689275 27988 raft_consensus.cc:2726] T 30c1740ef6874df0a332fbd6fd4b03ce P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260809 15:47:11.720005 23353 leader_election.cc:302] T f9e7cadcd24d45fcbfa84209225e8f2d P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:47:11.721282 23276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df20671a7b874dbb98c0ef28371c1308" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:11.728253 24452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3aa6a9a52804df0ba9b0ca3c82f38c3" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:47:11.722219 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2424fd9bd7f14e508fd106cbfda786a4" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:11.718657 23404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5c2e425a74b4e64ae70b5def750434c" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:47:11.729918 23402 raft_consensus.cc:2454] T 2424fd9bd7f14e508fd106cbfda786a4 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 7.
W20260809 15:47:11.724960 24410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 105352us
W20260809 15:47:11.703295 28229 consensus_meta.cc:287] T 4bcac4b24e7a4448abdc934bba5a624e P c0f38c1908414566aa9115bb2cbdd85d: Time spent flushing consensus metadata: real 0.530s user 0.009s sys 0.000s
I20260809 15:47:11.714252 24455 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3296ef50be254fc29cfc77d22a8f7a23" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:47:11.731585 27751 leader_election.cc:288] T d0721a4403a440f383318a3cea9d1442 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:11.728400 28061 consensus_queue.cc:1046] T ae6d2c08a7a94f58ac8dd698e3fd456c P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.003s
I20260809 15:47:11.732079 23274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0721a4403a440f383318a3cea9d1442" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:11.765254 23274 raft_consensus.cc:2454] T d0721a4403a440f383318a3cea9d1442 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 7.
I20260809 15:47:11.732703 23404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3622f6e3e3641cea1960e3a31498bb7" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:11.719012 28220 raft_consensus.cc:729] T 294d4bcc1e7a477cb56455581277a60f P 65c36d164b0540f099d592f38307ee6c [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:47:11.645543 28271 consensus_queue.cc:1046] T 294d4bcc1e7a477cb56455581277a60f P 65c36d164b0540f099d592f38307ee6c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.080s
I20260809 15:47:11.733381 27792 raft_consensus.cc:348] T 4b30f28311964a0b9149a50495e83d8f P 61e69695258f48baa984c9d395e8ce3f [term 3 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:11.734058 28209 leader_election.cc:288] T 0c48a3049897452383257aa757834fb5 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 6 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:11.734194 23397 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55320 (request call id 1578) took 1130 ms. Trace:
I20260809 15:47:11.735505 23280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30c1740ef6874df0a332fbd6fd4b03ce" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:47:11.736774 28248 raft_consensus.cc:2726] T f9e7cadcd24d45fcbfa84209225e8f2d P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
W20260809 15:47:11.778695 23399 consensus_meta.cc:287] T 4b30f28311964a0b9149a50495e83d8f P c0f38c1908414566aa9115bb2cbdd85d: Time spent flushing consensus metadata: real 0.568s user 0.006s sys 0.002s
I20260809 15:47:11.737924 23353 leader_election.cc:302] T 472816994d814cb198b69e5248697440 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:47:11.738363 28198 leader_election.cc:288] T 472816994d814cb198b69e5248697440 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
W20260809 15:47:11.732111 23472 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 90041.2us
I20260809 15:47:11.739833 23396 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55320 (request call id 1590) took 1072 ms. Trace:
I20260809 15:47:11.645756 28171 raft_consensus.cc:504] T 294d4bcc1e7a477cb56455581277a60f P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
W20260809 15:47:11.727720 23223 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.68:34689, user_credentials={real_user=slave}} blocked reactor thread for 49784.2us
I20260809 15:47:11.740291 23520 consensus_queue.cc:237] T ba56725e42f74a1baa4c04d9660727f9 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } attrs { replace: true } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } }
I20260809 15:47:11.744035 24452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472816994d814cb198b69e5248697440" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:47:11.718029 28131 raft_consensus.cc:2726] T 953eea268e17491ca76f3a555860a512 P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260809 15:47:11.739800 23273 raft_consensus.cc:504] T d2d4f749fcfd4fb3bfb8115af3471598 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:11.721791 28244 raft_consensus.cc:3032] T 0acc47ce21c3422698d1ef7277c3ed63 P c0f38c1908414566aa9115bb2cbdd85d [term 7 LEADER]: Stepping down as leader of term 7
I20260809 15:47:11.766126 23351 leader_election.cc:302] T d0721a4403a440f383318a3cea9d1442 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:47:11.773675 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2d4f749fcfd4fb3bfb8115af3471598" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:11.772445 23404 raft_consensus.cc:2379] T a3622f6e3e3641cea1960e3a31498bb7 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 in current term 6: Already voted for candidate c0f38c1908414566aa9115bb2cbdd85d in this term.
I20260809 15:47:11.772615 23395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1a134ba33ea4076bbaac0ae460c17ba" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:11.788337 23395 raft_consensus.cc:2454] T d1a134ba33ea4076bbaac0ae460c17ba P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 6.
I20260809 15:47:11.779951 23396 rpcz_store.cc:276] 0809 15:47:10.667158 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:11.123153 (+455995us) service_pool.cc:227] Handling call
0809 15:47:11.739688 (+616535us) spinlock_profiling.cc:243] Waited 493 ms on lock 0x7b540009f6d4. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e92242ae 00007f34ee161c27 00007f34e7849c04 00007f34e7849b92 00007f34e7849ac1 00007f34e7849a3c 00007f34e7848cf2 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:47:11.739785 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1132988288}
I20260809 15:47:11.743121 28229 raft_consensus.cc:2726] T 4bcac4b24e7a4448abdc934bba5a624e P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260809 15:47:11.779114 23399 raft_consensus.cc:2454] T 4b30f28311964a0b9149a50495e83d8f P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 5.
I20260809 15:47:11.729348 28280 raft_consensus.cc:3037] T d3e94b7e50754a81b3d72a07476cb7b9 P 61e69695258f48baa984c9d395e8ce3f [term 7 FOLLOWER]: Advancing to term 8
I20260809 15:47:11.793332 23404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "538a228ddfb54bc6a36528d5ff07597b" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:47:11.787117 23277 raft_consensus.cc:1226] T d2d4f749fcfd4fb3bfb8115af3471598 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 6. Current term is 7. Ops: []
W20260809 15:47:11.782063 24410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 51151.7us
I20260809 15:47:11.789743 23402 raft_consensus.cc:2379] T d2d4f749fcfd4fb3bfb8115af3471598 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 in current term 7: Already voted for candidate c0f38c1908414566aa9115bb2cbdd85d in this term.
I20260809 15:47:11.791378 23399 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55320 (request call id 1591) took 1119 ms. Trace:
W20260809 15:47:11.792490 28299 log.cc:924] Time spent T 0c29937a80724ddb9d07a27534a146f3 P 67b3aeadfa7a432abe8af4d292e04ab9: Append to log took a long time: real 0.053s user 0.000s sys 0.000s
I20260809 15:47:11.796805 23273 leader_election.cc:288] T d2d4f749fcfd4fb3bfb8115af3471598 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 7 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:11.792937 28248 consensus_queue.cc:1035] T cef3e7277d7d4760824a3436ed4a847d P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20260809 15:47:11.793466 28245 log.cc:924] Time spent T d3be1e356b254d22b995be32f1c12f6f P 61e69695258f48baa984c9d395e8ce3f: Append to log took a long time: real 0.290s user 0.000s sys 0.001s
I20260809 15:47:11.794147 23396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00a1881267a14ba8b78c81040f17a014" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:11.794780 23395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "907bacfa79d04a55a68c8dbe60056462" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:11.772639 23397 rpcz_store.cc:276] 0809 15:47:10.592707 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:10.595128 (+ 2421us) service_pool.cc:227] Handling call
0809 15:47:10.595482 (+ 354us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 15:47:10.609054 (+ 13572us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:47:10.609082 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 15:47:10.648950 (+ 39868us) log.cc:842] Serialized 26 byte log entry
0809 15:47:10.879501 (+230551us) raft_consensus.cc:1631] Marking committed up to 3
0809 15:47:10.879535 (+ 34us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:47:10.879594 (+ 59us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 15:47:11.722659 (+843065us) raft_consensus.cc:1671] finished
0809 15:47:11.722675 (+ 16us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:47:11.722789 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7413,"raft.queue_time_us":33352,"raft.run_cpu_time_us":134,"raft.run_wall_time_us":19940,"thread_start_us":206946,"threads_started":1,"wal-append.queue_time_us":504637}
I20260809 15:47:11.721136 28255 consensus_queue.cc:1046] T 563daf42d7784db195ea99c3cb21b370 P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.065s
I20260809 15:47:11.792683 23522 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "998c6cee72cf40228c6dba064c0db2da"
dest_uuid: "65c36d164b0540f099d592f38307ee6c"
mode: GRACEFUL
new_leader_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
from {username='slave'} at 127.0.0.1:48888
I20260809 15:47:11.805821 23522 raft_consensus.cc:595] T 998c6cee72cf40228c6dba064c0db2da P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Received request to transfer leadership to TS c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:11.806157 23522 raft_consensus.cc:603] T 998c6cee72cf40228c6dba064c0db2da P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260809 15:47:11.729776 23116 catalog_manager.cc:5495] T 0acc47ce21c3422698d1ef7277c3ed63 P 67b3aeadfa7a432abe8af4d292e04ab9 reported cstate change: term changed from 3 to 5, leader changed from 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65) to 65c36d164b0540f099d592f38307ee6c (127.20.202.67), no longer has a pending config: VOTER 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65), VOTER c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66), VOTER 65c36d164b0540f099d592f38307ee6c (127.20.202.67), NON_VOTER 61e69695258f48baa984c9d395e8ce3f (127.20.202.68). New cstate: current_term: 5 leader_uuid: "65c36d164b0540f099d592f38307ee6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } }
I20260809 15:47:11.797672 23399 rpcz_store.cc:276] 0809 15:47:10.672683 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:11.122904 (+450221us) service_pool.cc:227] Handling call
0809 15:47:11.791209 (+668305us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:47:11.807027 23399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae6d2c08a7a94f58ac8dd698e3fd456c" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:47:11.802440 23395 raft_consensus.cc:2379] T 907bacfa79d04a55a68c8dbe60056462 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 in current term 7: Already voted for candidate c0f38c1908414566aa9115bb2cbdd85d in this term.
I20260809 15:47:11.762220 27988 consensus_queue.cc:1035] T 05ad7d8fdca64e3da527280ca5be9280 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.037s
I20260809 15:47:11.790634 27492 leader_election.cc:288] T 30c1740ef6874df0a332fbd6fd4b03ce P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 8 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:47:11.813136 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a51c0baaa6e432997c9d3552af4f254" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:47:11.816035 23402 raft_consensus.cc:3037] T 0a51c0baaa6e432997c9d3552af4f254 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Advancing to term 7
I20260809 15:47:11.787526 28244 raft_consensus.cc:729] T 0acc47ce21c3422698d1ef7277c3ed63 P c0f38c1908414566aa9115bb2cbdd85d [term 7 LEADER]: Becoming Follower/Learner. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:47:11.706991 28294 raft_consensus.cc:504] T c0741ab44d99439abc3aa3bdd27ee160 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:11.817351 28230 consensus_queue.cc:1046] T 7660dfa2ef694927b8eda4100a2bcd9b P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 8, Time since last communication: 0.000s
I20260809 15:47:11.818142 28230 consensus_queue.cc:1225] T 7660dfa2ef694927b8eda4100a2bcd9b P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Peer 65c36d164b0540f099d592f38307ee6c log is divergent from this leader: its last log entry 6.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
I20260809 15:47:11.819449 28230 raft_consensus.cc:3032] T 7660dfa2ef694927b8eda4100a2bcd9b P c0f38c1908414566aa9115bb2cbdd85d [term 5 LEADER]: Stepping down as leader of term 5
I20260809 15:47:11.802296 28131 raft_consensus.cc:2781] T d0721a4403a440f383318a3cea9d1442 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260809 15:47:11.796792 28309 raft_consensus.cc:2781] T 472816994d814cb198b69e5248697440 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260809 15:47:11.823647 23116 catalog_manager.cc:5495] T 0c29937a80724ddb9d07a27534a146f3 P 67b3aeadfa7a432abe8af4d292e04ab9 reported cstate change: term changed from 1 to 5, leader changed from c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66) to 65c36d164b0540f099d592f38307ee6c (127.20.202.67). New cstate: current_term: 5 leader_uuid: "65c36d164b0540f099d592f38307ee6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } }
I20260809 15:47:11.819814 23353 leader_election.cc:302] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:47:11.820122 28230 raft_consensus.cc:729] T 7660dfa2ef694927b8eda4100a2bcd9b P c0f38c1908414566aa9115bb2cbdd85d [term 5 LEADER]: Becoming Follower/Learner. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:47:11.821559 23395 raft_consensus.cc:1261] T 907bacfa79d04a55a68c8dbe60056462 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Refusing update from remote peer 67b3aeadfa7a432abe8af4d292e04ab9: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260809 15:47:11.877997 23282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e14d118b2eaa42aa8a7cd4025aba7128" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
W20260809 15:47:11.877985 23353 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 58237.3us
I20260809 15:47:11.881496 23396 raft_consensus.cc:1226] T 05ad7d8fdca64e3da527280ca5be9280 P c0f38c1908414566aa9115bb2cbdd85d [term 7 LEADER]: Rejecting Update request from peer 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 6. Current term is 7. Ops: []
I20260809 15:47:11.882377 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0721a4403a440f383318a3cea9d1442" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:11.883033 23526 raft_consensus.cc:2454] T d0721a4403a440f383318a3cea9d1442 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 7.
I20260809 15:47:11.880402 28309 raft_consensus.cc:482] T 472816994d814cb198b69e5248697440 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Starting leader election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
I20260809 15:47:11.888355 23522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df20671a7b874dbb98c0ef28371c1308" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
W20260809 15:47:11.888674 23470 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 61232us
I20260809 15:47:11.891793 23526 raft_consensus.cc:1226] T 0acc47ce21c3422698d1ef7277c3ed63 P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Rejecting Update request from peer c0f38c1908414566aa9115bb2cbdd85d for earlier term 7. Current term is 8. Ops: []
I20260809 15:47:11.826974 28131 raft_consensus.cc:482] T d0721a4403a440f383318a3cea9d1442 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:47:11.895658 23116 catalog_manager.cc:5495] T 7baa2492df0340c4a5ac241959ce1ac3 P 67b3aeadfa7a432abe8af4d292e04ab9 reported cstate change: term changed from 1 to 4, leader changed from 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65) to c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66). New cstate: current_term: 4 leader_uuid: "c0f38c1908414566aa9115bb2cbdd85d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } }
I20260809 15:47:11.885160 28246 leader_election.cc:288] T e14d118b2eaa42aa8a7cd4025aba7128 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:11.926292 23392 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55320 (request call id 1581) took 1332 ms. Trace:
I20260809 15:47:11.928774 23394 raft_consensus.cc:1261] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 7 index: 6. (term mismatch)
I20260809 15:47:11.929207 23228 leader_election.cc:302] T 294d4bcc1e7a477cb56455581277a60f P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c
I20260809 15:47:11.930128 23392 rpcz_store.cc:276] 0809 15:47:10.593591 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:11.127079 (+533488us) service_pool.cc:227] Handling call
0809 15:47:11.926120 (+799041us) spinlock_profiling.cc:243] Waited 126 ms on lock 0x7b540031a1d4. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e92242ae 00007f34ee161c27 00007f34e7849c04 00007f34e7849b92 00007f34e7849ac1 00007f34e7849a3c 00007f34e7848cf2 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:47:11.926241 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":290013440}
I20260809 15:47:11.930675 23394 pending_rounds.cc:76] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P c0f38c1908414566aa9115bb2cbdd85d: Aborting all ops after (but not including) 5
I20260809 15:47:11.936728 23522 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "cee02fff2c9c48a19f43eb358a89e7b0"
dest_uuid: "65c36d164b0540f099d592f38307ee6c"
mode: GRACEFUL
new_leader_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
from {username='slave'} at 127.0.0.1:48888
I20260809 15:47:11.936962 23394 pending_rounds.cc:97] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P c0f38c1908414566aa9115bb2cbdd85d: Aborting uncommitted NO_OP operation due to leader change: 6.6
I20260809 15:47:11.937287 23522 raft_consensus.cc:595] T cee02fff2c9c48a19f43eb358a89e7b0 P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Received request to transfer leadership to TS 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:47:11.937346 23394 raft_consensus.cc:2866] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 15:47:11.937680 23522 raft_consensus.cc:603] T cee02fff2c9c48a19f43eb358a89e7b0 P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260809 15:47:11.887837 28309 raft_consensus.cc:3037] T 472816994d814cb198b69e5248697440 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Advancing to term 6
I20260809 15:47:11.920204 28131 raft_consensus.cc:3037] T d0721a4403a440f383318a3cea9d1442 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Advancing to term 8
I20260809 15:47:11.926368 28208 leader_election.cc:288] T 2424fd9bd7f14e508fd106cbfda786a4 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:11.894603 27792 raft_consensus.cc:729] T 4b30f28311964a0b9149a50495e83d8f P 61e69695258f48baa984c9d395e8ce3f [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: 61e69695258f48baa984c9d395e8ce3f, State: Initialized, Role: LEARNER
I20260809 15:47:11.946136 23228 leader_election.cc:302] T 7baa2492df0340c4a5ac241959ce1ac3 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:47:11.924659 27953 leader_election.cc:288] T 4564dfc6d4094c1b9240b367ccecf3b1 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:11.924490 28209 consensus_queue.cc:1035] T d3be1e356b254d22b995be32f1c12f6f P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.058s
I20260809 15:47:11.947418 28307 consensus_queue.cc:1046] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: INVALID_TERM, Last received: 4.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.160s
I20260809 15:47:11.983847 23394 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55320 (request call id 1592) took 1238 ms. Trace:
I20260809 15:47:11.984223 23394 rpcz_store.cc:276] 0809 15:47:10.745219 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:11.228667 (+483448us) service_pool.cc:227] Handling call
0809 15:47:11.926575 (+697908us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:47:11.938080 (+ 11505us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:47:11.983670 (+ 45590us) spinlock_profiling.cc:243] Waited 658 ms on lock 0x7b540031a1d0. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e921fa09 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:47:11.983794 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1514532096}
I20260809 15:47:11.822505 28220 consensus_queue.cc:260] T 294d4bcc1e7a477cb56455581277a60f P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:11.987900 28220 raft_consensus.cc:3037] T 294d4bcc1e7a477cb56455581277a60f P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Advancing to term 6
I20260809 15:47:11.990028 28280 raft_consensus.cc:2726] T d3e94b7e50754a81b3d72a07476cb7b9 P 61e69695258f48baa984c9d395e8ce3f [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260809 15:47:11.990180 28198 consensus_queue.cc:1046] T d2d4f749fcfd4fb3bfb8115af3471598 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260809 15:47:11.924901 27957 leader_election.cc:288] T 907bacfa79d04a55a68c8dbe60056462 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:11.947863 28315 raft_consensus.cc:2726] T 294d4bcc1e7a477cb56455581277a60f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260809 15:47:11.963827 24453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e14d118b2eaa42aa8a7cd4025aba7128" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:47:11.964257 28319 consensus_queue.cc:1035] T cee02fff2c9c48a19f43eb358a89e7b0 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.030s
I20260809 15:47:11.991607 28244 consensus_queue.cc:260] T 0acc47ce21c3422698d1ef7277c3ed63 P c0f38c1908414566aa9115bb2cbdd85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.3, Last appended by leader: 3, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:11.964524 23116 catalog_manager.cc:5495] T c5c2e425a74b4e64ae70b5def750434c P 67b3aeadfa7a432abe8af4d292e04ab9 reported cstate change: term changed from 2 to 4, leader changed from 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65) to c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66). New cstate: current_term: 4 leader_uuid: "c0f38c1908414566aa9115bb2cbdd85d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } }
I20260809 15:47:11.995492 23116 catalog_manager.cc:5495] T 4b30f28311964a0b9149a50495e83d8f P 67b3aeadfa7a432abe8af4d292e04ab9 reported cstate change: term changed from 2 to 3, leader changed from 65c36d164b0540f099d592f38307ee6c (127.20.202.67) to c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66). New cstate: current_term: 3 leader_uuid: "c0f38c1908414566aa9115bb2cbdd85d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } }
I20260809 15:47:11.996942 28307 raft_consensus.cc:3032] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 61e69695258f48baa984c9d395e8ce3f [term 4 LEADER]: Stepping down as leader of term 4
I20260809 15:47:11.997306 28307 raft_consensus.cc:729] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 61e69695258f48baa984c9d395e8ce3f [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 61e69695258f48baa984c9d395e8ce3f, State: Running, Role: LEADER
W20260809 15:47:11.997499 28318 log.cc:924] Time spent T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 61e69695258f48baa984c9d395e8ce3f: Append to log took a long time: real 0.112s user 0.001s sys 0.000s
I20260809 15:47:11.987828 23402 raft_consensus.cc:2454] T 0a51c0baaa6e432997c9d3552af4f254 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 7.
I20260809 15:47:11.942174 28330 raft_consensus.cc:2781] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Leader election won for term 5
I20260809 15:47:11.998874 28330 raft_consensus.cc:686] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P c0f38c1908414566aa9115bb2cbdd85d [term 5 LEADER]: Becoming Leader. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:47:11.999843 23468 leader_election.cc:302] T e14d118b2eaa42aa8a7cd4025aba7128 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 61e69695258f48baa984c9d395e8ce3f, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:47:11.999753 28330 consensus_queue.cc:237] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:11.964057 28230 consensus_queue.cc:260] T 7660dfa2ef694927b8eda4100a2bcd9b P c0f38c1908414566aa9115bb2cbdd85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
W20260809 15:47:12.003082 23223 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.68:34689, user_credentials={real_user=slave}} blocked reactor thread for 219579us
I20260809 15:47:11.963284 23273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0741ab44d99439abc3aa3bdd27ee160" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:12.003861 23273 raft_consensus.cc:2454] T c0741ab44d99439abc3aa3bdd27ee160 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 6.
I20260809 15:47:12.005357 23470 leader_election.cc:302] T c0741ab44d99439abc3aa3bdd27ee160 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:47:11.996702 23116 catalog_manager.cc:5495] T 6ae14a74a26f434db08c652e097385c8 P 67b3aeadfa7a432abe8af4d292e04ab9 reported cstate change: term changed from 4 to 5, leader changed from 65c36d164b0540f099d592f38307ee6c (127.20.202.67) to 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65). New cstate: current_term: 5 leader_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } health_report { overall_health: UNKNOWN } } }
W20260809 15:47:12.010643 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 64574.7us
I20260809 15:47:12.012269 23404 raft_consensus.cc:2454] T 538a228ddfb54bc6a36528d5ff07597b P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 7.
I20260809 15:47:11.991233 28198 raft_consensus.cc:3032] T d2d4f749fcfd4fb3bfb8115af3471598 P 65c36d164b0540f099d592f38307ee6c [term 6 LEADER]: Stepping down as leader of term 6
I20260809 15:47:11.980370 27792 consensus_queue.cc:260] T 4b30f28311964a0b9149a50495e83d8f P 61e69695258f48baa984c9d395e8ce3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } attrs { replace: true } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } }
I20260809 15:47:12.014164 23522 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "df20671a7b874dbb98c0ef28371c1308"
dest_uuid: "65c36d164b0540f099d592f38307ee6c"
mode: GRACEFUL
new_leader_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
from {username='slave'} at 127.0.0.1:48888
I20260809 15:47:12.014483 23228 leader_election.cc:302] T 3d880de657e544468c82157769622a26 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 61e69695258f48baa984c9d395e8ce3f
I20260809 15:47:12.014628 23116 catalog_manager.cc:5495] T 73f6f44c259547f1b05a3a434616fa54 P 67b3aeadfa7a432abe8af4d292e04ab9 reported cstate change: term changed from 1 to 5, leader changed from 65c36d164b0540f099d592f38307ee6c (127.20.202.67) to c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66). New cstate: current_term: 5 leader_uuid: "c0f38c1908414566aa9115bb2cbdd85d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } }
W20260809 15:47:11.981359 28310 log.cc:924] Time spent T 297c82ed906d4b48b184985eed3de601 P 67b3aeadfa7a432abe8af4d292e04ab9: Append to log took a long time: real 0.200s user 0.001s sys 0.000s
W20260809 15:47:12.017206 24410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 137196us
I20260809 15:47:11.962353 28294 leader_election.cc:288] T c0741ab44d99439abc3aa3bdd27ee160 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:11.980793 24455 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0741ab44d99439abc3aa3bdd27ee160" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:47:12.010469 27977 raft_consensus.cc:2726] T e14d118b2eaa42aa8a7cd4025aba7128 P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260809 15:47:12.014127 28246 raft_consensus.cc:2781] T c0741ab44d99439abc3aa3bdd27ee160 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260809 15:47:12.052940 28246 raft_consensus.cc:482] T c0741ab44d99439abc3aa3bdd27ee160 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:47:12.053718 23522 raft_consensus.cc:595] T df20671a7b874dbb98c0ef28371c1308 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Received request to transfer leadership to TS c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:12.054098 23522 raft_consensus.cc:603] T df20671a7b874dbb98c0ef28371c1308 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260809 15:47:12.053328 24455 raft_consensus.cc:2454] T c0741ab44d99439abc3aa3bdd27ee160 P 61e69695258f48baa984c9d395e8ce3f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 7.
I20260809 15:47:12.026435 28230 raft_consensus.cc:3037] T 7660dfa2ef694927b8eda4100a2bcd9b P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Advancing to term 6
I20260809 15:47:12.056403 23116 catalog_manager.cc:5495] T 0da4d5283c8a46878bd39b85e5c71f66 P 67b3aeadfa7a432abe8af4d292e04ab9 reported cstate change: term changed from 3 to 5, leader changed from 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65) to 65c36d164b0540f099d592f38307ee6c (127.20.202.67). New cstate: current_term: 5 leader_uuid: "65c36d164b0540f099d592f38307ee6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } }
I20260809 15:47:12.058573 27792 ts_tablet_manager.cc:1428] T 4b30f28311964a0b9149a50495e83d8f P 61e69695258f48baa984c9d395e8ce3f: Time spent starting tablet: real 0.786s user 0.003s sys 0.006s
I20260809 15:47:12.028241 23345 leader_election.cc:302] T 0a51c0baaa6e432997c9d3552af4f254 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 61e69695258f48baa984c9d395e8ce3f, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:47:12.030705 27493 raft_consensus.cc:3032] T ae6d2c08a7a94f58ac8dd698e3fd456c P c0f38c1908414566aa9115bb2cbdd85d [term 7 LEADER]: Stepping down as leader of term 7
I20260809 15:47:12.032248 28290 raft_consensus.cc:3032] T 563daf42d7784db195ea99c3cb21b370 P 61e69695258f48baa984c9d395e8ce3f [term 5 LEADER]: Stepping down as leader of term 5
I20260809 15:47:12.038801 24451 raft_consensus.cc:1261] T 16ca412a0dae4feebe85dd01bf1a9173 P 61e69695258f48baa984c9d395e8ce3f [term 6 FOLLOWER]: Refusing update from remote peer 67b3aeadfa7a432abe8af4d292e04ab9: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260809 15:47:12.002296 28309 raft_consensus.cc:504] T 472816994d814cb198b69e5248697440 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:11.998178 28307 consensus_queue.cc:260] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 61e69695258f48baa984c9d395e8ce3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
W20260809 15:47:12.058923 27792 ts_tablet_manager.cc:1460] T 4b30f28311964a0b9149a50495e83d8f P 61e69695258f48baa984c9d395e8ce3f: Tablet startup took 786ms
I20260809 15:47:12.152177 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "907bacfa79d04a55a68c8dbe60056462" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:47:12.155690 23402 raft_consensus.cc:3037] T 907bacfa79d04a55a68c8dbe60056462 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Advancing to term 8
I20260809 15:47:12.074662 23116 catalog_manager.cc:5495] T 0c48a3049897452383257aa757834fb5 P 67b3aeadfa7a432abe8af4d292e04ab9 reported cstate change: term changed from 1 to 3, leader changed from 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65) to 65c36d164b0540f099d592f38307ee6c (127.20.202.67). New cstate: current_term: 3 leader_uuid: "65c36d164b0540f099d592f38307ee6c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } } }
I20260809 15:47:12.157511 23116 catalog_manager.cc:5495] T 763e58d6ba644eea830a6c1664d39c9b P 67b3aeadfa7a432abe8af4d292e04ab9 reported cstate change: term changed from 2 to 5. New cstate: current_term: 5 leader_uuid: "65c36d164b0540f099d592f38307ee6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } }
I20260809 15:47:12.159755 23116 catalog_manager.cc:5495] T 28e17dcb5efe45b8b6a8c42da03b4819 P 67b3aeadfa7a432abe8af4d292e04ab9 reported cstate change: term changed from 2 to 6, leader changed from 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65) to c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66). New cstate: current_term: 6 leader_uuid: "c0f38c1908414566aa9115bb2cbdd85d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } }
I20260809 15:47:12.063036 28326 raft_consensus.cc:2781] T 3d880de657e544468c82157769622a26 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader election won for term 4
I20260809 15:47:12.164197 28309 leader_election.cc:288] T 472816994d814cb198b69e5248697440 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 6 election: Requested vote from peers 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:12.164736 28326 raft_consensus.cc:686] T 3d880de657e544468c82157769622a26 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 LEADER]: Becoming Leader. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Running, Role: LEADER
W20260809 15:47:12.165695 23345 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.68:34689, user_credentials={real_user=slave}} blocked reactor thread for 137543us
I20260809 15:47:12.059064 28198 raft_consensus.cc:729] T d2d4f749fcfd4fb3bfb8115af3471598 P 65c36d164b0540f099d592f38307ee6c [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:47:12.165948 28326 consensus_queue.cc:237] T 3d880de657e544468c82157769622a26 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:12.045864 28168 raft_consensus.cc:3032] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 LEADER]: Stepping down as leader of term 7
I20260809 15:47:12.167006 28198 consensus_queue.cc:260] T d2d4f749fcfd4fb3bfb8115af3471598 P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:12.168246 28198 raft_consensus.cc:3037] T d2d4f749fcfd4fb3bfb8115af3471598 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Advancing to term 7
I20260809 15:47:12.071918 24463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5712f6b6fffc4cc5a8ddfbe63e4097df" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:47:12.014108 28315 raft_consensus.cc:2781] T 7baa2492df0340c4a5ac241959ce1ac3 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Leader election won for term 5
I20260809 15:47:12.169493 28315 raft_consensus.cc:686] T 7baa2492df0340c4a5ac241959ce1ac3 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 LEADER]: Becoming Leader. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Running, Role: LEADER
I20260809 15:47:12.170454 28315 consensus_queue.cc:237] T 7baa2492df0340c4a5ac241959ce1ac3 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:12.064013 28244 raft_consensus.cc:3037] T 0acc47ce21c3422698d1ef7277c3ed63 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Advancing to term 8
I20260809 15:47:12.162460 23402 raft_consensus.cc:2454] T 907bacfa79d04a55a68c8dbe60056462 P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 8.
I20260809 15:47:12.053370 28246 raft_consensus.cc:3037] T c0741ab44d99439abc3aa3bdd27ee160 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Advancing to term 8
I20260809 15:47:12.178145 27493 raft_consensus.cc:729] T ae6d2c08a7a94f58ac8dd698e3fd456c P c0f38c1908414566aa9115bb2cbdd85d [term 7 LEADER]: Becoming Follower/Learner. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:47:12.179068 27493 consensus_queue.cc:260] T ae6d2c08a7a94f58ac8dd698e3fd456c P c0f38c1908414566aa9115bb2cbdd85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:12.180491 27493 raft_consensus.cc:3037] T ae6d2c08a7a94f58ac8dd698e3fd456c P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Advancing to term 8
I20260809 15:47:12.181391 28230 raft_consensus.cc:2726] T 0a51c0baaa6e432997c9d3552af4f254 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260809 15:47:12.185648 23394 raft_consensus.cc:1226] T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d [term 9 FOLLOWER]: Rejecting Update request from peer 61e69695258f48baa984c9d395e8ce3f for earlier term 8. Current term is 9. Ops: []
I20260809 15:47:12.187750 23402 raft_consensus.cc:1226] T 0a51c0baaa6e432997c9d3552af4f254 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Rejecting Update request from peer 61e69695258f48baa984c9d395e8ce3f for earlier term 6. Current term is 7. Ops: []
I20260809 15:47:12.188100 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16ca412a0dae4feebe85dd01bf1a9173" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:12.188786 23526 raft_consensus.cc:2454] T 16ca412a0dae4feebe85dd01bf1a9173 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 6.
I20260809 15:47:12.190383 23519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "294d4bcc1e7a477cb56455581277a60f" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:12.192086 23394 raft_consensus.cc:1226] T 563daf42d7784db195ea99c3cb21b370 P c0f38c1908414566aa9115bb2cbdd85d [term 6 LEADER]: Rejecting Update request from peer 61e69695258f48baa984c9d395e8ce3f for earlier term 5. Current term is 6. Ops: []
I20260809 15:47:12.194391 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c48a3049897452383257aa757834fb5" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:47:12.194998 23526 raft_consensus.cc:3037] T 0c48a3049897452383257aa757834fb5 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Advancing to term 6
I20260809 15:47:12.196107 23393 raft_consensus.cc:2454] T 7baa2492df0340c4a5ac241959ce1ac3 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 5.
I20260809 15:47:12.164795 28290 raft_consensus.cc:729] T 563daf42d7784db195ea99c3cb21b370 P 61e69695258f48baa984c9d395e8ce3f [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 61e69695258f48baa984c9d395e8ce3f, State: Running, Role: LEADER
W20260809 15:47:12.146526 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 132113us
I20260809 15:47:12.208369 23400 raft_consensus.cc:2454] T 4bcac4b24e7a4448abdc934bba5a624e P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 4.
I20260809 15:47:12.198721 23282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "907bacfa79d04a55a68c8dbe60056462" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:12.212625 28313 consensus_queue.cc:1035] T 16ca412a0dae4feebe85dd01bf1a9173 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260809 15:47:12.213984 28313 consensus_queue.cc:1046] T 05ad7d8fdca64e3da527280ca5be9280 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260809 15:47:12.167439 28168 raft_consensus.cc:729] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Running, Role: LEADER
I20260809 15:47:12.203459 23286 raft_consensus.cc:1261] T 563daf42d7784db195ea99c3cb21b370 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260809 15:47:12.194644 28274 raft_consensus.cc:482] T 953eea268e17491ca76f3a555860a512 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
I20260809 15:47:12.206302 23277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4564dfc6d4094c1b9240b367ccecf3b1" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:12.206563 23274 raft_consensus.cc:3032] T 05ad7d8fdca64e3da527280ca5be9280 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 LEADER]: Stepping down as leader of term 6
I20260809 15:47:12.223471 23283 raft_consensus.cc:1261] T 998c6cee72cf40228c6dba064c0db2da P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260809 15:47:12.218279 23274 raft_consensus.cc:729] T 05ad7d8fdca64e3da527280ca5be9280 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Running, Role: LEADER
I20260809 15:47:12.226491 23276 raft_consensus.cc:1226] T d2d4f749fcfd4fb3bfb8115af3471598 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 6. Current term is 7. Ops: []
I20260809 15:47:12.226867 23273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2424fd9bd7f14e508fd106cbfda786a4" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:12.231834 23526 raft_consensus.cc:2454] T 0c48a3049897452383257aa757834fb5 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 6.
I20260809 15:47:12.235193 23522 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f9e7cadcd24d45fcbfa84209225e8f2d"
dest_uuid: "65c36d164b0540f099d592f38307ee6c"
mode: GRACEFUL
new_leader_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
from {username='slave'} at 127.0.0.1:48888
I20260809 15:47:12.235750 23522 raft_consensus.cc:595] T f9e7cadcd24d45fcbfa84209225e8f2d P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Received request to transfer leadership to TS c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:12.236088 23522 raft_consensus.cc:603] T f9e7cadcd24d45fcbfa84209225e8f2d P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260809 15:47:12.066195 28171 leader_election.cc:288] T 294d4bcc1e7a477cb56455581277a60f P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:12.239681 23345 leader_election.cc:302] T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c
I20260809 15:47:12.244280 23345 leader_election.cc:302] T e71a05ba1d4d42bd95c3f85f9313399e P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 61e69695258f48baa984c9d395e8ce3f, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:47:12.238407 23274 consensus_queue.cc:260] T 05ad7d8fdca64e3da527280ca5be9280 P 67b3aeadfa7a432abe8af4d292e04ab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:12.245402 23274 raft_consensus.cc:3037] T 05ad7d8fdca64e3da527280ca5be9280 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Advancing to term 7
I20260809 15:47:12.246279 21289 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260809 15:47:12.248062 21289 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 15:47:12.248699 21289 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 15:47:12.248760 23519 raft_consensus.cc:2454] T 294d4bcc1e7a477cb56455581277a60f P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 6.
I20260809 15:47:12.164597 28131 raft_consensus.cc:504] T d0721a4403a440f383318a3cea9d1442 P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:12.270969 23353 leader_election.cc:302] T df20671a7b874dbb98c0ef28371c1308 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:47:12.272387 24410 leader_election.cc:302] T 7660dfa2ef694927b8eda4100a2bcd9b P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:12.217480 28274 raft_consensus.cc:504] T 953eea268e17491ca76f3a555860a512 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:12.281322 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4564dfc6d4094c1b9240b367ccecf3b1" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:12.282128 23526 raft_consensus.cc:2396] T 4564dfc6d4094c1b9240b367ccecf3b1 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d for term 8 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20260809 15:47:12.278617 23345 leader_election.cc:302] T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c
I20260809 15:47:12.285249 28131 leader_election.cc:288] T d0721a4403a440f383318a3cea9d1442 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 8 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:12.247084 28230 raft_consensus.cc:2726] T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260809 15:47:12.286952 23528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2424fd9bd7f14e508fd106cbfda786a4" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:12.250648 28290 consensus_queue.cc:260] T 563daf42d7784db195ea99c3cb21b370 P 61e69695258f48baa984c9d395e8ce3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:12.288169 28290 raft_consensus.cc:3037] T 563daf42d7784db195ea99c3cb21b370 P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Advancing to term 6
I20260809 15:47:12.290254 23273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0721a4403a440f383318a3cea9d1442" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:47:12.290861 28274 leader_election.cc:288] T 953eea268e17491ca76f3a555860a512 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
W20260809 15:47:12.291467 24410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 50488.8us
I20260809 15:47:12.292214 24410 leader_election.cc:302] T 4bcac4b24e7a4448abdc934bba5a624e P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c; no voters:
I20260809 15:47:12.290807 23273 raft_consensus.cc:3037] T d0721a4403a440f383318a3cea9d1442 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Advancing to term 8
I20260809 15:47:12.293006 24410 leader_election.cc:302] T 16ca412a0dae4feebe85dd01bf1a9173 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c; no voters:
W20260809 15:47:12.278083 27956 consensus_meta.cc:287] T 0c48a3049897452383257aa757834fb5 P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent flushing consensus metadata: real 0.635s user 0.009s sys 0.005s
I20260809 15:47:12.293769 24410 leader_election.cc:302] T 0c48a3049897452383257aa757834fb5 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c; no voters:
I20260809 15:47:12.299361 23400 raft_consensus.cc:1226] T 472816994d814cb198b69e5248697440 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Rejecting Update request from peer 61e69695258f48baa984c9d395e8ce3f for earlier term 5. Current term is 6. Ops: []
I20260809 15:47:12.233057 28229 consensus_queue.cc:1035] T 998c6cee72cf40228c6dba064c0db2da P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.017s
I20260809 15:47:12.300823 23521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472816994d814cb198b69e5248697440" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:47:12.302772 23521 raft_consensus.cc:3037] T 472816994d814cb198b69e5248697440 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Advancing to term 6
I20260809 15:47:12.302896 24410 leader_election.cc:302] T 907bacfa79d04a55a68c8dbe60056462 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d; no voters:
W20260809 15:47:12.155297 27792 ts_tablet_manager.cc:1462] T 4b30f28311964a0b9149a50495e83d8f P 61e69695258f48baa984c9d395e8ce3f: Trace:
0809 15:47:01.704353 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:01.727545 (+ 23192us) service_pool.cc:227] Handling call
0809 15:47:01.926205 (+198660us) ts_tablet_manager.cc:928] T 4b30f28311964a0b9149a50495e83d8f P 61e69695258f48baa984c9d395e8ce3f: Initiating tablet copy from peer c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
0809 15:47:02.069603 (+143398us) tablet_copy_client.cc:361] Tablet copy session begun
0809 15:47:02.759991 (+690388us) tablet_metadata.cc:641] Metadata flushed
0809 15:47:02.760295 (+ 304us) tablet_copy_client.cc:490] Wrote new tablet metadata
0809 15:47:02.878867 (+118572us) tablet_copy_client.cc:411] Wrote new consensus metadata
0809 15:47:02.939415 (+ 60548us) tablet_replica.cc:205] Creating consensus instance
0809 15:47:03.043438 (+104023us) ts_tablet_manager.cc:983] Replica registered.
0809 15:47:03.047884 (+ 4446us) inbound_call.cc:173] Queueing success response
0809 15:47:08.008539 (+4960655us) ts_tablet_manager.cc:1373] Bootstrapping tablet
0809 15:47:11.272985 (+3264446us) ts_tablet_manager.cc:1429] Starting tablet replica
0809 15:47:11.512537 (+239552us) tablet_replica.cc:259] Starting instrumentation
0809 15:47:11.644691 (+132154us) tablet_replica.cc:278] Starting consensus
Metrics: {"dns_us":277,"raft.queue_time_us":2305,"raft.run_cpu_time_us":140,"raft.run_wall_time_us":141,"tablet-copy.queue_time_us":138364,"tablet-download-pool.queue_time_us":20126,"tablet-download-pool.run_cpu_time_us":1212,"tablet-download-pool.run_wall_time_us":3242515,"thread_start_us":94364,"threads_started":2}
I20260809 15:47:12.303767 23274 raft_consensus.cc:1261] T 05ad7d8fdca64e3da527280ca5be9280 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260809 15:47:12.307751 23280 raft_consensus.cc:1261] T ba56725e42f74a1baa4c04d9660727f9 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260809 15:47:12.308102 23520 raft_consensus.cc:1261] T 998c6cee72cf40228c6dba064c0db2da P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260809 15:47:12.302546 23273 raft_consensus.cc:2454] T d0721a4403a440f383318a3cea9d1442 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 8.
I20260809 15:47:12.304896 23345 leader_election.cc:302] T 8e4ffcf83c234ae7910ddef3ec0b4bda P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 61e69695258f48baa984c9d395e8ce3f, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:47:12.218209 28168 consensus_queue.cc:260] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:12.303674 23353 leader_election.cc:302] T 2424fd9bd7f14e508fd106cbfda786a4 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:47:12.305462 27988 raft_consensus.cc:482] T 763e58d6ba644eea830a6c1664d39c9b P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:47:12.310101 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0721a4403a440f383318a3cea9d1442" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:47:12.312614 23527 raft_consensus.cc:3037] T d0721a4403a440f383318a3cea9d1442 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Advancing to term 8
I20260809 15:47:12.310359 23276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "953eea268e17491ca76f3a555860a512" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:12.311527 23351 leader_election.cc:302] T d0721a4403a440f383318a3cea9d1442 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:47:12.274906 28277 raft_consensus.cc:2726] T e71a05ba1d4d42bd95c3f85f9313399e P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260809 15:47:12.299615 28307 raft_consensus.cc:3037] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Advancing to term 5
I20260809 15:47:12.274326 28293 raft_consensus.cc:2726] T 4b30f28311964a0b9149a50495e83d8f P 65c36d164b0540f099d592f38307ee6c [term 5 LEADER]: Leader election lost for term 4. Reason: could not achieve majority
I20260809 15:47:12.307096 28209 raft_consensus.cc:2781] T 0c48a3049897452383257aa757834fb5 P 61e69695258f48baa984c9d395e8ce3f [term 6 FOLLOWER]: Leader election won for term 6
I20260809 15:47:12.310230 28255 raft_consensus.cc:2781] T 907bacfa79d04a55a68c8dbe60056462 P 61e69695258f48baa984c9d395e8ce3f [term 8 FOLLOWER]: Leader election won for term 8
I20260809 15:47:12.324941 23521 raft_consensus.cc:2454] T 472816994d814cb198b69e5248697440 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 6.
I20260809 15:47:12.313097 28230 raft_consensus.cc:2726] T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260809 15:47:12.325673 23528 raft_consensus.cc:1226] T 472816994d814cb198b69e5248697440 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Rejecting Update request from peer 61e69695258f48baa984c9d395e8ce3f for earlier term 5. Current term is 6. Ops: []
W20260809 15:47:12.326182 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 116198us
I20260809 15:47:12.319934 27751 raft_consensus.cc:2726] T 8e4ffcf83c234ae7910ddef3ec0b4bda P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260809 15:47:12.306535 28171 raft_consensus.cc:2781] T 4bcac4b24e7a4448abdc934bba5a624e P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Leader election won for term 4
W20260809 15:47:12.328979 23472 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 548846us
I20260809 15:47:12.320331 28168 raft_consensus.cc:3037] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Advancing to term 8
I20260809 15:47:12.316578 28280 consensus_queue.cc:1046] T 0a51c0baaa6e432997c9d3552af4f254 P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: INVALID_TERM, Last received: 6.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.012s
I20260809 15:47:12.293776 27956 raft_consensus.cc:504] T 0c48a3049897452383257aa757834fb5 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:12.317996 23353 leader_election.cc:302] T 4564dfc6d4094c1b9240b367ccecf3b1 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:47:12.346217 28248 raft_consensus.cc:2726] T 4564dfc6d4094c1b9240b367ccecf3b1 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260809 15:47:12.346577 27956 leader_election.cc:288] T 0c48a3049897452383257aa757834fb5 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 6 election: Requested vote from peers 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:12.346804 28248 consensus_queue.cc:1035] T 998c6cee72cf40228c6dba064c0db2da P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260809 15:47:12.319849 27957 raft_consensus.cc:2726] T 2424fd9bd7f14e508fd106cbfda786a4 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260809 15:47:12.317270 28131 consensus_queue.cc:1035] T 05ad7d8fdca64e3da527280ca5be9280 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.014s
I20260809 15:47:12.349215 28280 raft_consensus.cc:3032] T 0a51c0baaa6e432997c9d3552af4f254 P 61e69695258f48baa984c9d395e8ce3f [term 6 LEADER]: Stepping down as leader of term 6
I20260809 15:47:12.349601 28280 raft_consensus.cc:729] T 0a51c0baaa6e432997c9d3552af4f254 P 61e69695258f48baa984c9d395e8ce3f [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 61e69695258f48baa984c9d395e8ce3f, State: Running, Role: LEADER
I20260809 15:47:12.350432 28280 consensus_queue.cc:260] T 0a51c0baaa6e432997c9d3552af4f254 P 61e69695258f48baa984c9d395e8ce3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:12.353523 28302 raft_consensus.cc:2726] T 7660dfa2ef694927b8eda4100a2bcd9b P 61e69695258f48baa984c9d395e8ce3f [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260809 15:47:12.354264 28316 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 7 current term: 7 is higher.
I20260809 15:47:12.333835 28325 consensus_queue.cc:1035] T 563daf42d7784db195ea99c3cb21b370 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.118s
I20260809 15:47:12.323184 28203 consensus_queue.cc:1046] T 472816994d814cb198b69e5248697440 P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.013s
I20260809 15:47:12.340119 28353 raft_consensus.cc:2781] T d0721a4403a440f383318a3cea9d1442 P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Leader election won for term 8
I20260809 15:47:12.340973 27751 raft_consensus.cc:482] T 73f6f44c259547f1b05a3a434616fa54 P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:12.343905 23424 tablet_copy_service.cc:342] P c0f38c1908414566aa9115bb2cbdd85d: Request end of tablet copy session 61e69695258f48baa984c9d395e8ce3f-4b30f28311964a0b9149a50495e83d8f received from {username='slave'} at 127.0.0.1:52856
I20260809 15:47:12.356132 28171 raft_consensus.cc:686] T 4bcac4b24e7a4448abdc934bba5a624e P 61e69695258f48baa984c9d395e8ce3f [term 4 LEADER]: Becoming Leader. State: Replica: 61e69695258f48baa984c9d395e8ce3f, State: Running, Role: LEADER
I20260809 15:47:12.356530 23424 tablet_copy_service.cc:433] P c0f38c1908414566aa9115bb2cbdd85d: ending tablet copy session 61e69695258f48baa984c9d395e8ce3f-4b30f28311964a0b9149a50495e83d8f on tablet 4b30f28311964a0b9149a50495e83d8f with peer 61e69695258f48baa984c9d395e8ce3f
I20260809 15:47:12.344300 28319 consensus_queue.cc:1046] T 4564dfc6d4094c1b9240b367ccecf3b1 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 15:47:12.346520 23353 leader_election.cc:302] T 472816994d814cb198b69e5248697440 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:47:12.356734 27751 raft_consensus.cc:504] T 73f6f44c259547f1b05a3a434616fa54 P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:12.357025 28171 consensus_queue.cc:237] T 4bcac4b24e7a4448abdc934bba5a624e P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:12.320160 27988 raft_consensus.cc:504] T 763e58d6ba644eea830a6c1664d39c9b P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:12.358726 27751 leader_election.cc:288] T 73f6f44c259547f1b05a3a434616fa54 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:12.315546 23345 leader_election.cc:302] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c
I20260809 15:47:12.356036 28353 raft_consensus.cc:686] T d0721a4403a440f383318a3cea9d1442 P c0f38c1908414566aa9115bb2cbdd85d [term 8 LEADER]: Becoming Leader. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:47:12.356038 28203 raft_consensus.cc:3032] T 472816994d814cb198b69e5248697440 P 61e69695258f48baa984c9d395e8ce3f [term 5 LEADER]: Stepping down as leader of term 5
I20260809 15:47:12.360796 28203 raft_consensus.cc:729] T 472816994d814cb198b69e5248697440 P 61e69695258f48baa984c9d395e8ce3f [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 61e69695258f48baa984c9d395e8ce3f, State: Running, Role: LEADER
I20260809 15:47:12.361593 28203 consensus_queue.cc:260] T 472816994d814cb198b69e5248697440 P 61e69695258f48baa984c9d395e8ce3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:12.359447 27988 leader_election.cc:288] T 763e58d6ba644eea830a6c1664d39c9b P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:12.363149 27988 raft_consensus.cc:2726] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260809 15:47:12.360121 23345 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.68:34689, user_credentials={real_user=slave}} blocked reactor thread for 44647.5us
I20260809 15:47:12.360334 28353 consensus_queue.cc:237] T d0721a4403a440f383318a3cea9d1442 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:12.363452 28246 raft_consensus.cc:504] T c0741ab44d99439abc3aa3bdd27ee160 P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:12.365089 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73f6f44c259547f1b05a3a434616fa54" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:12.365402 28246 leader_election.cc:288] T c0741ab44d99439abc3aa3bdd27ee160 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 8 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:12.365866 23520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "763e58d6ba644eea830a6c1664d39c9b" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:12.367025 23345 leader_election.cc:302] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c
I20260809 15:47:12.368072 23345 leader_election.cc:302] T 3296ef50be254fc29cfc77d22a8f7a23 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 61e69695258f48baa984c9d395e8ce3f, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:47:12.368755 27751 raft_consensus.cc:2726] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260809 15:47:12.369432 24406 leader_election.cc:334] T 0c48a3049897452383257aa757834fb5 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459): Timed out: RequestConsensusVote RPC to 127.20.202.65:35459 timed out after 1.830s (SENT)
I20260809 15:47:12.370477 23404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99bdb93503db4de4a1cfb0161ea0c07c" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:12.370923 28197 consensus_queue.cc:1035] T ba56725e42f74a1baa4c04d9660727f9 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.052s
I20260809 15:47:12.372624 28337 consensus_queue.cc:1046] T 3924b648461e463b9d7925e5671d68ae P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: INVALID_TERM, Last received: 8.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.059s
I20260809 15:47:12.370875 23283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "763e58d6ba644eea830a6c1664d39c9b" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:12.362820 28203 raft_consensus.cc:3037] T 472816994d814cb198b69e5248697440 P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Advancing to term 6
I20260809 15:47:12.378296 23404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df20671a7b874dbb98c0ef28371c1308" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:12.379602 23280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0741ab44d99439abc3aa3bdd27ee160" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:47:12.379976 23353 leader_election.cc:302] T 763e58d6ba644eea830a6c1664d39c9b P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:47:12.380436 23280 raft_consensus.cc:3037] T c0741ab44d99439abc3aa3bdd27ee160 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Advancing to term 8
I20260809 15:47:12.375334 27751 raft_consensus.cc:2726] T 3296ef50be254fc29cfc77d22a8f7a23 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 15:47:12.380844 23527 raft_consensus.cc:2454] T d0721a4403a440f383318a3cea9d1442 P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 8.
I20260809 15:47:12.373689 28337 raft_consensus.cc:3032] T 3924b648461e463b9d7925e5671d68ae P 61e69695258f48baa984c9d395e8ce3f [term 8 LEADER]: Stepping down as leader of term 8
I20260809 15:47:12.373828 23273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73f6f44c259547f1b05a3a434616fa54" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:12.397395 23404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3aa6a9a52804df0ba9b0ca3c82f38c3" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
W20260809 15:47:12.384477 23472 leader_election.cc:334] T 3924b648461e463b9d7925e5671d68ae P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263): Timed out: RequestConsensusVote RPC to 127.20.202.66:36263 timed out after 1.572s (SENT)
I20260809 15:47:12.392028 23393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00a1881267a14ba8b78c81040f17a014" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:12.399638 23472 leader_election.cc:302] T 3924b648461e463b9d7925e5671d68ae P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:12.400224 23393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472816994d814cb198b69e5248697440" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:12.392364 23394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cef3e7277d7d4760824a3436ed4a847d" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:12.369498 24448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c48a3049897452383257aa757834fb5" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f"
I20260809 15:47:12.398007 23404 raft_consensus.cc:2454] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 7.
I20260809 15:47:12.379498 28335 raft_consensus.cc:2781] T 16ca412a0dae4feebe85dd01bf1a9173 P 61e69695258f48baa984c9d395e8ce3f [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260809 15:47:12.408806 23404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "953eea268e17491ca76f3a555860a512" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:12.398619 23351 leader_election.cc:302] T 73f6f44c259547f1b05a3a434616fa54 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:47:12.411502 23395 raft_consensus.cc:1226] T 213543cdbe814c89a17a963469922c1b P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 3. Current term is 4. Ops: []
I20260809 15:47:12.411767 23393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30c1740ef6874df0a332fbd6fd4b03ce" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:47:12.412298 23400 raft_consensus.cc:1261] T 7660dfa2ef694927b8eda4100a2bcd9b P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260809 15:47:12.445304 23393 raft_consensus.cc:2379] T 30c1740ef6874df0a332fbd6fd4b03ce P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c in current term 8: Already voted for candidate c0f38c1908414566aa9115bb2cbdd85d in this term.
I20260809 15:47:12.448151 23392 raft_consensus.cc:1226] T 0c29937a80724ddb9d07a27534a146f3 P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 7. Current term is 8. Ops: [7.3-7.3]
I20260809 15:47:12.359211 27493 raft_consensus.cc:2781] T 472816994d814cb198b69e5248697440 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Leader election won for term 6
W20260809 15:47:12.451164 23472 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 66707.1us
I20260809 15:47:12.417918 24410 leader_election.cc:302] T 294d4bcc1e7a477cb56455581277a60f P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c; no voters:
I20260809 15:47:12.355983 28280 raft_consensus.cc:3037] T 0a51c0baaa6e432997c9d3552af4f254 P 61e69695258f48baa984c9d395e8ce3f [term 6 FOLLOWER]: Advancing to term 7
I20260809 15:47:12.437536 23404 raft_consensus.cc:2379] T 953eea268e17491ca76f3a555860a512 P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c in current term 8: Already voted for candidate c0f38c1908414566aa9115bb2cbdd85d in this term.
I20260809 15:47:12.396413 28337 raft_consensus.cc:729] T 3924b648461e463b9d7925e5671d68ae P 61e69695258f48baa984c9d395e8ce3f [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 61e69695258f48baa984c9d395e8ce3f, State: Running, Role: LEADER
I20260809 15:47:12.458688 23395 raft_consensus.cc:1226] T 0da4d5283c8a46878bd39b85e5c71f66 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 5. Current term is 6. Ops: []
I20260809 15:47:12.460718 23400 raft_consensus.cc:1261] T 4b30f28311964a0b9149a50495e83d8f P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260809 15:47:12.461891 23392 raft_consensus.cc:1226] T ba56725e42f74a1baa4c04d9660727f9 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 5. Current term is 6. Ops: []
I20260809 15:47:12.411368 27751 raft_consensus.cc:482] T 2db057fd10e149a9895dff6ef8c9f038 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:12.396908 28230 raft_consensus.cc:2726] T 763e58d6ba644eea830a6c1664d39c9b P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260809 15:47:12.459806 27493 raft_consensus.cc:686] T 472816994d814cb198b69e5248697440 P c0f38c1908414566aa9115bb2cbdd85d [term 6 LEADER]: Becoming Leader. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:47:12.418277 28335 raft_consensus.cc:482] T 16ca412a0dae4feebe85dd01bf1a9173 P 61e69695258f48baa984c9d395e8ce3f [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:47:12.465408 28290 raft_consensus.cc:2781] T 294d4bcc1e7a477cb56455581277a60f P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260809 15:47:12.470749 24468 raft_consensus.cc:1226] T 7660dfa2ef694927b8eda4100a2bcd9b P 61e69695258f48baa984c9d395e8ce3f [term 6 FOLLOWER]: Rejecting Update request from peer c0f38c1908414566aa9115bb2cbdd85d for earlier term 5. Current term is 6. Ops: []
W20260809 15:47:12.471385 23468 consensus_peers.cc:487] T ba56725e42f74a1baa4c04d9660727f9 P 65c36d164b0540f099d592f38307ee6c -> Peer 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689): Couldn't send request to peer 61e69695258f48baa984c9d395e8ce3f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ba56725e42f74a1baa4c04d9660727f9. This is attempt 1: this message will repeat every 5th retry.
I20260809 15:47:12.456670 28220 raft_consensus.cc:2726] T 3924b648461e463b9d7925e5671d68ae P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260809 15:47:12.447862 28248 raft_consensus.cc:2726] T 73f6f44c259547f1b05a3a434616fa54 P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260809 15:47:12.476624 23468 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.68:34689, user_credentials={real_user=slave}} blocked reactor thread for 64568.2us
I20260809 15:47:12.463830 28319 raft_consensus.cc:3032] T 4564dfc6d4094c1b9240b367ccecf3b1 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 LEADER]: Stepping down as leader of term 6
I20260809 15:47:12.475841 28335 raft_consensus.cc:3037] T 16ca412a0dae4feebe85dd01bf1a9173 P 61e69695258f48baa984c9d395e8ce3f [term 6 FOLLOWER]: Advancing to term 7
W20260809 15:47:12.526955 23472 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 51407.7us
I20260809 15:47:12.530421 24468 raft_consensus.cc:3037] T 4b30f28311964a0b9149a50495e83d8f P 61e69695258f48baa984c9d395e8ce3f [term 3 LEARNER]: Advancing to term 5
W20260809 15:47:12.530870 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 196403us
I20260809 15:47:12.476451 28290 raft_consensus.cc:482] T 294d4bcc1e7a477cb56455581277a60f P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Starting leader election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:47:12.543010 24472 raft_consensus.cc:1226] T d3e94b7e50754a81b3d72a07476cb7b9 P 61e69695258f48baa984c9d395e8ce3f [term 8 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 7. Current term is 8. Ops: []
I20260809 15:47:12.548813 23280 raft_consensus.cc:2454] T c0741ab44d99439abc3aa3bdd27ee160 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 8.
I20260809 15:47:12.471792 27956 consensus_queue.cc:1035] T 16ca412a0dae4feebe85dd01bf1a9173 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.054s
I20260809 15:47:12.523288 28319 raft_consensus.cc:729] T 4564dfc6d4094c1b9240b367ccecf3b1 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Running, Role: LEADER
I20260809 15:47:12.553736 23470 leader_election.cc:302] T c0741ab44d99439abc3aa3bdd27ee160 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:47:12.476466 27751 raft_consensus.cc:504] T 2db057fd10e149a9895dff6ef8c9f038 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:12.560592 28209 raft_consensus.cc:686] T 0c48a3049897452383257aa757834fb5 P 61e69695258f48baa984c9d395e8ce3f [term 6 LEADER]: Becoming Leader. State: Replica: 61e69695258f48baa984c9d395e8ce3f, State: Running, Role: LEADER
I20260809 15:47:12.581493 23228 leader_election.cc:302] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c
I20260809 15:47:12.584740 24468 raft_consensus.cc:1261] T 4b30f28311964a0b9149a50495e83d8f P 61e69695258f48baa984c9d395e8ce3f [term 5 LEARNER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260809 15:47:12.546674 28290 raft_consensus.cc:3037] T 294d4bcc1e7a477cb56455581277a60f P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Advancing to term 6
W20260809 15:47:12.642756 23470 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 83721.6us
I20260809 15:47:12.644778 28284 raft_consensus.cc:482] T 3d880de657e544468c82157769622a26 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:12.645270 28284 raft_consensus.cc:504] T 3d880de657e544468c82157769622a26 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:12.648056 28271 consensus_queue.cc:1035] T 4b30f28311964a0b9149a50495e83d8f P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20260809 15:47:12.647213 28284 leader_election.cc:288] T 3d880de657e544468c82157769622a26 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:12.652021 28284 consensus_queue.cc:1046] T 297c82ed906d4b48b184985eed3de601 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260809 15:47:12.654178 28274 raft_consensus.cc:3032] T 297c82ed906d4b48b184985eed3de601 P 65c36d164b0540f099d592f38307ee6c [term 8 LEADER]: Stepping down as leader of term 8
I20260809 15:47:12.654524 28274 raft_consensus.cc:729] T 297c82ed906d4b48b184985eed3de601 P 65c36d164b0540f099d592f38307ee6c [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:47:12.655342 28274 consensus_queue.cc:260] T 297c82ed906d4b48b184985eed3de601 P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:12.652199 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d880de657e544468c82157769622a26" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:47:12.657074 24468 raft_consensus.cc:2454] T 3d880de657e544468c82157769622a26 P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 4.
I20260809 15:47:12.658022 23468 leader_election.cc:302] T 3d880de657e544468c82157769622a26 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c; no voters:
I20260809 15:47:12.658836 28271 raft_consensus.cc:2781] T 3d880de657e544468c82157769622a26 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260809 15:47:12.659204 28271 raft_consensus.cc:482] T 3d880de657e544468c82157769622a26 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:47:12.659529 28271 raft_consensus.cc:3037] T 3d880de657e544468c82157769622a26 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Advancing to term 5
W20260809 15:47:12.660735 23472 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 80835.1us
I20260809 15:47:12.578859 28309 raft_consensus.cc:2726] T df20671a7b874dbb98c0ef28371c1308 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260809 15:47:12.579551 28302 raft_consensus.cc:482] T cee02fff2c9c48a19f43eb358a89e7b0 P 61e69695258f48baa984c9d395e8ce3f [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
W20260809 15:47:12.687879 23223 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.68:34689, user_credentials={real_user=slave}} blocked reactor thread for 317751us
W20260809 15:47:12.678366 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 121968us
I20260809 15:47:12.654804 28319 consensus_queue.cc:260] T 4564dfc6d4094c1b9240b367ccecf3b1 P 67b3aeadfa7a432abe8af4d292e04ab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:12.692027 23228 leader_election.cc:302] T a88421dba80d4d3388aa5eb62e23853b P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:12.641276 28246 raft_consensus.cc:2781] T c0741ab44d99439abc3aa3bdd27ee160 P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Leader election won for term 8
I20260809 15:47:12.662248 28315 raft_consensus.cc:2781] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260809 15:47:12.736579 28209 consensus_queue.cc:237] T 0c48a3049897452383257aa757834fb5 P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:12.731019 23270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d880de657e544468c82157769622a26" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:12.579291 28324 consensus_queue.cc:1035] T f9e7cadcd24d45fcbfa84209225e8f2d P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 2, Time since last communication: 0.116s
W20260809 15:47:12.748939 23470 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 55036.2us
W20260809 15:47:12.749560 23472 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 56583.7us
I20260809 15:47:12.819350 23472 leader_election.cc:302] T f9e7cadcd24d45fcbfa84209225e8f2d P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:12.793998 23396 raft_consensus.cc:1226] T 2db057fd10e149a9895dff6ef8c9f038 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20260809 15:47:12.796056 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472816994d814cb198b69e5248697440" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f"
I20260809 15:47:12.545799 28337 consensus_queue.cc:260] T 3924b648461e463b9d7925e5671d68ae P 61e69695258f48baa984c9d395e8ce3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:12.580534 26966 consensus_queue.cc:1046] T d3e94b7e50754a81b3d72a07476cb7b9 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.113s
I20260809 15:47:12.807646 24452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "907bacfa79d04a55a68c8dbe60056462" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
W20260809 15:47:12.794749 28364 log.cc:924] Time spent T 5ad8a3b25d7b49e5a2142d5709edd0c7 P c0f38c1908414566aa9115bb2cbdd85d: Append to log took a long time: real 0.130s user 0.000s sys 0.001s
I20260809 15:47:12.758798 28230 raft_consensus.cc:482] T ba56725e42f74a1baa4c04d9660727f9 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:12.757849 28255 raft_consensus.cc:686] T 907bacfa79d04a55a68c8dbe60056462 P 61e69695258f48baa984c9d395e8ce3f [term 8 LEADER]: Becoming Leader. State: Replica: 61e69695258f48baa984c9d395e8ce3f, State: Running, Role: LEADER
I20260809 15:47:12.663118 27493 consensus_queue.cc:237] T 472816994d814cb198b69e5248697440 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:12.804841 27988 raft_consensus.cc:482] T f7715d71dbac447b9351fc5eb36c5e80 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:47:12.867663 23345 leader_election.cc:302] T 907bacfa79d04a55a68c8dbe60056462 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 61e69695258f48baa984c9d395e8ce3f, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:47:12.811743 28315 raft_consensus.cc:482] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:47:12.696600 28302 raft_consensus.cc:504] T cee02fff2c9c48a19f43eb358a89e7b0 P 61e69695258f48baa984c9d395e8ce3f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:12.746652 28285 consensus_queue.cc:1046] T 73f6f44c259547f1b05a3a434616fa54 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.125s
W20260809 15:47:12.878235 23472 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 58992.5us
I20260809 15:47:12.846277 28319 raft_consensus.cc:3037] T 4564dfc6d4094c1b9240b367ccecf3b1 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Advancing to term 7
W20260809 15:47:12.879500 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 187536us
I20260809 15:47:12.888881 23228 leader_election.cc:302] T 0c29937a80724ddb9d07a27534a146f3 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:12.887876 28323 raft_consensus.cc:3037] T a88421dba80d4d3388aa5eb62e23853b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Advancing to term 6
I20260809 15:47:12.877405 28315 raft_consensus.cc:3037] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Advancing to term 5
I20260809 15:47:12.916513 23228 leader_election.cc:302] T f7715d71dbac447b9351fc5eb36c5e80 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:12.916862 23399 raft_consensus.cc:1261] T 4bcac4b24e7a4448abdc934bba5a624e P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Refusing update from remote peer 61e69695258f48baa984c9d395e8ce3f: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
W20260809 15:47:12.923470 28394 log.cc:924] Time spent T 05ad7d8fdca64e3da527280ca5be9280 P 67b3aeadfa7a432abe8af4d292e04ab9: Append to log took a long time: real 0.078s user 0.001s sys 0.000s
W20260809 15:47:12.925447 28386 log.cc:924] Time spent T ba56725e42f74a1baa4c04d9660727f9 P 67b3aeadfa7a432abe8af4d292e04ab9: Append to log took a long time: real 0.132s user 0.000s sys 0.000s
I20260809 15:47:12.926757 28323 raft_consensus.cc:2726] T a88421dba80d4d3388aa5eb62e23853b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260809 15:47:12.929556 27751 leader_election.cc:288] T 2db057fd10e149a9895dff6ef8c9f038 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:12.930104 27751 raft_consensus.cc:2726] T 907bacfa79d04a55a68c8dbe60056462 P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260809 15:47:12.931020 27751 raft_consensus.cc:482] T 0da4d5283c8a46878bd39b85e5c71f66 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:12.931469 27751 raft_consensus.cc:504] T 0da4d5283c8a46878bd39b85e5c71f66 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:12.941448 28255 consensus_queue.cc:237] T 907bacfa79d04a55a68c8dbe60056462 P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
W20260809 15:47:12.943609 28203 consensus_meta.cc:287] T 472816994d814cb198b69e5248697440 P 61e69695258f48baa984c9d395e8ce3f: Time spent flushing consensus metadata: real 0.568s user 0.004s sys 0.004s
I20260809 15:47:12.948086 28220 consensus_queue.cc:1046] T a88421dba80d4d3388aa5eb62e23853b P 65c36d164b0540f099d592f38307ee6c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260809 15:47:12.949757 28220 raft_consensus.cc:3037] T f9e7cadcd24d45fcbfa84209225e8f2d P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Advancing to term 8
I20260809 15:47:12.952257 24471 raft_consensus.cc:1261] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
W20260809 15:47:12.954373 28168 consensus_meta.cc:287] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent flushing consensus metadata: real 0.618s user 0.006s sys 0.002s
I20260809 15:47:12.877235 27988 raft_consensus.cc:504] T f7715d71dbac447b9351fc5eb36c5e80 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:12.956609 27988 leader_election.cc:288] T f7715d71dbac447b9351fc5eb36c5e80 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:12.918351 24455 raft_consensus.cc:1261] T 563daf42d7784db195ea99c3cb21b370 P 61e69695258f48baa984c9d395e8ce3f [term 6 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260809 15:47:12.966398 24468 raft_consensus.cc:2454] T 472816994d814cb198b69e5248697440 P 61e69695258f48baa984c9d395e8ce3f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 6.
I20260809 15:47:12.865849 28230 raft_consensus.cc:504] T ba56725e42f74a1baa4c04d9660727f9 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:12.869328 28390 consensus_queue.cc:1046] T 953eea268e17491ca76f3a555860a512 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.126s
I20260809 15:47:12.942639 28290 raft_consensus.cc:504] T 294d4bcc1e7a477cb56455581277a60f P 61e69695258f48baa984c9d395e8ce3f [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:12.984450 28280 consensus_queue.cc:1035] T 4bcac4b24e7a4448abdc934bba5a624e P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.024s
I20260809 15:47:13.000027 28324 raft_consensus.cc:3037] T 0c29937a80724ddb9d07a27534a146f3 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Advancing to term 8
I20260809 15:47:13.000507 23522 raft_consensus.cc:1261] T 4bcac4b24e7a4448abdc934bba5a624e P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Refusing update from remote peer 61e69695258f48baa984c9d395e8ce3f: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
W20260809 15:47:13.005925 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 89480.5us
W20260809 15:47:13.028553 23345 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.68:34689, user_credentials={real_user=slave}} blocked reactor thread for 160970us
W20260809 15:47:13.038065 23468 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.68:34689, user_credentials={real_user=slave}} blocked reactor thread for 290629us
I20260809 15:47:13.044346 23228 leader_election.cc:302] T 28e17dcb5efe45b8b6a8c42da03b4819 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:13.019111 28403 raft_consensus.cc:2726] T f7715d71dbac447b9351fc5eb36c5e80 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260809 15:47:13.017230 28390 raft_consensus.cc:3032] T 953eea268e17491ca76f3a555860a512 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 LEADER]: Stepping down as leader of term 7
W20260809 15:47:13.075053 23470 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 89282.9us
I20260809 15:47:13.010423 28412 raft_consensus.cc:3032] T 73f6f44c259547f1b05a3a434616fa54 P 65c36d164b0540f099d592f38307ee6c [term 7 LEADER]: Stepping down as leader of term 7
I20260809 15:47:13.018349 24455 raft_consensus.cc:1261] T 3d880de657e544468c82157769622a26 P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Refusing update from remote peer 67b3aeadfa7a432abe8af4d292e04ab9: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260809 15:47:13.080828 23228 leader_election.cc:302] T 0acc47ce21c3422698d1ef7277c3ed63 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:13.043603 28131 raft_consensus.cc:482] T 213543cdbe814c89a17a963469922c1b P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:13.087960 28309 raft_consensus.cc:482] T d2d4f749fcfd4fb3bfb8115af3471598 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:13.073132 28168 raft_consensus.cc:2726] T 28e17dcb5efe45b8b6a8c42da03b4819 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260809 15:47:13.091599 28425 raft_consensus.cc:3032] T a88421dba80d4d3388aa5eb62e23853b P 65c36d164b0540f099d592f38307ee6c [term 5 LEADER]: Stepping down as leader of term 5
I20260809 15:47:13.091849 27751 leader_election.cc:288] T 0da4d5283c8a46878bd39b85e5c71f66 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:13.092533 28230 leader_election.cc:288] T ba56725e42f74a1baa4c04d9660727f9 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:13.134652 23270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7715d71dbac447b9351fc5eb36c5e80" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:13.000223 28246 raft_consensus.cc:686] T c0741ab44d99439abc3aa3bdd27ee160 P 65c36d164b0540f099d592f38307ee6c [term 8 LEADER]: Becoming Leader. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:47:13.121358 23280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0da4d5283c8a46878bd39b85e5c71f66" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:13.139226 23519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7715d71dbac447b9351fc5eb36c5e80" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:13.120301 23228 leader_election.cc:302] T 2db057fd10e149a9895dff6ef8c9f038 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:13.118662 23273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2db057fd10e149a9895dff6ef8c9f038" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:13.076496 28345 raft_consensus.cc:482] T a3622f6e3e3641cea1960e3a31498bb7 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:13.082242 28412 raft_consensus.cc:729] T 73f6f44c259547f1b05a3a434616fa54 P 65c36d164b0540f099d592f38307ee6c [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:47:13.078258 28390 raft_consensus.cc:729] T 953eea268e17491ca76f3a555860a512 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Running, Role: LEADER
W20260809 15:47:13.138091 23353 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 80683.5us
I20260809 15:47:13.128206 28419 raft_consensus.cc:2726] T 0acc47ce21c3422698d1ef7277c3ed63 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260809 15:47:13.156147 23519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0da4d5283c8a46878bd39b85e5c71f66" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:13.159162 23521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2db057fd10e149a9895dff6ef8c9f038" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:13.156834 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba56725e42f74a1baa4c04d9660727f9" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:13.131984 28425 raft_consensus.cc:729] T a88421dba80d4d3388aa5eb62e23853b P 65c36d164b0540f099d592f38307ee6c [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:47:13.119302 27988 consensus_queue.cc:1035] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.036s
I20260809 15:47:13.141278 28404 raft_consensus.cc:482] T cee02fff2c9c48a19f43eb358a89e7b0 P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 15:47:13.175420 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 55182.5us
I20260809 15:47:13.176131 23228 leader_election.cc:302] T c5c2e425a74b4e64ae70b5def750434c P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:13.159574 28230 raft_consensus.cc:482] T a88421dba80d4d3388aa5eb62e23853b P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:13.163630 28353 raft_consensus.cc:482] T 297c82ed906d4b48b184985eed3de601 P c0f38c1908414566aa9115bb2cbdd85d [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:13.163219 24451 raft_consensus.cc:1226] T 907bacfa79d04a55a68c8dbe60056462 P 61e69695258f48baa984c9d395e8ce3f [term 8 LEADER]: Rejecting Update request from peer 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 7. Current term is 8. Ops: [7.8-7.8]
I20260809 15:47:13.179445 23228 leader_election.cc:302] T 2424fd9bd7f14e508fd106cbfda786a4 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c
I20260809 15:47:13.128279 28131 raft_consensus.cc:504] T 213543cdbe814c89a17a963469922c1b P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:13.193675 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "213543cdbe814c89a17a963469922c1b" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:13.187537 23280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba56725e42f74a1baa4c04d9660727f9" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:13.154290 28280 consensus_queue.cc:1035] T 4bcac4b24e7a4448abdc934bba5a624e P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.030s
I20260809 15:47:13.133551 28309 raft_consensus.cc:504] T d2d4f749fcfd4fb3bfb8115af3471598 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:13.203569 23521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "294d4bcc1e7a477cb56455581277a60f" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:47:13.184691 28416 raft_consensus.cc:3037] T 2db057fd10e149a9895dff6ef8c9f038 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Advancing to term 7
I20260809 15:47:13.183475 28290 leader_election.cc:288] T 294d4bcc1e7a477cb56455581277a60f P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 6 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:13.154969 28411 consensus_queue.cc:1046] T d3e94b7e50754a81b3d72a07476cb7b9 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.046s
I20260809 15:47:13.166482 28345 raft_consensus.cc:504] T a3622f6e3e3641cea1960e3a31498bb7 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:13.187497 27892 raft_consensus.cc:2726] T c5c2e425a74b4e64ae70b5def750434c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260809 15:47:13.193663 28131 leader_election.cc:288] T 213543cdbe814c89a17a963469922c1b P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:13.199856 28353 raft_consensus.cc:504] T 297c82ed906d4b48b184985eed3de601 P c0f38c1908414566aa9115bb2cbdd85d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:13.280516 23521 raft_consensus.cc:2454] T 294d4bcc1e7a477cb56455581277a60f P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 6.
I20260809 15:47:13.281572 24410 leader_election.cc:302] T 294d4bcc1e7a477cb56455581277a60f P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c; no voters:
I20260809 15:47:13.194705 28230 raft_consensus.cc:504] T a88421dba80d4d3388aa5eb62e23853b P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:47:13.244231 28370 log.cc:924] Time spent T 05ad7d8fdca64e3da527280ca5be9280 P 65c36d164b0540f099d592f38307ee6c: Append to log took a long time: real 0.580s user 0.002s sys 0.000s
I20260809 15:47:13.172106 28168 consensus_queue.cc:1035] T 3d880de657e544468c82157769622a26 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.035s
I20260809 15:47:13.155206 28431 consensus_queue.cc:1035] T 563daf42d7784db195ea99c3cb21b370 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.079s
I20260809 15:47:13.245131 28425 consensus_queue.cc:260] T a88421dba80d4d3388aa5eb62e23853b P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:47:13.258535 28373 log.cc:924] Time spent T d3be1e356b254d22b995be32f1c12f6f P c0f38c1908414566aa9115bb2cbdd85d: Append to log took a long time: real 0.126s user 0.000s sys 0.000s
W20260809 15:47:13.250463 28271 consensus_meta.cc:287] T 3d880de657e544468c82157769622a26 P 65c36d164b0540f099d592f38307ee6c: Time spent flushing consensus metadata: real 0.591s user 0.007s sys 0.000s
W20260809 15:47:13.254613 28335 consensus_meta.cc:287] T 16ca412a0dae4feebe85dd01bf1a9173 P 61e69695258f48baa984c9d395e8ce3f: Time spent flushing consensus metadata: real 0.700s user 0.006s sys 0.000s
W20260809 15:47:13.274000 23470 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 103112us
I20260809 15:47:13.212235 28410 raft_consensus.cc:2781] T 2424fd9bd7f14e508fd106cbfda786a4 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260809 15:47:13.259223 28220 raft_consensus.cc:2726] T f9e7cadcd24d45fcbfa84209225e8f2d P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260809 15:47:13.313326 28171 raft_consensus.cc:2781] T 294d4bcc1e7a477cb56455581277a60f P 61e69695258f48baa984c9d395e8ce3f [term 6 FOLLOWER]: Leader election won for term 6
I20260809 15:47:13.184870 28404 raft_consensus.cc:504] T cee02fff2c9c48a19f43eb358a89e7b0 P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:13.202910 28390 consensus_queue.cc:260] T 953eea268e17491ca76f3a555860a512 P 67b3aeadfa7a432abe8af4d292e04ab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:13.206357 28246 consensus_queue.cc:237] T c0741ab44d99439abc3aa3bdd27ee160 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:13.341428 21289 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20260809 15:47:13.364470 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 185104us
I20260809 15:47:13.356793 23529 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56766 (request call id 2108) took 1076 ms. Trace:
W20260809 15:47:13.365888 24406 leader_election.cc:334] T 3d880de657e544468c82157769622a26 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459): Timed out: RequestConsensusVote RPC to 127.20.202.65:35459 timed out after 1.604s (SENT)
W20260809 15:47:13.366629 23472 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 461457us
I20260809 15:47:13.366308 24406 leader_election.cc:302] T 3d880de657e544468c82157769622a26 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
W20260809 15:47:13.340005 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 152417us
I20260809 15:47:13.365324 23529 rpcz_store.cc:276] 0809 15:47:12.280507 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:12.282488 (+ 1981us) service_pool.cc:227] Handling call
0809 15:47:12.288047 (+ 5559us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 15:47:12.288331 (+ 284us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:47:12.288355 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 15:47:12.288718 (+ 363us) log.cc:842] Serialized 26 byte log entry
0809 15:47:12.516698 (+227980us) raft_consensus.cc:1631] Marking committed up to 5
0809 15:47:12.516735 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:47:12.516805 (+ 70us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 15:47:13.183907 (+667102us) spinlock_profiling.cc:243] Waited 3.23 ms on lock 0x7b44000232cc. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e613cf11 00007f34e620aa95 00007f34e620b4b2 00007f34e8eb3511 00007f34e8eaee1a 00007f34e8ea2bee 00007f34e8ea1f36 00007f34e8ea1b58 00007f34e8eab551 00007f34e8eab509 00007f34e8eab499
0809 15:47:13.356516 (+172609us) raft_consensus.cc:1671] finished
0809 15:47:13.356535 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:47:13.356736 (+ 201us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":3126,"raft.run_cpu_time_us":88,"raft.run_wall_time_us":88,"spinlock_wait_cycles":7429376,"thread_start_us":215147,"threads_started":1,"wal-append.queue_time_us":248359}
I20260809 15:47:13.378383 23528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a88421dba80d4d3388aa5eb62e23853b" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
W20260809 15:47:13.353752 28433 log.cc:924] Time spent T 00000000000000000000000000000000 P a86a2513b8024a869d80303dcbf44163: Append to log took a long time: real 0.151s user 0.000s sys 0.000s
I20260809 15:47:13.332557 28271 raft_consensus.cc:504] T 3d880de657e544468c82157769622a26 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:13.372045 28290 raft_consensus.cc:482] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:13.388428 28290 raft_consensus.cc:504] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:13.390318 28290 leader_election.cc:288] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:13.332728 28335 raft_consensus.cc:504] T 16ca412a0dae4feebe85dd01bf1a9173 P 61e69695258f48baa984c9d395e8ce3f [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:13.391501 24463 raft_consensus.cc:1226] T 16ca412a0dae4feebe85dd01bf1a9173 P 61e69695258f48baa984c9d395e8ce3f [term 7 FOLLOWER]: Rejecting Update request from peer 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20260809 15:47:13.392328 28335 leader_election.cc:288] T 16ca412a0dae4feebe85dd01bf1a9173 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 7 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:13.394018 28271 leader_election.cc:288] T 3d880de657e544468c82157769622a26 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 5 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:13.398330 23529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad8a3b25d7b49e5a2142d5709edd0c7" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:13.398983 23529 raft_consensus.cc:2454] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 5.
I20260809 15:47:13.375841 28315 raft_consensus.cc:504] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:13.401013 28315 leader_election.cc:288] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:13.219954 28412 consensus_queue.cc:260] T 73f6f44c259547f1b05a3a434616fa54 P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:13.402177 28425 raft_consensus.cc:3037] T a88421dba80d4d3388aa5eb62e23853b P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Advancing to term 6
I20260809 15:47:13.408767 28302 leader_election.cc:288] T cee02fff2c9c48a19f43eb358a89e7b0 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:13.409950 23274 raft_consensus.cc:1226] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 4. Current term is 5. Ops: []
I20260809 15:47:13.364193 28230 leader_election.cc:288] T a88421dba80d4d3388aa5eb62e23853b P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:13.333716 28410 raft_consensus.cc:482] T 2424fd9bd7f14e508fd106cbfda786a4 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:47:13.336213 28208 raft_consensus.cc:482] T 6ae14a74a26f434db08c652e097385c8 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
I20260809 15:47:13.418999 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee02fff2c9c48a19f43eb358a89e7b0" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:13.419680 24410 leader_election.cc:302] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c; no voters:
I20260809 15:47:13.378021 28416 raft_consensus.cc:2726] T 2db057fd10e149a9895dff6ef8c9f038 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260809 15:47:13.447842 23526 raft_consensus.cc:1261] T 0c48a3049897452383257aa757834fb5 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Refusing update from remote peer 61e69695258f48baa984c9d395e8ce3f: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260809 15:47:13.452877 23529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16ca412a0dae4feebe85dd01bf1a9173" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:47:13.453483 23529 raft_consensus.cc:3037] T 16ca412a0dae4feebe85dd01bf1a9173 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Advancing to term 7
I20260809 15:47:13.455500 28442 raft_consensus.cc:482] T 7660dfa2ef694927b8eda4100a2bcd9b P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:13.456007 28442 raft_consensus.cc:504] T 7660dfa2ef694927b8eda4100a2bcd9b P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:13.457942 28442 leader_election.cc:288] T 7660dfa2ef694927b8eda4100a2bcd9b P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:13.396806 28255 raft_consensus.cc:2726] T 3d880de657e544468c82157769622a26 P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260809 15:47:13.458590 23402 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52856 (request call id 389) took 1269 ms. Trace:
I20260809 15:47:13.411425 28309 leader_election.cc:288] T d2d4f749fcfd4fb3bfb8115af3471598 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:13.459625 23396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad8a3b25d7b49e5a2142d5709edd0c7" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:13.460197 23529 raft_consensus.cc:2454] T 16ca412a0dae4feebe85dd01bf1a9173 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 7.
W20260809 15:47:13.460835 28360 log.cc:924] Time spent T f9e7cadcd24d45fcbfa84209225e8f2d P 67b3aeadfa7a432abe8af4d292e04ab9: Append to log took a long time: real 0.819s user 0.001s sys 0.001s
I20260809 15:47:13.455441 28353 leader_election.cc:288] T 297c82ed906d4b48b184985eed3de601 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
W20260809 15:47:13.462634 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 84911.6us
I20260809 15:47:13.463320 23228 leader_election.cc:302] T d1a134ba33ea4076bbaac0ae460c17ba P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c
I20260809 15:47:13.463572 24448 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38722 (request call id 543) took 1095 ms. Trace:
I20260809 15:47:13.463903 24448 rpcz_store.cc:276] 0809 15:47:12.367953 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:12.369262 (+ 1309us) service_pool.cc:227] Handling call
0809 15:47:13.463377 (+1094115us) spinlock_profiling.cc:243] Waited 757 ms on lock 0x7b5400521a54. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e92242ae 00007f34ee161c27 00007f34e7849c04 00007f34e7849b92 00007f34e7849ac1 00007f34e7849a3c 00007f34e7848cf2 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:47:13.463496 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1740321152}
I20260809 15:47:13.466284 24453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0741ab44d99439abc3aa3bdd27ee160" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f"
I20260809 15:47:13.466910 24453 raft_consensus.cc:3037] T c0741ab44d99439abc3aa3bdd27ee160 P 61e69695258f48baa984c9d395e8ce3f [term 7 FOLLOWER]: Advancing to term 8
I20260809 15:47:13.471719 23228 leader_election.cc:302] T a3622f6e3e3641cea1960e3a31498bb7 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:13.447434 23527 raft_consensus.cc:2454] T cee02fff2c9c48a19f43eb358a89e7b0 P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 8.
W20260809 15:47:13.478983 23468 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.68:34689, user_credentials={real_user=slave}} blocked reactor thread for 106575us
I20260809 15:47:13.458940 23402 rpcz_store.cc:276] 0809 15:47:12.189295 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:12.193238 (+ 3943us) service_pool.cc:227] Handling call
0809 15:47:12.193565 (+ 327us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 15:47:12.193869 (+ 304us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:47:12.193893 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 15:47:12.194419 (+ 526us) log.cc:842] Serialized 26 byte log entry
0809 15:47:12.561627 (+367208us) raft_consensus.cc:1631] Marking committed up to 6
0809 15:47:12.561701 (+ 74us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:47:12.561910 (+ 209us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 15:47:13.257850 (+695940us) spinlock_profiling.cc:243] Waited 3.41 ms on lock 0x7b44000232cc. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e613cf11 00007f34e620aa95 00007f34e620b4b2 00007f34e8eb3511 00007f34e8eaee1a 00007f34e8ea2bee 00007f34e8ea1f36 00007f34e8ea1b58 00007f34e8eab551 00007f34e8eab509 00007f34e8eab499
0809 15:47:13.448676 (+190826us) raft_consensus.cc:1671] finished
0809 15:47:13.448696 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:47:13.458528 (+ 9832us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":4274,"raft.run_cpu_time_us":104,"raft.run_wall_time_us":17165,"spinlock_wait_cycles":7848832,"thread_start_us":366576,"threads_started":1,"wal-append.queue_time_us":837566}
I20260809 15:47:13.482587 23228 leader_election.cc:302] T d2d4f749fcfd4fb3bfb8115af3471598 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
W20260809 15:47:13.483238 23472 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 108052us
W20260809 15:47:13.479768 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 110570us
I20260809 15:47:13.485749 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2d4f749fcfd4fb3bfb8115af3471598" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:13.486016 23529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "297c82ed906d4b48b184985eed3de601" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 10 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:13.486438 23527 raft_consensus.cc:2454] T d2d4f749fcfd4fb3bfb8115af3471598 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 7.
I20260809 15:47:13.510378 24410 leader_election.cc:302] T 16ca412a0dae4feebe85dd01bf1a9173 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c; no voters:
I20260809 15:47:13.449716 28390 raft_consensus.cc:3037] T 953eea268e17491ca76f3a555860a512 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Advancing to term 8
I20260809 15:47:13.511193 24410 leader_election.cc:302] T cee02fff2c9c48a19f43eb358a89e7b0 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c; no voters:
I20260809 15:47:13.514145 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7660dfa2ef694927b8eda4100a2bcd9b" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:13.514338 24448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7660dfa2ef694927b8eda4100a2bcd9b" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:47:13.446291 28131 raft_consensus.cc:482] T 07fcabb569414bddbd39df6cedd24e64 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:47:13.397259 28171 raft_consensus.cc:686] T 294d4bcc1e7a477cb56455581277a60f P 61e69695258f48baa984c9d395e8ce3f [term 6 LEADER]: Becoming Leader. State: Replica: 61e69695258f48baa984c9d395e8ce3f, State: Running, Role: LEADER
I20260809 15:47:13.450301 28410 raft_consensus.cc:3037] T 2424fd9bd7f14e508fd106cbfda786a4 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Advancing to term 8
I20260809 15:47:13.479254 28168 raft_consensus.cc:2781] T d1a134ba33ea4076bbaac0ae460c17ba P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260809 15:47:13.525708 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d880de657e544468c82157769622a26" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f"
W20260809 15:47:13.530128 23353 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 48877.7us
I20260809 15:47:13.515879 28335 raft_consensus.cc:2781] T 16ca412a0dae4feebe85dd01bf1a9173 P 61e69695258f48baa984c9d395e8ce3f [term 7 FOLLOWER]: Leader election won for term 7
W20260809 15:47:13.530035 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 45876.5us
I20260809 15:47:13.534268 23351 leader_election.cc:302] T f7715d71dbac447b9351fc5eb36c5e80 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:47:13.522644 28426 raft_consensus.cc:2781] T cee02fff2c9c48a19f43eb358a89e7b0 P 61e69695258f48baa984c9d395e8ce3f [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260809 15:47:13.518772 28416 raft_consensus.cc:2726] T d2d4f749fcfd4fb3bfb8115af3471598 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260809 15:47:13.536348 23351 leader_election.cc:302] T 0da4d5283c8a46878bd39b85e5c71f66 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:47:13.539131 23228 leader_election.cc:302] T 00a1881267a14ba8b78c81040f17a014 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:13.540210 23228 leader_election.cc:302] T ae6d2c08a7a94f58ac8dd698e3fd456c P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:13.541217 28426 raft_consensus.cc:482] T cee02fff2c9c48a19f43eb358a89e7b0 P 61e69695258f48baa984c9d395e8ce3f [term 8 FOLLOWER]: Starting leader election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
I20260809 15:47:13.541676 28426 raft_consensus.cc:3037] T cee02fff2c9c48a19f43eb358a89e7b0 P 61e69695258f48baa984c9d395e8ce3f [term 8 FOLLOWER]: Advancing to term 9
I20260809 15:47:13.543509 28438 raft_consensus.cc:3037] T a3622f6e3e3641cea1960e3a31498bb7 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Advancing to term 6
I20260809 15:47:13.544236 28458 raft_consensus.cc:2726] T f7715d71dbac447b9351fc5eb36c5e80 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260809 15:47:13.545207 28171 consensus_queue.cc:237] T 294d4bcc1e7a477cb56455581277a60f P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:13.547638 27892 raft_consensus.cc:2726] T 00a1881267a14ba8b78c81040f17a014 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260809 15:47:13.478957 28208 raft_consensus.cc:504] T 6ae14a74a26f434db08c652e097385c8 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:13.526456 27956 raft_consensus.cc:482] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:13.526486 28168 raft_consensus.cc:482] T d1a134ba33ea4076bbaac0ae460c17ba P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:47:13.549788 23228 leader_election.cc:302] T 0a51c0baaa6e432997c9d3552af4f254 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters: 61e69695258f48baa984c9d395e8ce3f
I20260809 15:47:13.530315 28324 raft_consensus.cc:2726] T 0c29937a80724ddb9d07a27534a146f3 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260809 15:47:13.515731 28255 consensus_queue.cc:1035] T 0c48a3049897452383257aa757834fb5 P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.023s
I20260809 15:47:13.604156 23402 raft_consensus.cc:1261] T 907bacfa79d04a55a68c8dbe60056462 P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Refusing update from remote peer 61e69695258f48baa984c9d395e8ce3f: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260809 15:47:13.553723 28416 raft_consensus.cc:2726] T ae6d2c08a7a94f58ac8dd698e3fd456c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260809 15:47:13.525489 28131 raft_consensus.cc:504] T 07fcabb569414bddbd39df6cedd24e64 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:47:13.608289 23468 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.68:34689, user_credentials={real_user=slave}} blocked reactor thread for 122119us
W20260809 15:47:13.608232 23470 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 85269.1us
I20260809 15:47:13.601291 28335 raft_consensus.cc:686] T 16ca412a0dae4feebe85dd01bf1a9173 P 61e69695258f48baa984c9d395e8ce3f [term 7 LEADER]: Becoming Leader. State: Replica: 61e69695258f48baa984c9d395e8ce3f, State: Running, Role: LEADER
I20260809 15:47:13.555195 23351 leader_election.cc:302] T 2db057fd10e149a9895dff6ef8c9f038 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:47:13.628520 23277 raft_consensus.cc:1226] T 0c29937a80724ddb9d07a27534a146f3 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 7. Current term is 8. Ops: []
W20260809 15:47:13.628916 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 73805.3us
I20260809 15:47:13.629623 23351 leader_election.cc:302] T ba56725e42f74a1baa4c04d9660727f9 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:47:13.618100 23472 leader_election.cc:302] T 7baa2492df0340c4a5ac241959ce1ac3 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters: 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:47:13.538384 28302 raft_consensus.cc:2781] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260809 15:47:13.625088 23353 leader_election.cc:302] T d2d4f749fcfd4fb3bfb8115af3471598 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:47:13.626112 23280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee02fff2c9c48a19f43eb358a89e7b0" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:13.610015 28458 raft_consensus.cc:2726] T 0da4d5283c8a46878bd39b85e5c71f66 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260809 15:47:13.633277 28421 raft_consensus.cc:2726] T 2db057fd10e149a9895dff6ef8c9f038 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260809 15:47:13.633415 28449 consensus_queue.cc:1046] T a3622f6e3e3641cea1960e3a31498bb7 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.097s
I20260809 15:47:13.634203 23353 leader_election.cc:302] T 7660dfa2ef694927b8eda4100a2bcd9b P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c
I20260809 15:47:13.634717 28457 raft_consensus.cc:2781] T d2d4f749fcfd4fb3bfb8115af3471598 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260809 15:47:13.617303 28168 raft_consensus.cc:3037] T d1a134ba33ea4076bbaac0ae460c17ba P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Advancing to term 7
W20260809 15:47:13.634779 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 85055.8us
I20260809 15:47:13.634455 23276 raft_consensus.cc:2932] T ba56725e42f74a1baa4c04d9660727f9 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Committing config change with OpId 5.4: config changed from index -1 to 4, NON_VOTER 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } attrs { replace: true } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } } }
I20260809 15:47:13.635828 28309 raft_consensus.cc:2726] T ba56725e42f74a1baa4c04d9660727f9 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260809 15:47:13.637630 23273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "213543cdbe814c89a17a963469922c1b" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:13.638868 23285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2d4f749fcfd4fb3bfb8115af3471598" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:13.639882 28383 raft_consensus.cc:2781] T 0a51c0baaa6e432997c9d3552af4f254 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Leader election won for term 7
I20260809 15:47:13.640023 23286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a88421dba80d4d3388aa5eb62e23853b" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:13.643627 23282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "297c82ed906d4b48b184985eed3de601" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 10 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:13.644196 23283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3622f6e3e3641cea1960e3a31498bb7" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:13.645437 27988 raft_consensus.cc:482] T 0c29937a80724ddb9d07a27534a146f3 P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:13.724802 24448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee02fff2c9c48a19f43eb358a89e7b0" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:47:13.652127 28208 leader_election.cc:288] T 6ae14a74a26f434db08c652e097385c8 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:13.633309 23526 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "538a228ddfb54bc6a36528d5ff07597b"
dest_uuid: "65c36d164b0540f099d592f38307ee6c"
mode: GRACEFUL
new_leader_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
from {username='slave'} at 127.0.0.1:48888
I20260809 15:47:13.731171 23526 raft_consensus.cc:595] T 538a228ddfb54bc6a36528d5ff07597b P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Received request to transfer leadership to TS c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:13.731555 23526 raft_consensus.cc:603] T 538a228ddfb54bc6a36528d5ff07597b P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260809 15:47:13.654217 28404 leader_election.cc:288] T cee02fff2c9c48a19f43eb358a89e7b0 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:13.602391 27956 raft_consensus.cc:504] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:13.665132 23228 leader_election.cc:302] T 538a228ddfb54bc6a36528d5ff07597b P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c
W20260809 15:47:13.737356 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 72296.3us
I20260809 15:47:13.740867 23521 raft_consensus.cc:1261] T cee02fff2c9c48a19f43eb358a89e7b0 P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Refusing update from remote peer 67b3aeadfa7a432abe8af4d292e04ab9: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260809 15:47:13.636015 23270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d880de657e544468c82157769622a26" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:47:13.722651 23285 raft_consensus.cc:2454] T d2d4f749fcfd4fb3bfb8115af3471598 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 7.
I20260809 15:47:13.728241 23282 raft_consensus.cc:2396] T 297c82ed906d4b48b184985eed3de601 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d for term 10 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20260809 15:47:13.663764 28457 raft_consensus.cc:482] T d2d4f749fcfd4fb3bfb8115af3471598 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260809 15:47:13.733229 23470 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 91013.6us
I20260809 15:47:13.659061 24453 raft_consensus.cc:2454] T c0741ab44d99439abc3aa3bdd27ee160 P 61e69695258f48baa984c9d395e8ce3f [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 8.
I20260809 15:47:13.743392 23399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0721a4403a440f383318a3cea9d1442" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:47:13.750815 28455 raft_consensus.cc:482] T 953eea268e17491ca76f3a555860a512 P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:13.755681 28455 raft_consensus.cc:504] T 953eea268e17491ca76f3a555860a512 P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:13.756834 28335 consensus_queue.cc:237] T 16ca412a0dae4feebe85dd01bf1a9173 P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:13.757611 28455 leader_election.cc:288] T 953eea268e17491ca76f3a555860a512 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:13.757916 23397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba56725e42f74a1baa4c04d9660727f9" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:13.758677 23397 raft_consensus.cc:2379] T ba56725e42f74a1baa4c04d9660727f9 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 in current term 6: Already voted for candidate c0f38c1908414566aa9115bb2cbdd85d in this term.
I20260809 15:47:13.761710 23396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998c6cee72cf40228c6dba064c0db2da" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:47:13.762487 28285 raft_consensus.cc:3032] T d3e94b7e50754a81b3d72a07476cb7b9 P 65c36d164b0540f099d592f38307ee6c [term 7 LEADER]: Stepping down as leader of term 7
I20260809 15:47:13.766880 23394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b30f28311964a0b9149a50495e83d8f" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:13.767143 23393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "563daf42d7784db195ea99c3cb21b370" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
W20260809 15:47:13.751839 23472 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 143384us
I20260809 15:47:13.767884 23404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2d4f749fcfd4fb3bfb8115af3471598" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: true dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:47:13.770172 23394 raft_consensus.cc:2354] T 4b30f28311964a0b9149a50495e83d8f P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 4. Current term is 5.
I20260809 15:47:13.770740 23399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3be1e356b254d22b995be32f1c12f6f" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:13.772070 23393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "538a228ddfb54bc6a36528d5ff07597b" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:13.772661 23393 raft_consensus.cc:2362] T 538a228ddfb54bc6a36528d5ff07597b P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 7. Re-sending same reply.
I20260809 15:47:13.773583 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2d4f749fcfd4fb3bfb8115af3471598" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: true dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:47:13.773254 23392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73f6f44c259547f1b05a3a434616fa54" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:13.775610 23521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5712f6b6fffc4cc5a8ddfbe63e4097df" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:13.776042 23392 raft_consensus.cc:2379] T 73f6f44c259547f1b05a3a434616fa54 P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 in current term 8: Already voted for candidate c0f38c1908414566aa9115bb2cbdd85d in this term.
W20260809 15:47:13.776573 23468 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.68:34689, user_credentials={real_user=slave}} blocked reactor thread for 90819.1us
I20260809 15:47:13.736634 28462 raft_consensus.cc:2741] T 7baa2492df0340c4a5ac241959ce1ac3 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260809 15:47:13.777926 23393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae6d2c08a7a94f58ac8dd698e3fd456c" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:13.726462 28421 raft_consensus.cc:2726] T 7660dfa2ef694927b8eda4100a2bcd9b P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260809 15:47:13.778476 23393 raft_consensus.cc:2454] T ae6d2c08a7a94f58ac8dd698e3fd456c P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 8.
I20260809 15:47:13.780421 23528 raft_consensus.cc:2454] T a88421dba80d4d3388aa5eb62e23853b P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 6.
I20260809 15:47:13.740477 27751 raft_consensus.cc:482] T 0acc47ce21c3422698d1ef7277c3ed63 P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:13.781925 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e4ffcf83c234ae7910ddef3ec0b4bda" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:13.782534 23470 leader_election.cc:302] T 3d880de657e544468c82157769622a26 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 61e69695258f48baa984c9d395e8ce3f, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:47:13.776440 23394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "763e58d6ba644eea830a6c1664d39c9b" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:13.789501 23394 raft_consensus.cc:2454] T 763e58d6ba644eea830a6c1664d39c9b P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 7.
I20260809 15:47:13.790112 23472 leader_election.cc:302] T 4564dfc6d4094c1b9240b367ccecf3b1 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:13.783593 23399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5c2e425a74b4e64ae70b5def750434c" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:13.775549 23526 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6ae14a74a26f434db08c652e097385c8"
dest_uuid: "65c36d164b0540f099d592f38307ee6c"
mode: GRACEFUL
new_leader_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
from {username='slave'} at 127.0.0.1:48888
I20260809 15:47:13.784940 23518 raft_consensus.cc:1261] T 472816994d814cb198b69e5248697440 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260809 15:47:13.779604 23400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0da4d5283c8a46878bd39b85e5c71f66" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:13.785431 23468 leader_election.cc:302] T cee02fff2c9c48a19f43eb358a89e7b0 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 61e69695258f48baa984c9d395e8ce3f, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:47:13.791735 23400 raft_consensus.cc:2379] T 0da4d5283c8a46878bd39b85e5c71f66 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 in current term 6: Already voted for candidate c0f38c1908414566aa9115bb2cbdd85d in this term.
W20260809 15:47:13.787855 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 141157us
I20260809 15:47:13.792717 23520 raft_consensus.cc:1226] T 30c1740ef6874df0a332fbd6fd4b03ce P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Rejecting Update request from peer 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 7. Current term is 8. Ops: []
I20260809 15:47:13.751547 28457 raft_consensus.cc:3037] T d2d4f749fcfd4fb3bfb8115af3471598 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Advancing to term 8
I20260809 15:47:13.786157 23393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "297c82ed906d4b48b184985eed3de601" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:13.798378 23393 raft_consensus.cc:2379] T 297c82ed906d4b48b184985eed3de601 P c0f38c1908414566aa9115bb2cbdd85d [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 in current term 9: Already voted for candidate c0f38c1908414566aa9115bb2cbdd85d in this term.
I20260809 15:47:13.787014 23396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3296ef50be254fc29cfc77d22a8f7a23" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:13.774008 24453 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38734 (request call id 402) took 1381 ms. Trace:
I20260809 15:47:13.804206 23394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f9d6565aa4c45a3b3f01210d0eb71f7" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:13.778549 23395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cef3e7277d7d4760824a3436ed4a847d" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:47:13.779019 23397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07fcabb569414bddbd39df6cedd24e64" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:13.790810 23399 raft_consensus.cc:2454] T c5c2e425a74b4e64ae70b5def750434c P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 7.
I20260809 15:47:13.796443 23521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ae14a74a26f434db08c652e097385c8" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:13.762873 28285 raft_consensus.cc:729] T d3e94b7e50754a81b3d72a07476cb7b9 P 65c36d164b0540f099d592f38307ee6c [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:47:13.791168 23526 raft_consensus.cc:595] T 6ae14a74a26f434db08c652e097385c8 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Received request to transfer leadership to TS 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:47:13.733633 27988 raft_consensus.cc:504] T 0c29937a80724ddb9d07a27534a146f3 P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:13.801898 23518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "953eea268e17491ca76f3a555860a512" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:13.787405 23527 raft_consensus.cc:2454] T d2d4f749fcfd4fb3bfb8115af3471598 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 7.
I20260809 15:47:13.804296 23285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "953eea268e17491ca76f3a555860a512" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:13.810532 23526 raft_consensus.cc:603] T 6ae14a74a26f434db08c652e097385c8 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260809 15:47:13.811234 23518 raft_consensus.cc:2396] T 953eea268e17491ca76f3a555860a512 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d for term 9 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20260809 15:47:13.812212 23286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16ca412a0dae4feebe85dd01bf1a9173" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:13.816486 23270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ae14a74a26f434db08c652e097385c8" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:13.815881 23521 raft_consensus.cc:2454] T 6ae14a74a26f434db08c652e097385c8 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 7.
I20260809 15:47:13.817763 23393 raft_consensus.cc:1226] T 30c1740ef6874df0a332fbd6fd4b03ce P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Rejecting Update request from peer 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20260809 15:47:13.808704 23397 raft_consensus.cc:2454] T 07fcabb569414bddbd39df6cedd24e64 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 7.
I20260809 15:47:13.824545 23283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "907bacfa79d04a55a68c8dbe60056462" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
W20260809 15:47:13.861256 23472 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 81711.7us
I20260809 15:47:13.861503 23282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d880de657e544468c82157769622a26" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:47:13.861279 23283 raft_consensus.cc:2396] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 61e69695258f48baa984c9d395e8ce3f for term 8 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
I20260809 15:47:13.827559 23351 leader_election.cc:302] T 213543cdbe814c89a17a963469922c1b P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:47:13.863292 23527 raft_consensus.cc:1226] T 16ca412a0dae4feebe85dd01bf1a9173 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Rejecting Update request from peer 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20260809 15:47:13.864686 23395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ced718c1a5c4da79cb8c1ebd1435b87" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:47:13.865052 23396 raft_consensus.cc:1226] T f9e7cadcd24d45fcbfa84209225e8f2d P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Rejecting Update request from peer 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 7. Current term is 8. Ops: [5.5-7.6]
I20260809 15:47:13.868961 23283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c48a3049897452383257aa757834fb5" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:47:13.869114 23270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "294d4bcc1e7a477cb56455581277a60f" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:13.782941 28426 raft_consensus.cc:504] T cee02fff2c9c48a19f43eb358a89e7b0 P 61e69695258f48baa984c9d395e8ce3f [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
W20260809 15:47:13.872618 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 45157.5us
I20260809 15:47:13.869755 23270 raft_consensus.cc:2379] T 294d4bcc1e7a477cb56455581277a60f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 61e69695258f48baa984c9d395e8ce3f in current term 6: Already voted for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in this term.
I20260809 15:47:13.828006 23522 raft_consensus.cc:1226] T f9e7cadcd24d45fcbfa84209225e8f2d P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Rejecting Update request from peer 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 7. Current term is 8. Ops: []
I20260809 15:47:13.869675 23283 raft_consensus.cc:2379] T 0c48a3049897452383257aa757834fb5 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 61e69695258f48baa984c9d395e8ce3f in current term 6: Already voted for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in this term.
I20260809 15:47:13.876097 23527 raft_consensus.cc:1261] T 7baa2492df0340c4a5ac241959ce1ac3 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Refusing update from remote peer 67b3aeadfa7a432abe8af4d292e04ab9: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260809 15:47:13.810366 24453 rpcz_store.cc:276] 0809 15:47:12.388528 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:12.395769 (+ 7241us) service_pool.cc:227] Handling call
0809 15:47:13.466125 (+1070356us) spinlock_profiling.cc:243] Waited 3.44 ms on lock 0x7b5800005498. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e613cf11 00007f34e62c0851 00007f34e62c0702 00007f34ef1b0b09 00007f34ee0ec730 00007f34ee1b279b 00007f34ee1b2821 00007f34ee1621b0 00007f34ee161a94 00007f34e7849c04 00007f34e7849b92
0809 15:47:13.769991 (+303866us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7923456}
I20260809 15:47:13.878153 27956 leader_election.cc:288] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:13.798148 27751 raft_consensus.cc:504] T 0acc47ce21c3422698d1ef7277c3ed63 P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:13.879563 28447 raft_consensus.cc:482] T 538a228ddfb54bc6a36528d5ff07597b P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:13.880119 27751 leader_election.cc:288] T 0acc47ce21c3422698d1ef7277c3ed63 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:13.881366 23283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07fcabb569414bddbd39df6cedd24e64" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:13.882527 23283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0acc47ce21c3422698d1ef7277c3ed63" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 9 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:13.883528 28450 raft_consensus.cc:2726] T 213543cdbe814c89a17a963469922c1b P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260809 15:47:13.884240 28337 raft_consensus.cc:3037] T 3924b648461e463b9d7925e5671d68ae P 61e69695258f48baa984c9d395e8ce3f [term 8 FOLLOWER]: Advancing to term 9
I20260809 15:47:13.885789 23472 leader_election.cc:302] T 99bdb93503db4de4a1cfb0161ea0c07c P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:13.886195 28404 consensus_queue.cc:1035] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260809 15:47:13.887787 28430 raft_consensus.cc:2726] T 99bdb93503db4de4a1cfb0161ea0c07c P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260809 15:47:13.889578 28319 raft_consensus.cc:2781] T 538a228ddfb54bc6a36528d5ff07597b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Leader election won for term 7
I20260809 15:47:13.890107 28319 raft_consensus.cc:686] T 538a228ddfb54bc6a36528d5ff07597b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 LEADER]: Becoming Leader. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Running, Role: LEADER
I20260809 15:47:13.890898 28319 consensus_queue.cc:237] T 538a228ddfb54bc6a36528d5ff07597b P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:47:13.891878 28469 log.cc:924] Time spent T cee02fff2c9c48a19f43eb358a89e7b0 P 67b3aeadfa7a432abe8af4d292e04ab9: Append to log took a long time: real 0.109s user 0.000s sys 0.000s
I20260809 15:47:13.892719 28419 raft_consensus.cc:482] T 294d4bcc1e7a477cb56455581277a60f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:13.883080 23283 raft_consensus.cc:2454] T 0acc47ce21c3422698d1ef7277c3ed63 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 8.
I20260809 15:47:13.895596 28284 raft_consensus.cc:1013] T 4b30f28311964a0b9149a50495e83d8f P 65c36d164b0540f099d592f38307ee6c [term 5 LEADER]: attempt to promote peer 61e69695258f48baa984c9d395e8ce3f: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20260809 15:47:13.878638 28412 raft_consensus.cc:3037] T 73f6f44c259547f1b05a3a434616fa54 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Advancing to term 8
I20260809 15:47:13.898178 28458 raft_consensus.cc:482] T 4b30f28311964a0b9149a50495e83d8f P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:13.908140 23517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3622f6e3e3641cea1960e3a31498bb7" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:13.909482 23351 leader_election.cc:302] T 0acc47ce21c3422698d1ef7277c3ed63 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:47:13.910579 23353 leader_election.cc:302] T a3622f6e3e3641cea1960e3a31498bb7 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:47:13.886931 23472 leader_election.cc:302] T df20671a7b874dbb98c0ef28371c1308 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:13.917618 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c48a3049897452383257aa757834fb5" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:47:13.918922 23285 raft_consensus.cc:2396] T 953eea268e17491ca76f3a555860a512 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d for term 9 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20260809 15:47:13.805338 28220 raft_consensus.cc:2726] T 3d880de657e544468c82157769622a26 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260809 15:47:13.926113 23282 raft_consensus.cc:1226] T 563daf42d7784db195ea99c3cb21b370 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Rejecting Update request from peer 61e69695258f48baa984c9d395e8ce3f for earlier term 5. Current term is 6. Ops: []
I20260809 15:47:13.922601 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0acc47ce21c3422698d1ef7277c3ed63" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 9 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:13.927881 23527 raft_consensus.cc:2454] T 0acc47ce21c3422698d1ef7277c3ed63 P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 8.
I20260809 15:47:13.907079 28345 leader_election.cc:288] T a3622f6e3e3641cea1960e3a31498bb7 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:13.960450 27988 leader_election.cc:288] T 0c29937a80724ddb9d07a27534a146f3 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:13.960940 23283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c29937a80724ddb9d07a27534a146f3" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 9 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
W20260809 15:47:13.921396 28370 log.cc:924] Time spent T 05ad7d8fdca64e3da527280ca5be9280 P 65c36d164b0540f099d592f38307ee6c: Append to log took a long time: real 0.060s user 0.000s sys 0.000s
I20260809 15:47:13.805867 28255 consensus_queue.cc:1035] T 907bacfa79d04a55a68c8dbe60056462 P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.124s
I20260809 15:47:13.997685 23517 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "763e58d6ba644eea830a6c1664d39c9b"
dest_uuid: "65c36d164b0540f099d592f38307ee6c"
mode: GRACEFUL
new_leader_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
from {username='slave'} at 127.0.0.1:48888
W20260809 15:47:13.963342 24406 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 227400us
I20260809 15:47:13.999037 23282 raft_consensus.cc:1261] T d3be1e356b254d22b995be32f1c12f6f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 9 FOLLOWER]: Refusing update from remote peer 61e69695258f48baa984c9d395e8ce3f: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20260809 15:47:14.002851 23517 raft_consensus.cc:595] T 763e58d6ba644eea830a6c1664d39c9b P 65c36d164b0540f099d592f38307ee6c [term 7 LEADER]: Received request to transfer leadership to TS c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:13.917009 28383 raft_consensus.cc:686] T 0a51c0baaa6e432997c9d3552af4f254 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 LEADER]: Becoming Leader. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Running, Role: LEADER
I20260809 15:47:14.009907 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ced718c1a5c4da79cb8c1ebd1435b87" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
W20260809 15:47:14.013411 23353 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 102895us
W20260809 15:47:14.015815 23472 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 128955us
I20260809 15:47:14.016466 23472 leader_election.cc:302] T 00a1881267a14ba8b78c81040f17a014 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:13.818238 28309 consensus_queue.cc:1035] T 472816994d814cb198b69e5248697440 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.188s
I20260809 15:47:13.955052 28131 leader_election.cc:288] T 07fcabb569414bddbd39df6cedd24e64 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:14.001858 28455 raft_consensus.cc:2781] T 0acc47ce21c3422698d1ef7277c3ed63 P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260809 15:47:14.006449 28255 consensus_queue.cc:1225] T 907bacfa79d04a55a68c8dbe60056462 P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Peer c0f38c1908414566aa9115bb2cbdd85d log is divergent from this leader: its last log entry 5.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260809 15:47:13.961018 27988 raft_consensus.cc:2726] T a3622f6e3e3641cea1960e3a31498bb7 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260809 15:47:14.005196 28345 raft_consensus.cc:482] T d1a134ba33ea4076bbaac0ae460c17ba P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:47:14.004091 28476 raft_consensus.cc:2726] T df20671a7b874dbb98c0ef28371c1308 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260809 15:47:14.030053 23282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "294d4bcc1e7a477cb56455581277a60f" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:47:14.039377 23283 raft_consensus.cc:1261] T 0c48a3049897452383257aa757834fb5 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Refusing update from remote peer 61e69695258f48baa984c9d395e8ce3f: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260809 15:47:14.031040 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c29937a80724ddb9d07a27534a146f3" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 9 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:14.031590 23518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07fcabb569414bddbd39df6cedd24e64" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:14.023425 28455 raft_consensus.cc:482] T 0acc47ce21c3422698d1ef7277c3ed63 P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:47:14.035071 28302 raft_consensus.cc:482] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Starting leader election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:47:14.041610 23517 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a88421dba80d4d3388aa5eb62e23853b"
dest_uuid: "65c36d164b0540f099d592f38307ee6c"
mode: GRACEFUL
new_leader_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
from {username='slave'} at 127.0.0.1:48888
I20260809 15:47:14.074668 23517 raft_consensus.cc:595] T a88421dba80d4d3388aa5eb62e23853b P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Received request to transfer leadership to TS c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:14.075289 23517 raft_consensus.cc:603] T a88421dba80d4d3388aa5eb62e23853b P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260809 15:47:14.077211 23283 raft_consensus.cc:1226] T 2db057fd10e149a9895dff6ef8c9f038 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 6. Current term is 7. Ops: []
I20260809 15:47:14.019299 28430 raft_consensus.cc:2726] T 00a1881267a14ba8b78c81040f17a014 P 65c36d164b0540f099d592f38307ee6c [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260809 15:47:14.094482 23472 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 72856.9us
I20260809 15:47:14.027633 28450 raft_consensus.cc:482] T 30c1740ef6874df0a332fbd6fd4b03ce P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:14.072361 28455 raft_consensus.cc:3037] T 0acc47ce21c3422698d1ef7277c3ed63 P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Advancing to term 9
W20260809 15:47:14.101670 28486 log.cc:924] Time spent T 998c6cee72cf40228c6dba064c0db2da P c0f38c1908414566aa9115bb2cbdd85d: Append to log took a long time: real 0.277s user 0.001s sys 0.000s
I20260809 15:47:14.071691 28345 raft_consensus.cc:504] T d1a134ba33ea4076bbaac0ae460c17ba P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:14.026394 24453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "907bacfa79d04a55a68c8dbe60056462" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:47:14.102922 28451 raft_consensus.cc:3037] T 4564dfc6d4094c1b9240b367ccecf3b1 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Advancing to term 7
W20260809 15:47:14.076097 28373 log.cc:924] Time spent T d3be1e356b254d22b995be32f1c12f6f P c0f38c1908414566aa9115bb2cbdd85d: Append to log took a long time: real 0.053s user 0.000s sys 0.000s
I20260809 15:47:13.893175 28419 raft_consensus.cc:504] T 294d4bcc1e7a477cb56455581277a60f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:13.880036 28447 raft_consensus.cc:504] T 538a228ddfb54bc6a36528d5ff07597b P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:13.929996 28285 consensus_queue.cc:260] T d3e94b7e50754a81b3d72a07476cb7b9 P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:13.898681 28458 raft_consensus.cc:504] T 4b30f28311964a0b9149a50495e83d8f P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } attrs { replace: true } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } }
I20260809 15:47:14.085101 28383 consensus_queue.cc:237] T 0a51c0baaa6e432997c9d3552af4f254 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:14.182696 23276 raft_consensus.cc:1261] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Refusing update from remote peer 61e69695258f48baa984c9d395e8ce3f: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 8. (term mismatch)
I20260809 15:47:14.181887 23280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee02fff2c9c48a19f43eb358a89e7b0" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:47:14.202375 23276 pending_rounds.cc:76] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9: Aborting all ops after (but not including) 7
I20260809 15:47:14.202888 23276 pending_rounds.cc:97] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9: Aborting uncommitted NO_OP operation due to leader change: 7.8
I20260809 15:47:14.203171 23276 raft_consensus.cc:2866] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 15:47:14.193066 23273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee02fff2c9c48a19f43eb358a89e7b0" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:14.129541 23283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16ca412a0dae4feebe85dd01bf1a9173" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:47:14.074431 28280 raft_consensus.cc:482] T 0a51c0baaa6e432997c9d3552af4f254 P 61e69695258f48baa984c9d395e8ce3f [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:14.207515 23517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee02fff2c9c48a19f43eb358a89e7b0" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:47:14.074712 28437 raft_consensus.cc:2726] T cee02fff2c9c48a19f43eb358a89e7b0 P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260809 15:47:14.209240 23228 leader_election.cc:302] T d0721a4403a440f383318a3cea9d1442 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:14.211491 23526 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ba56725e42f74a1baa4c04d9660727f9"
dest_uuid: "65c36d164b0540f099d592f38307ee6c"
mode: GRACEFUL
new_leader_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
from {username='slave'} at 127.0.0.1:48888
I20260809 15:47:14.214810 23526 raft_consensus.cc:595] T ba56725e42f74a1baa4c04d9660727f9 P 65c36d164b0540f099d592f38307ee6c [term 5 LEADER]: Received request to transfer leadership to TS c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:14.086323 28302 raft_consensus.cc:3037] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Advancing to term 6
I20260809 15:47:14.216153 28285 raft_consensus.cc:3037] T d3e94b7e50754a81b3d72a07476cb7b9 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Advancing to term 8
W20260809 15:47:14.102384 28478 log.cc:924] Time spent T 563daf42d7784db195ea99c3cb21b370 P 61e69695258f48baa984c9d395e8ce3f: Append to log took a long time: real 0.277s user 0.000s sys 0.001s
I20260809 15:47:14.218988 28286 consensus_queue.cc:1035] T d3be1e356b254d22b995be32f1c12f6f P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.002s
I20260809 15:47:14.220597 23400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "907bacfa79d04a55a68c8dbe60056462" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:14.221151 28522 raft_consensus.cc:482] T e71a05ba1d4d42bd95c3f85f9313399e P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:14.222579 28210 consensus_queue.cc:1035] T 0c48a3049897452383257aa757834fb5 P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
W20260809 15:47:14.185333 28492 log.cc:924] Time spent T 4bcac4b24e7a4448abdc934bba5a624e P 65c36d164b0540f099d592f38307ee6c: Append to log took a long time: real 0.091s user 0.000s sys 0.001s
I20260809 15:47:14.232983 21289 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260809 15:47:14.233214 23282 raft_consensus.cc:2379] T 294d4bcc1e7a477cb56455581277a60f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 61e69695258f48baa984c9d395e8ce3f in current term 6: Already voted for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in this term.
I20260809 15:47:14.225418 23517 raft_consensus.cc:3037] T cee02fff2c9c48a19f43eb358a89e7b0 P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Advancing to term 9
W20260809 15:47:14.141479 28495 log.cc:924] Time spent T 294d4bcc1e7a477cb56455581277a60f P 61e69695258f48baa984c9d395e8ce3f: Append to log took a long time: real 0.066s user 0.000s sys 0.000s
W20260809 15:47:14.146766 28410 consensus_meta.cc:287] T 2424fd9bd7f14e508fd106cbfda786a4 P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent flushing consensus metadata: real 0.616s user 0.008s sys 0.000s
I20260809 15:47:14.189189 26966 raft_consensus.cc:1013] T 4b30f28311964a0b9149a50495e83d8f P 65c36d164b0540f099d592f38307ee6c [term 5 LEADER]: attempt to promote peer 61e69695258f48baa984c9d395e8ce3f: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20260809 15:47:14.228267 23400 raft_consensus.cc:2454] T 907bacfa79d04a55a68c8dbe60056462 P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 8.
I20260809 15:47:14.151723 28406 raft_consensus.cc:482] T 4564dfc6d4094c1b9240b367ccecf3b1 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:14.231580 23472 leader_election.cc:302] T 472816994d814cb198b69e5248697440 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:14.193924 28426 leader_election.cc:288] T cee02fff2c9c48a19f43eb358a89e7b0 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 9 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:14.244092 26966 raft_consensus.cc:2726] T 472816994d814cb198b69e5248697440 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260809 15:47:14.149618 28450 raft_consensus.cc:504] T 30c1740ef6874df0a332fbd6fd4b03ce P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:14.246483 28450 leader_election.cc:288] T 30c1740ef6874df0a332fbd6fd4b03ce P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:14.193753 28168 raft_consensus.cc:504] T d1a134ba33ea4076bbaac0ae460c17ba P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:14.248536 28457 raft_consensus.cc:504] T d2d4f749fcfd4fb3bfb8115af3471598 P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:14.249001 28168 leader_election.cc:288] T d1a134ba33ea4076bbaac0ae460c17ba P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 7 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:14.250468 28457 leader_election.cc:288] T d2d4f749fcfd4fb3bfb8115af3471598 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 8 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
W20260809 15:47:14.242372 28438 consensus_meta.cc:287] T a3622f6e3e3641cea1960e3a31498bb7 P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent flushing consensus metadata: real 0.698s user 0.010s sys 0.000s
I20260809 15:47:14.251335 28438 raft_consensus.cc:2726] T a3622f6e3e3641cea1960e3a31498bb7 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260809 15:47:14.252110 23404 raft_consensus.cc:2354] T d2d4f749fcfd4fb3bfb8115af3471598 P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 7. Current term is 8.
I20260809 15:47:14.250859 23283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30c1740ef6874df0a332fbd6fd4b03ce" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 9 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:14.254786 28458 leader_election.cc:288] T 4b30f28311964a0b9149a50495e83d8f P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:14.252113 23517 raft_consensus.cc:2454] T cee02fff2c9c48a19f43eb358a89e7b0 P 65c36d164b0540f099d592f38307ee6c [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 9.
I20260809 15:47:14.259076 23286 raft_consensus.cc:1226] T d1a134ba33ea4076bbaac0ae460c17ba P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 6. Current term is 7. Ops: []
I20260809 15:47:14.261379 26966 consensus_queue.cc:1046] T 213543cdbe814c89a17a963469922c1b P 65c36d164b0540f099d592f38307ee6c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 15:47:14.262099 28455 raft_consensus.cc:504] T 0acc47ce21c3422698d1ef7277c3ed63 P c0f38c1908414566aa9115bb2cbdd85d [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:14.263036 23283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2d4f749fcfd4fb3bfb8115af3471598" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:47:14.224905 28280 raft_consensus.cc:504] T 0a51c0baaa6e432997c9d3552af4f254 P 61e69695258f48baa984c9d395e8ce3f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:14.263505 23273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b30f28311964a0b9149a50495e83d8f" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:14.264132 28437 raft_consensus.cc:3032] T 213543cdbe814c89a17a963469922c1b P 65c36d164b0540f099d592f38307ee6c [term 3 LEADER]: Stepping down as leader of term 3
I20260809 15:47:14.264516 23276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1a134ba33ea4076bbaac0ae460c17ba" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:14.264953 28280 leader_election.cc:288] T 0a51c0baaa6e432997c9d3552af4f254 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:47:14.256259 28345 leader_election.cc:288] T d1a134ba33ea4076bbaac0ae460c17ba P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:14.237432 28522 raft_consensus.cc:504] T e71a05ba1d4d42bd95c3f85f9313399e P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:14.240880 28410 raft_consensus.cc:504] T 2424fd9bd7f14e508fd106cbfda786a4 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:14.264518 28437 raft_consensus.cc:729] T 213543cdbe814c89a17a963469922c1b P 65c36d164b0540f099d592f38307ee6c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:47:14.267747 23270 raft_consensus.cc:1261] T 294d4bcc1e7a477cb56455581277a60f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Refusing update from remote peer 61e69695258f48baa984c9d395e8ce3f: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260809 15:47:14.268198 28437 consensus_queue.cc:260] T 213543cdbe814c89a17a963469922c1b P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:14.268841 28410 leader_election.cc:288] T 2424fd9bd7f14e508fd106cbfda786a4 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 8 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:14.270251 23282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e71a05ba1d4d42bd95c3f85f9313399e" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:14.236963 28416 raft_consensus.cc:2726] T d0721a4403a440f383318a3cea9d1442 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260809 15:47:14.271471 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99bdb93503db4de4a1cfb0161ea0c07c" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:14.264077 23273 raft_consensus.cc:2454] T 4b30f28311964a0b9149a50495e83d8f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 5.
I20260809 15:47:14.263643 23283 raft_consensus.cc:3037] T d2d4f749fcfd4fb3bfb8115af3471598 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Advancing to term 8
I20260809 15:47:14.256323 28419 leader_election.cc:288] T 294d4bcc1e7a477cb56455581277a60f P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:14.276573 23351 leader_election.cc:302] T 4b30f28311964a0b9149a50495e83d8f P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:47:14.276347 28345 raft_consensus.cc:482] T 2424fd9bd7f14e508fd106cbfda786a4 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:47:14.281896 28345 raft_consensus.cc:504] T 2424fd9bd7f14e508fd106cbfda786a4 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:14.282109 24453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e71a05ba1d4d42bd95c3f85f9313399e" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:47:14.267354 28522 leader_election.cc:288] T e71a05ba1d4d42bd95c3f85f9313399e P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:14.284919 28271 raft_consensus.cc:3032] T a3622f6e3e3641cea1960e3a31498bb7 P 65c36d164b0540f099d592f38307ee6c [term 5 LEADER]: Stepping down as leader of term 5
I20260809 15:47:14.285303 28271 raft_consensus.cc:729] T a3622f6e3e3641cea1960e3a31498bb7 P 65c36d164b0540f099d592f38307ee6c [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:47:14.286046 28271 consensus_queue.cc:260] T a3622f6e3e3641cea1960e3a31498bb7 P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:14.288467 23519 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44018 (request call id 325) took 1036 ms. Trace:
I20260809 15:47:14.288851 23519 rpcz_store.cc:276] 0809 15:47:13.251661 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:13.254926 (+ 3265us) service_pool.cc:227] Handling call
0809 15:47:13.255250 (+ 324us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 15:47:13.423068 (+167818us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:47:13.423099 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 15:47:13.466963 (+ 43864us) log.cc:842] Serialized 26 byte log entry
0809 15:47:13.930340 (+463377us) raft_consensus.cc:1631] Marking committed up to 5
0809 15:47:13.930379 (+ 39us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:47:13.930441 (+ 62us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 15:47:14.288262 (+357821us) raft_consensus.cc:1671] finished
0809 15:47:14.288281 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:47:14.288414 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":167431,"raft.queue_time_us":262472,"raft.run_cpu_time_us":91,"raft.run_wall_time_us":4631,"thread_start_us":463107,"threads_started":1,"wal-append.queue_time_us":555967}
I20260809 15:47:14.292102 23345 leader_election.cc:302] T e71a05ba1d4d42bd95c3f85f9313399e P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 61e69695258f48baa984c9d395e8ce3f, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:47:14.291285 28513 raft_consensus.cc:2781] T 4b30f28311964a0b9149a50495e83d8f P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260809 15:47:14.302000 23276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2424fd9bd7f14e508fd106cbfda786a4" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:14.303525 23273 raft_consensus.cc:1261] T c0741ab44d99439abc3aa3bdd27ee160 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260809 15:47:14.299055 23353 leader_election.cc:302] T 0c29937a80724ddb9d07a27534a146f3 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:47:14.294680 28522 raft_consensus.cc:2726] T e71a05ba1d4d42bd95c3f85f9313399e P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260809 15:47:14.307180 23282 raft_consensus.cc:1226] T 2424fd9bd7f14e508fd106cbfda786a4 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 7. Current term is 8. Ops: []
I20260809 15:47:14.327666 23283 raft_consensus.cc:2454] T d2d4f749fcfd4fb3bfb8115af3471598 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 8.
I20260809 15:47:14.328840 23351 leader_election.cc:302] T d2d4f749fcfd4fb3bfb8115af3471598 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters:
W20260809 15:47:14.334950 24406 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 115898us
W20260809 15:47:14.335417 23472 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 74232.2us
I20260809 15:47:14.336457 28319 consensus_queue.cc:1046] T 16ca412a0dae4feebe85dd01bf1a9173 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260809 15:47:14.337361 28319 raft_consensus.cc:3032] T 16ca412a0dae4feebe85dd01bf1a9173 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 LEADER]: Stepping down as leader of term 6
I20260809 15:47:14.337738 28319 raft_consensus.cc:729] T 16ca412a0dae4feebe85dd01bf1a9173 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Running, Role: LEADER
I20260809 15:47:14.338158 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "294d4bcc1e7a477cb56455581277a60f" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:47:14.338476 28319 consensus_queue.cc:260] T 16ca412a0dae4feebe85dd01bf1a9173 P 67b3aeadfa7a432abe8af4d292e04ab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:14.339809 28319 raft_consensus.cc:3037] T 16ca412a0dae4feebe85dd01bf1a9173 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Advancing to term 7
I20260809 15:47:14.341593 28532 raft_consensus.cc:482] T d3e94b7e50754a81b3d72a07476cb7b9 P 61e69695258f48baa984c9d395e8ce3f [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:14.342204 28532 raft_consensus.cc:504] T d3e94b7e50754a81b3d72a07476cb7b9 P 61e69695258f48baa984c9d395e8ce3f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:14.337282 23472 leader_election.cc:302] T 30c1740ef6874df0a332fbd6fd4b03ce P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:14.300381 28325 raft_consensus.cc:482] T 7baa2492df0340c4a5ac241959ce1ac3 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:14.298573 28248 raft_consensus.cc:482] T 6ced718c1a5c4da79cb8c1ebd1435b87 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:47:14.271878 28406 raft_consensus.cc:504] T 4564dfc6d4094c1b9240b367ccecf3b1 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:47:14.355230 23353 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 56260.7us
I20260809 15:47:14.302723 28513 raft_consensus.cc:482] T 4b30f28311964a0b9149a50495e83d8f P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:47:14.358089 23353 leader_election.cc:302] T 07fcabb569414bddbd39df6cedd24e64 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
W20260809 15:47:14.357411 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 148237us
I20260809 15:47:14.360371 23228 leader_election.cc:302] T ba56725e42f74a1baa4c04d9660727f9 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:14.358788 27988 raft_consensus.cc:2726] T 0c29937a80724ddb9d07a27534a146f3 P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260809 15:47:14.373791 23228 leader_election.cc:302] T 998c6cee72cf40228c6dba064c0db2da P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:14.374629 23353 leader_election.cc:302] T 6ae14a74a26f434db08c652e097385c8 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters: 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:47:14.374945 23273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0acc47ce21c3422698d1ef7277c3ed63" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 9 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:47:14.375550 23273 raft_consensus.cc:3037] T 0acc47ce21c3422698d1ef7277c3ed63 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Advancing to term 9
I20260809 15:47:14.376415 28419 raft_consensus.cc:2726] T 998c6cee72cf40228c6dba064c0db2da P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260809 15:47:14.376782 28447 leader_election.cc:288] T 538a228ddfb54bc6a36528d5ff07597b P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:14.377280 28447 raft_consensus.cc:2726] T 07fcabb569414bddbd39df6cedd24e64 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260809 15:47:14.378017 28508 raft_consensus.cc:482] T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:14.379568 23353 leader_election.cc:302] T a88421dba80d4d3388aa5eb62e23853b P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters: 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:47:14.364778 28513 raft_consensus.cc:3037] T 4b30f28311964a0b9149a50495e83d8f P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Advancing to term 6
I20260809 15:47:14.365362 28319 raft_consensus.cc:3037] T ba56725e42f74a1baa4c04d9660727f9 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Advancing to term 6
W20260809 15:47:14.380730 23472 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 43549.1us
I20260809 15:47:14.372102 28280 raft_consensus.cc:482] T 3924b648461e463b9d7925e5671d68ae P 61e69695258f48baa984c9d395e8ce3f [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:14.373564 28345 leader_election.cc:288] T 2424fd9bd7f14e508fd106cbfda786a4 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:14.387841 23282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "538a228ddfb54bc6a36528d5ff07597b" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:14.391577 23228 leader_election.cc:302] T 563daf42d7784db195ea99c3cb21b370 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:14.394119 23528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2d4f749fcfd4fb3bfb8115af3471598" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:47:14.394557 23528 raft_consensus.cc:3037] T d2d4f749fcfd4fb3bfb8115af3471598 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Advancing to term 8
I20260809 15:47:14.399385 28523 raft_consensus.cc:2781] T d2d4f749fcfd4fb3bfb8115af3471598 P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Leader election won for term 8
I20260809 15:47:14.399911 28523 raft_consensus.cc:686] T d2d4f749fcfd4fb3bfb8115af3471598 P c0f38c1908414566aa9115bb2cbdd85d [term 8 LEADER]: Becoming Leader. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:47:14.401160 23521 raft_consensus.cc:1261] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260809 15:47:14.403959 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30c1740ef6874df0a332fbd6fd4b03ce" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 9 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:14.383162 27988 raft_consensus.cc:2781] T 6ae14a74a26f434db08c652e097385c8 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260809 15:47:14.408421 23273 raft_consensus.cc:2454] T 0acc47ce21c3422698d1ef7277c3ed63 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 9.
I20260809 15:47:14.409533 23351 leader_election.cc:302] T 0acc47ce21c3422698d1ef7277c3ed63 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:47:14.415740 23518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b30f28311964a0b9149a50495e83d8f" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:14.418021 28532 leader_election.cc:288] T d3e94b7e50754a81b3d72a07476cb7b9 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:14.418203 28456 consensus_queue.cc:1035] T 7660dfa2ef694927b8eda4100a2bcd9b P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260809 15:47:14.390175 28280 raft_consensus.cc:504] T 3924b648461e463b9d7925e5671d68ae P 61e69695258f48baa984c9d395e8ce3f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:14.418730 28348 raft_consensus.cc:482] T 0a51c0baaa6e432997c9d3552af4f254 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:14.419222 28348 raft_consensus.cc:504] T 0a51c0baaa6e432997c9d3552af4f254 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:14.419857 23472 leader_election.cc:302] T 953eea268e17491ca76f3a555860a512 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:14.420564 28456 raft_consensus.cc:3037] T 953eea268e17491ca76f3a555860a512 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Advancing to term 8
I20260809 15:47:14.421321 28348 leader_election.cc:288] T 0a51c0baaa6e432997c9d3552af4f254 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:14.401643 28345 raft_consensus.cc:2781] T a88421dba80d4d3388aa5eb62e23853b P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260809 15:47:14.422338 28345 raft_consensus.cc:482] T a88421dba80d4d3388aa5eb62e23853b P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:47:14.422683 28345 raft_consensus.cc:3037] T a88421dba80d4d3388aa5eb62e23853b P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Advancing to term 7
I20260809 15:47:14.400782 28523 consensus_queue.cc:237] T d2d4f749fcfd4fb3bfb8115af3471598 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:14.402614 28455 leader_election.cc:288] T 0acc47ce21c3422698d1ef7277c3ed63 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 9 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:14.423496 23521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1a134ba33ea4076bbaac0ae460c17ba" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:14.423631 28455 raft_consensus.cc:2781] T 0acc47ce21c3422698d1ef7277c3ed63 P c0f38c1908414566aa9115bb2cbdd85d [term 9 FOLLOWER]: Leader election won for term 9
I20260809 15:47:14.424153 28455 raft_consensus.cc:686] T 0acc47ce21c3422698d1ef7277c3ed63 P c0f38c1908414566aa9115bb2cbdd85d [term 9 LEADER]: Becoming Leader. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:47:14.382977 28411 consensus_queue.cc:1046] T 2db057fd10e149a9895dff6ef8c9f038 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.025s
I20260809 15:47:14.425145 28415 consensus_queue.cc:1035] T c0741ab44d99439abc3aa3bdd27ee160 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.039s
I20260809 15:47:14.378508 28508 raft_consensus.cc:504] T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:14.426481 28525 raft_consensus.cc:2726] T 30c1740ef6874df0a332fbd6fd4b03ce P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260809 15:47:14.426869 28456 raft_consensus.cc:2726] T 953eea268e17491ca76f3a555860a512 P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260809 15:47:14.427179 28509 consensus_queue.cc:1046] T 2424fd9bd7f14e508fd106cbfda786a4 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: INVALID_TERM, Last received: 7.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.041s
I20260809 15:47:14.427860 28508 leader_election.cc:288] T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:14.359005 28248 raft_consensus.cc:504] T 6ced718c1a5c4da79cb8c1ebd1435b87 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:14.432252 23526 raft_consensus.cc:2396] T 30c1740ef6874df0a332fbd6fd4b03ce P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d for term 9 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
I20260809 15:47:14.402994 28383 raft_consensus.cc:2726] T 563daf42d7784db195ea99c3cb21b370 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260809 15:47:14.427951 24451 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38722 (request call id 541) took 2060 ms. Trace:
I20260809 15:47:14.430814 23273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a51c0baaa6e432997c9d3552af4f254" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:14.372224 28325 raft_consensus.cc:504] T 7baa2492df0340c4a5ac241959ce1ac3 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:14.416893 23228 leader_election.cc:302] T d3be1e356b254d22b995be32f1c12f6f P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:14.424957 28455 consensus_queue.cc:237] T 0acc47ce21c3422698d1ef7277c3ed63 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.3, Last appended by leader: 3, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:14.424896 28411 raft_consensus.cc:3032] T 2db057fd10e149a9895dff6ef8c9f038 P 65c36d164b0540f099d592f38307ee6c [term 6 LEADER]: Stepping down as leader of term 6
I20260809 15:47:14.439158 23353 leader_election.cc:302] T d1a134ba33ea4076bbaac0ae460c17ba P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:47:14.441480 23522 raft_consensus.cc:1261] T d0721a4403a440f383318a3cea9d1442 P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260809 15:47:14.436403 23351 leader_election.cc:302] T 953eea268e17491ca76f3a555860a512 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:47:14.440878 24451 rpcz_store.cc:276] 0809 15:47:12.367363 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:12.407980 (+ 40617us) service_pool.cc:227] Handling call
0809 15:47:12.453618 (+ 45638us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 15:47:13.197188 (+743570us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:47:14.427703 (+1230515us) spinlock_profiling.cc:243] Waited 679 ms on lock 0x7b54005334d4. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e9220f51 00007f34e921f99e 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772 00007f34e764a94c 00007f34e7649d86
0809 15:47:14.427871 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1561626880}
I20260809 15:47:14.443440 24455 raft_consensus.cc:1261] T c0741ab44d99439abc3aa3bdd27ee160 P 61e69695258f48baa984c9d395e8ce3f [term 8 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260809 15:47:14.443668 24463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a51c0baaa6e432997c9d3552af4f254" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:47:14.457248 24463 raft_consensus.cc:2454] T 0a51c0baaa6e432997c9d3552af4f254 P 61e69695258f48baa984c9d395e8ce3f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 7.
I20260809 15:47:14.443867 28248 leader_election.cc:288] T 6ced718c1a5c4da79cb8c1ebd1435b87 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
W20260809 15:47:14.428238 28528 log.cc:924] Time spent T 0c48a3049897452383257aa757834fb5 P 65c36d164b0540f099d592f38307ee6c: Append to log took a long time: real 0.104s user 0.001s sys 0.000s
I20260809 15:47:14.445502 28280 leader_election.cc:288] T 3924b648461e463b9d7925e5671d68ae P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:14.446007 23521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2424fd9bd7f14e508fd106cbfda786a4" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:14.406564 27988 raft_consensus.cc:482] T 6ae14a74a26f434db08c652e097385c8 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Starting leader election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
I20260809 15:47:14.468626 27988 raft_consensus.cc:3037] T 6ae14a74a26f434db08c652e097385c8 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Advancing to term 8
I20260809 15:47:14.469133 23521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3e94b7e50754a81b3d72a07476cb7b9" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:14.469515 28325 leader_election.cc:288] T 7baa2492df0340c4a5ac241959ce1ac3 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:14.469692 23521 raft_consensus.cc:2454] T d3e94b7e50754a81b3d72a07476cb7b9 P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 8.
I20260809 15:47:14.456894 23518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "538a228ddfb54bc6a36528d5ff07597b" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:14.470546 23518 raft_consensus.cc:2454] T 538a228ddfb54bc6a36528d5ff07597b P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 7.
I20260809 15:47:14.471503 28284 consensus_queue.cc:1046] T ba56725e42f74a1baa4c04d9660727f9 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } attrs { replace: true }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 15:47:14.460456 23345 leader_election.cc:302] T 0a51c0baaa6e432997c9d3552af4f254 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:47:14.472518 28284 consensus_queue.cc:1046] T 2db057fd10e149a9895dff6ef8c9f038 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260809 15:47:14.473605 28476 consensus_queue.cc:1035] T c0741ab44d99439abc3aa3bdd27ee160 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20260809 15:47:14.401412 28285 consensus_queue.cc:1046] T 0c29937a80724ddb9d07a27534a146f3 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: INVALID_TERM, Last received: 5.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.038s
I20260809 15:47:14.462472 23228 leader_election.cc:398] T 4b30f28311964a0b9149a50495e83d8f P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 pre-election: Vote denied by peer c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263) with higher term. Message: Invalid argument: T 4b30f28311964a0b9149a50495e83d8f P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 4. Current term is 5.
I20260809 15:47:14.464635 23273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ced718c1a5c4da79cb8c1ebd1435b87" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:14.432669 28456 raft_consensus.cc:3032] T 2424fd9bd7f14e508fd106cbfda786a4 P 65c36d164b0540f099d592f38307ee6c [term 7 LEADER]: Stepping down as leader of term 7
I20260809 15:47:14.467470 24475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3e94b7e50754a81b3d72a07476cb7b9" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:47:14.467649 23522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0acc47ce21c3422698d1ef7277c3ed63" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 9 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:47:14.454754 28411 raft_consensus.cc:729] T 2db057fd10e149a9895dff6ef8c9f038 P 65c36d164b0540f099d592f38307ee6c [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:47:14.436170 28451 raft_consensus.cc:2726] T 4564dfc6d4094c1b9240b367ccecf3b1 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260809 15:47:14.442438 28508 raft_consensus.cc:482] T 73f6f44c259547f1b05a3a434616fa54 P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:14.485373 24475 raft_consensus.cc:2454] T d3e94b7e50754a81b3d72a07476cb7b9 P 61e69695258f48baa984c9d395e8ce3f [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 8.
I20260809 15:47:14.485380 23522 raft_consensus.cc:3037] T 0acc47ce21c3422698d1ef7277c3ed63 P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Advancing to term 9
I20260809 15:47:14.495726 23228 leader_election.cc:401] T 4b30f28311964a0b9149a50495e83d8f P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
W20260809 15:47:14.495338 23353 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 56267.5us
I20260809 15:47:14.484731 28248 consensus_queue.cc:1035] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.013s
I20260809 15:47:14.501012 23528 raft_consensus.cc:2454] T d2d4f749fcfd4fb3bfb8115af3471598 P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 8.
I20260809 15:47:14.500918 23518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7baa2492df0340c4a5ac241959ce1ac3" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:14.501657 23518 raft_consensus.cc:2454] T 7baa2492df0340c4a5ac241959ce1ac3 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 5.
W20260809 15:47:14.504268 24468 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38724 (request call id 562) took 1752 ms (client timeout 1987 ms). Trace:
W20260809 15:47:14.504669 24468 rpcz_store.cc:269] 0809 15:47:12.751865 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:12.751982 (+ 117us) service_pool.cc:227] Handling call
0809 15:47:14.504120 (+1752138us) spinlock_profiling.cc:243] Waited 124 ms on lock 0x7b5400720454. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e92242ae 00007f34ee161c27 00007f34e7849c04 00007f34e7849b92 00007f34e7849ac1 00007f34e7849a3c 00007f34e7848cf2 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:47:14.504216 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":284522880}
I20260809 15:47:14.500612 23520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ced718c1a5c4da79cb8c1ebd1435b87" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:14.507869 23522 raft_consensus.cc:2454] T 0acc47ce21c3422698d1ef7277c3ed63 P 65c36d164b0540f099d592f38307ee6c [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 9.
I20260809 15:47:14.508279 24472 raft_consensus.cc:1261] T 472816994d814cb198b69e5248697440 P 61e69695258f48baa984c9d395e8ce3f [term 6 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260809 15:47:14.509567 23399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7660dfa2ef694927b8eda4100a2bcd9b" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:47:14.510658 23394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "538a228ddfb54bc6a36528d5ff07597b" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:47:14.533033 23345 leader_election.cc:302] T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:47:14.510963 23393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b30f28311964a0b9149a50495e83d8f" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:47:14.569763 28545 raft_consensus.cc:2726] T d1a134ba33ea4076bbaac0ae460c17ba P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260809 15:47:14.569295 28549 raft_consensus.cc:2726] T 953eea268e17491ca76f3a555860a512 P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260809 15:47:14.572867 23394 raft_consensus.cc:2379] T 538a228ddfb54bc6a36528d5ff07597b P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c in current term 7: Already voted for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in this term.
W20260809 15:47:14.581895 23345 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.68:34689, user_credentials={real_user=slave}} blocked reactor thread for 48934.5us
I20260809 15:47:14.577661 23392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cef3e7277d7d4760824a3436ed4a847d" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:47:14.577944 23395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05ad7d8fdca64e3da527280ca5be9280" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:14.477434 28410 raft_consensus.cc:2726] T d3be1e356b254d22b995be32f1c12f6f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260809 15:47:14.582620 23394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ae14a74a26f434db08c652e097385c8" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:14.571782 28406 leader_election.cc:288] T 4564dfc6d4094c1b9240b367ccecf3b1 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
W20260809 15:47:14.587950 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 125951us
I20260809 15:47:14.593621 23397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae6d2c08a7a94f58ac8dd698e3fd456c" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:47:14.592265 23400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0acc47ce21c3422698d1ef7277c3ed63" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:47:14.591912 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a823d35f4f348d1b52ba9e6d80b00d2" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:14.591666 23399 raft_consensus.cc:2454] T 7660dfa2ef694927b8eda4100a2bcd9b P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 6.
I20260809 15:47:14.583811 28522 raft_consensus.cc:2781] T 0a51c0baaa6e432997c9d3552af4f254 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260809 15:47:14.461014 28430 consensus_queue.cc:1046] T d1a134ba33ea4076bbaac0ae460c17ba P 65c36d164b0540f099d592f38307ee6c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.121s
I20260809 15:47:14.465312 28198 consensus_queue.cc:1046] T 0da4d5283c8a46878bd39b85e5c71f66 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.109s
I20260809 15:47:14.500641 28508 raft_consensus.cc:504] T 73f6f44c259547f1b05a3a434616fa54 P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:14.635478 28411 consensus_queue.cc:260] T 2db057fd10e149a9895dff6ef8c9f038 P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:14.647630 23395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5c2e425a74b4e64ae70b5def750434c" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:14.649320 23273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7baa2492df0340c4a5ac241959ce1ac3" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:14.496501 28456 raft_consensus.cc:729] T 2424fd9bd7f14e508fd106cbfda786a4 P 65c36d164b0540f099d592f38307ee6c [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:47:14.656880 28456 consensus_queue.cc:260] T 2424fd9bd7f14e508fd106cbfda786a4 P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.3, Last appended by leader: 3, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:14.658293 28456 raft_consensus.cc:3037] T 2424fd9bd7f14e508fd106cbfda786a4 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Advancing to term 8
I20260809 15:47:14.581422 28525 raft_consensus.cc:465] T 4b30f28311964a0b9149a50495e83d8f P 65c36d164b0540f099d592f38307ee6c [term 5 LEADER]: Not starting pre-election -- already a leader
I20260809 15:47:14.665716 24463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472816994d814cb198b69e5248697440" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f"
I20260809 15:47:14.666342 24463 raft_consensus.cc:2362] T 472816994d814cb198b69e5248697440 P 61e69695258f48baa984c9d395e8ce3f [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 6. Re-sending same reply.
I20260809 15:47:14.655135 23395 raft_consensus.cc:2454] T c5c2e425a74b4e64ae70b5def750434c P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 7.
W20260809 15:47:14.667351 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 211421us
I20260809 15:47:14.647656 28285 raft_consensus.cc:482] T d0721a4403a440f383318a3cea9d1442 P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:14.683413 23472 leader_election.cc:302] T 538a228ddfb54bc6a36528d5ff07597b P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:14.651541 28549 raft_consensus.cc:2781] T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260809 15:47:14.694149 23392 raft_consensus.cc:1226] T 73f6f44c259547f1b05a3a434616fa54 P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 7. Current term is 8. Ops: []
I20260809 15:47:14.649286 28522 raft_consensus.cc:482] T 0a51c0baaa6e432997c9d3552af4f254 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:47:14.581709 28509 consensus_queue.cc:1035] T 4b30f28311964a0b9149a50495e83d8f P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.085s
I20260809 15:47:14.666497 28406 raft_consensus.cc:2726] T 4b30f28311964a0b9149a50495e83d8f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263) with higher term. Message: Invalid argument: T 4b30f28311964a0b9149a50495e83d8f P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 4. Current term is 5.
I20260809 15:47:14.723237 28508 leader_election.cc:288] T 73f6f44c259547f1b05a3a434616fa54 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:14.593308 28302 raft_consensus.cc:504] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 61e69695258f48baa984c9d395e8ce3f [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
W20260809 15:47:14.733047 23353 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 233388us
I20260809 15:47:14.701498 28549 raft_consensus.cc:482] T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:47:14.735565 24463 raft_consensus.cc:1261] T 0a51c0baaa6e432997c9d3552af4f254 P 61e69695258f48baa984c9d395e8ce3f [term 7 FOLLOWER]: Refusing update from remote peer 67b3aeadfa7a432abe8af4d292e04ab9: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260809 15:47:14.736490 28513 raft_consensus.cc:504] T 4b30f28311964a0b9149a50495e83d8f P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } attrs { replace: true } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } }
I20260809 15:47:14.739645 28546 consensus_queue.cc:1035] T d0721a4403a440f383318a3cea9d1442 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.105s
W20260809 15:47:14.741602 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 59473us
I20260809 15:47:14.760802 23228 leader_election.cc:302] T 73f6f44c259547f1b05a3a434616fa54 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:14.664050 27988 raft_consensus.cc:504] T 6ae14a74a26f434db08c652e097385c8 P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:14.665222 28345 raft_consensus.cc:504] T a88421dba80d4d3388aa5eb62e23853b P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:14.750388 24471 raft_consensus.cc:1226] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 61e69695258f48baa984c9d395e8ce3f [term 6 FOLLOWER]: Rejecting Update request from peer c0f38c1908414566aa9115bb2cbdd85d for earlier term 5. Current term is 6. Ops: []
I20260809 15:47:14.749881 23397 raft_consensus.cc:2454] T ae6d2c08a7a94f58ac8dd698e3fd456c P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 8.
I20260809 15:47:14.715911 28522 raft_consensus.cc:3037] T 0a51c0baaa6e432997c9d3552af4f254 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Advancing to term 8
W20260809 15:47:14.751708 23472 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 68553.1us
I20260809 15:47:14.765041 23472 leader_election.cc:302] T 05ad7d8fdca64e3da527280ca5be9280 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:14.732790 28456 raft_consensus.cc:2726] T 538a228ddfb54bc6a36528d5ff07597b P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260809 15:47:14.702970 28285 raft_consensus.cc:504] T d0721a4403a440f383318a3cea9d1442 P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:14.754581 23353 leader_election.cc:302] T 2424fd9bd7f14e508fd106cbfda786a4 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:47:14.771883 23228 leader_election.cc:302] T 763e58d6ba644eea830a6c1664d39c9b P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c
I20260809 15:47:14.773479 23472 leader_election.cc:302] T cef3e7277d7d4760824a3436ed4a847d P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:14.775581 23393 raft_consensus.cc:2354] T 4b30f28311964a0b9149a50495e83d8f P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c for earlier term 5. Current term is 6.
I20260809 15:47:14.792821 23353 leader_election.cc:302] T 538a228ddfb54bc6a36528d5ff07597b P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters: 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:47:14.763336 28549 raft_consensus.cc:3037] T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Advancing to term 9
I20260809 15:47:14.769240 28538 raft_consensus.cc:3037] T 73f6f44c259547f1b05a3a434616fa54 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Advancing to term 8
I20260809 15:47:14.796975 23353 leader_election.cc:302] T 7baa2492df0340c4a5ac241959ce1ac3 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters: 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:47:14.806340 23228 leader_election.cc:302] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:14.817878 23273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73f6f44c259547f1b05a3a434616fa54" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:14.783987 28383 raft_consensus.cc:2781] T 763e58d6ba644eea830a6c1664d39c9b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260809 15:47:14.826570 28383 raft_consensus.cc:482] T 763e58d6ba644eea830a6c1664d39c9b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:47:14.826951 28383 raft_consensus.cc:3037] T 763e58d6ba644eea830a6c1664d39c9b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Advancing to term 8
I20260809 15:47:14.826162 28406 raft_consensus.cc:2726] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260809 15:47:14.827459 28430 raft_consensus.cc:2726] T 05ad7d8fdca64e3da527280ca5be9280 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260809 15:47:14.826582 28563 raft_consensus.cc:2781] T 7baa2492df0340c4a5ac241959ce1ac3 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260809 15:47:14.828293 28563 raft_consensus.cc:482] T 7baa2492df0340c4a5ac241959ce1ac3 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:47:14.828631 28563 raft_consensus.cc:3037] T 7baa2492df0340c4a5ac241959ce1ac3 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Advancing to term 6
I20260809 15:47:14.829070 28430 raft_consensus.cc:2726] T cef3e7277d7d4760824a3436ed4a847d P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260809 15:47:14.795503 28560 raft_consensus.cc:2726] T 2424fd9bd7f14e508fd106cbfda786a4 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260809 15:47:14.841271 23286 raft_consensus.cc:1261] T 4b30f28311964a0b9149a50495e83d8f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260809 15:47:14.839699 28532 raft_consensus.cc:482] T 472816994d814cb198b69e5248697440 P 61e69695258f48baa984c9d395e8ce3f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:14.844744 28532 raft_consensus.cc:504] T 472816994d814cb198b69e5248697440 P 61e69695258f48baa984c9d395e8ce3f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:14.807590 28431 raft_consensus.cc:2781] T 538a228ddfb54bc6a36528d5ff07597b P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260809 15:47:14.776108 28419 consensus_queue.cc:1035] T 0a51c0baaa6e432997c9d3552af4f254 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.049s
W20260809 15:47:14.848791 28572 log.cc:924] Time spent T b3aa6a9a52804df0ba9b0ca3c82f38c3 P c0f38c1908414566aa9115bb2cbdd85d: Append to log took a long time: real 0.064s user 0.000s sys 0.000s
W20260809 15:47:14.849310 28561 log.cc:924] Time spent T 472816994d814cb198b69e5248697440 P 65c36d164b0540f099d592f38307ee6c: Append to log took a long time: real 0.096s user 0.000s sys 0.000s
I20260809 15:47:14.849814 28319 raft_consensus.cc:2726] T ba56725e42f74a1baa4c04d9660727f9 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260809 15:47:14.850728 23283 raft_consensus.cc:1226] T ba56725e42f74a1baa4c04d9660727f9 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 5. Current term is 6. Ops: []
I20260809 15:47:14.851823 28563 raft_consensus.cc:504] T 7baa2492df0340c4a5ac241959ce1ac3 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:14.852273 28285 leader_election.cc:288] T d0721a4403a440f383318a3cea9d1442 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:47:14.852919 28451 raft_consensus.cc:1013] T 4b30f28311964a0b9149a50495e83d8f P 65c36d164b0540f099d592f38307ee6c [term 5 LEADER]: attempt to promote peer 61e69695258f48baa984c9d395e8ce3f: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20260809 15:47:14.761883 28293 raft_consensus.cc:2932] T ba56725e42f74a1baa4c04d9660727f9 P 65c36d164b0540f099d592f38307ee6c [term 5 LEADER]: Committing config change with OpId 5.4: config changed from index -1 to 4, NON_VOTER 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } attrs { replace: true } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } } }
I20260809 15:47:14.853614 23228 leader_election.cc:302] T 0da4d5283c8a46878bd39b85e5c71f66 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
W20260809 15:47:14.854192 23353 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 57292.3us
I20260809 15:47:14.856477 23353 leader_election.cc:302] T 6ced718c1a5c4da79cb8c1ebd1435b87 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:47:14.856981 24475 raft_consensus.cc:1226] T cee02fff2c9c48a19f43eb358a89e7b0 P 61e69695258f48baa984c9d395e8ce3f [term 9 FOLLOWER]: Rejecting Update request from peer 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 8. Current term is 9. Ops: []
I20260809 15:47:14.863515 23286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7baa2492df0340c4a5ac241959ce1ac3" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
W20260809 15:47:14.859957 23472 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 86551.4us
I20260809 15:47:14.864670 23472 leader_election.cc:302] T 6ae14a74a26f434db08c652e097385c8 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:14.861506 23520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73f6f44c259547f1b05a3a434616fa54" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:14.867976 28563 leader_election.cc:288] T 7baa2492df0340c4a5ac241959ce1ac3 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 6 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:14.871881 27988 leader_election.cc:288] T 6ae14a74a26f434db08c652e097385c8 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 8 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:14.872386 23286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ae14a74a26f434db08c652e097385c8" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:47:14.871327 28563 raft_consensus.cc:482] T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:14.877326 28532 leader_election.cc:288] T 472816994d814cb198b69e5248697440 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:14.847865 28431 raft_consensus.cc:482] T 538a228ddfb54bc6a36528d5ff07597b P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:47:14.895961 28431 raft_consensus.cc:3037] T 538a228ddfb54bc6a36528d5ff07597b P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Advancing to term 8
I20260809 15:47:14.896879 28319 raft_consensus.cc:482] T 953eea268e17491ca76f3a555860a512 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:14.897948 28302 leader_election.cc:288] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 6 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:14.898372 28406 consensus_queue.cc:1046] T cee02fff2c9c48a19f43eb358a89e7b0 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260809 15:47:14.899283 28406 raft_consensus.cc:3032] T cee02fff2c9c48a19f43eb358a89e7b0 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 LEADER]: Stepping down as leader of term 8
I20260809 15:47:14.899588 28406 raft_consensus.cc:729] T cee02fff2c9c48a19f43eb358a89e7b0 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Running, Role: LEADER
I20260809 15:47:14.901582 23472 leader_election.cc:302] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:14.877054 28563 raft_consensus.cc:504] T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:14.863230 28560 raft_consensus.cc:2726] T 6ced718c1a5c4da79cb8c1ebd1435b87 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260809 15:47:14.891806 27751 raft_consensus.cc:482] T df20671a7b874dbb98c0ef28371c1308 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
W20260809 15:47:14.893404 23470 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 49517.3us
I20260809 15:47:14.893481 28456 consensus_queue.cc:1035] T 4b30f28311964a0b9149a50495e83d8f P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260809 15:47:14.894399 23286 raft_consensus.cc:1261] T d0721a4403a440f383318a3cea9d1442 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260809 15:47:14.865445 23520 raft_consensus.cc:2454] T 73f6f44c259547f1b05a3a434616fa54 P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 8.
I20260809 15:47:14.897379 28319 raft_consensus.cc:504] T 953eea268e17491ca76f3a555860a512 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:47:14.907718 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 54168.4us
I20260809 15:47:14.896157 27892 raft_consensus.cc:3037] T 0da4d5283c8a46878bd39b85e5c71f66 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Advancing to term 6
I20260809 15:47:14.911751 23228 leader_election.cc:302] T 297c82ed906d4b48b184985eed3de601 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:14.904675 28578 raft_consensus.cc:2726] T 6ae14a74a26f434db08c652e097385c8 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260809 15:47:14.921737 23472 leader_election.cc:302] T 0acc47ce21c3422698d1ef7277c3ed63 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:14.910177 27751 raft_consensus.cc:504] T df20671a7b874dbb98c0ef28371c1308 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:14.925442 23276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0721a4403a440f383318a3cea9d1442" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:14.927701 23276 raft_consensus.cc:2454] T d0721a4403a440f383318a3cea9d1442 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 8.
I20260809 15:47:14.934288 23470 leader_election.cc:302] T d0721a4403a440f383318a3cea9d1442 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:47:14.935019 23472 leader_election.cc:302] T c5c2e425a74b4e64ae70b5def750434c P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters: 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:47:14.919727 28586 raft_consensus.cc:2726] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 65c36d164b0540f099d592f38307ee6c [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260809 15:47:14.942375 23472 leader_election.cc:302] T ae6d2c08a7a94f58ac8dd698e3fd456c P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters: 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:47:14.952790 23404 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55332 (request call id 1496) took 1178 ms. Trace:
I20260809 15:47:14.926368 28546 consensus_queue.cc:1035] T d0721a4403a440f383318a3cea9d1442 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.017s
I20260809 15:47:14.939043 28578 raft_consensus.cc:2726] T 0acc47ce21c3422698d1ef7277c3ed63 P 65c36d164b0540f099d592f38307ee6c [term 9 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260809 15:47:14.953135 23404 rpcz_store.cc:276] 0809 15:47:13.767075 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:13.767652 (+ 577us) service_pool.cc:227] Handling call
0809 15:47:14.945721 (+1178069us) spinlock_profiling.cc:243] Waited 483 ms on lock 0x7b54001cf7d4. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e92242ae 00007f34ee161c27 00007f34e7849c04 00007f34e7849b92 00007f34e7849ac1 00007f34e7849a3c 00007f34e7848cf2 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:47:14.945841 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1111930752}
I20260809 15:47:14.947419 28577 ts_tablet_manager.cc:927] T ba56725e42f74a1baa4c04d9660727f9 P 61e69695258f48baa984c9d395e8ce3f: Initiating tablet copy from peer 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:14.945621 28419 raft_consensus.cc:3037] T 297c82ed906d4b48b184985eed3de601 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Advancing to term 9
I20260809 15:47:14.940446 28576 raft_consensus.cc:482] T 4bcac4b24e7a4448abdc934bba5a624e P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
I20260809 15:47:14.944975 28598 raft_consensus.cc:2781] T c5c2e425a74b4e64ae70b5def750434c P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260809 15:47:14.972010 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3e94b7e50754a81b3d72a07476cb7b9" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:14.972271 23518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3924b648461e463b9d7925e5671d68ae" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:14.973199 23520 raft_consensus.cc:1261] T 294d4bcc1e7a477cb56455581277a60f P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Refusing update from remote peer 61e69695258f48baa984c9d395e8ce3f: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260809 15:47:14.973944 23518 raft_consensus.cc:2454] T 3924b648461e463b9d7925e5671d68ae P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 8.
I20260809 15:47:14.973876 23527 raft_consensus.cc:2454] T d3e94b7e50754a81b3d72a07476cb7b9 P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 8.
I20260809 15:47:14.926196 28406 consensus_queue.cc:260] T cee02fff2c9c48a19f43eb358a89e7b0 P 67b3aeadfa7a432abe8af4d292e04ab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:14.978958 23404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0721a4403a440f383318a3cea9d1442" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:14.987704 23402 pending_rounds.cc:76] T 907bacfa79d04a55a68c8dbe60056462 P c0f38c1908414566aa9115bb2cbdd85d: Aborting all ops after (but not including) 6
I20260809 15:47:14.988023 23402 pending_rounds.cc:97] T 907bacfa79d04a55a68c8dbe60056462 P c0f38c1908414566aa9115bb2cbdd85d: Aborting uncommitted NO_OP operation due to leader change: 5.7
W20260809 15:47:14.981534 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 69845.6us
I20260809 15:47:14.988251 23402 raft_consensus.cc:2866] T 907bacfa79d04a55a68c8dbe60056462 P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 15:47:14.988853 23228 leader_election.cc:302] T 3296ef50be254fc29cfc77d22a8f7a23 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:14.994932 23521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472816994d814cb198b69e5248697440" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:14.999936 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad8a3b25d7b49e5a2142d5709edd0c7" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:47:14.966624 28586 raft_consensus.cc:2781] T ae6d2c08a7a94f58ac8dd698e3fd456c P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Leader election won for term 8
I20260809 15:47:15.001781 24410 leader_election.cc:302] T 3924b648461e463b9d7925e5671d68ae P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c; no voters:
I20260809 15:47:15.010701 24410 leader_election.cc:302] T d3e94b7e50754a81b3d72a07476cb7b9 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c; no voters:
W20260809 15:47:15.000946 23353 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 135235us
I20260809 15:47:15.012583 23353 leader_election.cc:302] T 73f6f44c259547f1b05a3a434616fa54 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:47:15.013865 23392 raft_consensus.cc:1226] T ba56725e42f74a1baa4c04d9660727f9 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20260809 15:47:14.973673 28598 raft_consensus.cc:482] T c5c2e425a74b4e64ae70b5def750434c P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Starting leader election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:47:14.964344 28587 raft_consensus.cc:2781] T d0721a4403a440f383318a3cea9d1442 P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260809 15:47:14.988206 28255 raft_consensus.cc:482] T 3d880de657e544468c82157769622a26 P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
I20260809 15:47:15.066393 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ae14a74a26f434db08c652e097385c8" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:47:15.069085 23353 leader_election.cc:302] T 6ae14a74a26f434db08c652e097385c8 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:47:15.068958 23521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7baa2492df0340c4a5ac241959ce1ac3" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:47:15.072876 23521 raft_consensus.cc:3037] T 7baa2492df0340c4a5ac241959ce1ac3 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Advancing to term 6
I20260809 15:47:15.013312 27493 raft_consensus.cc:2781] T 73f6f44c259547f1b05a3a434616fa54 P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260809 15:47:15.086534 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3924b648461e463b9d7925e5671d68ae" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:15.088191 23527 raft_consensus.cc:2454] T 3924b648461e463b9d7925e5671d68ae P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 8.
I20260809 15:47:15.103385 23353 leader_election.cc:302] T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:47:15.062328 28532 raft_consensus.cc:2781] T d3e94b7e50754a81b3d72a07476cb7b9 P 61e69695258f48baa984c9d395e8ce3f [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260809 15:47:15.102968 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 114181us
I20260809 15:47:15.105448 23527 raft_consensus.cc:1261] T 0acc47ce21c3422698d1ef7277c3ed63 P 65c36d164b0540f099d592f38307ee6c [term 9 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 7 index: 3. Preceding OpId from leader: term: 9 index: 4. (index mismatch)
I20260809 15:47:15.106383 23397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a51c0baaa6e432997c9d3552af4f254" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:15.049677 28302 raft_consensus.cc:2781] T 3924b648461e463b9d7925e5671d68ae P 61e69695258f48baa984c9d395e8ce3f [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260809 15:47:15.031276 27892 raft_consensus.cc:2726] T 0da4d5283c8a46878bd39b85e5c71f66 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260809 15:47:15.115460 23404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3e94b7e50754a81b3d72a07476cb7b9" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:15.063293 28598 raft_consensus.cc:3037] T c5c2e425a74b4e64ae70b5def750434c P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Advancing to term 7
I20260809 15:47:15.071756 28563 leader_election.cc:288] T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:15.013756 28335 consensus_queue.cc:1035] T 294d4bcc1e7a477cb56455581277a60f P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.095s
I20260809 15:47:15.069561 28546 raft_consensus.cc:482] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
I20260809 15:47:15.065690 28586 raft_consensus.cc:686] T ae6d2c08a7a94f58ac8dd698e3fd456c P 65c36d164b0540f099d592f38307ee6c [term 8 LEADER]: Becoming Leader. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:47:14.979702 28576 raft_consensus.cc:504] T 4bcac4b24e7a4448abdc934bba5a624e P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:15.143523 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3924b648461e463b9d7925e5671d68ae" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:47:15.110770 28560 raft_consensus.cc:2781] T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260809 15:47:15.129856 23399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3924b648461e463b9d7925e5671d68ae" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:15.101841 28319 leader_election.cc:288] T 953eea268e17491ca76f3a555860a512 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:15.104538 28532 raft_consensus.cc:482] T d3e94b7e50754a81b3d72a07476cb7b9 P 61e69695258f48baa984c9d395e8ce3f [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:47:15.111027 27493 raft_consensus.cc:482] T 73f6f44c259547f1b05a3a434616fa54 P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:47:15.125126 23282 raft_consensus.cc:1226] T 0da4d5283c8a46878bd39b85e5c71f66 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 5. Current term is 6. Ops: []
I20260809 15:47:15.143206 23395 raft_consensus.cc:1226] T 0c29937a80724ddb9d07a27534a146f3 P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 7. Current term is 8. Ops: [7.3-7.3]
I20260809 15:47:15.145370 23520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df20671a7b874dbb98c0ef28371c1308" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:15.111326 28302 raft_consensus.cc:482] T 3924b648461e463b9d7925e5671d68ae P 61e69695258f48baa984c9d395e8ce3f [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:47:15.110770 28588 raft_consensus.cc:2726] T 6ae14a74a26f434db08c652e097385c8 P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260809 15:47:15.151028 23228 leader_election.cc:302] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:15.159029 23521 raft_consensus.cc:2454] T 7baa2492df0340c4a5ac241959ce1ac3 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 6.
I20260809 15:47:15.087740 28255 raft_consensus.cc:504] T 3d880de657e544468c82157769622a26 P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:15.142274 27892 raft_consensus.cc:2726] T 3296ef50be254fc29cfc77d22a8f7a23 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 15:47:15.116310 28293 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.BulkChangeConfig from 127.0.0.1:44010 (request call id 7) took 3401 ms. Trace:
I20260809 15:47:15.163540 23353 leader_election.cc:302] T 7baa2492df0340c4a5ac241959ce1ac3 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters: 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:47:15.173692 28602 raft_consensus.cc:2781] T 7baa2492df0340c4a5ac241959ce1ac3 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Leader election won for term 6
I20260809 15:47:15.174207 28437 raft_consensus.cc:3037] T 213543cdbe814c89a17a963469922c1b P 65c36d164b0540f099d592f38307ee6c [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:47:15.116068 28421 consensus_queue.cc:1035] T 0acc47ce21c3422698d1ef7277c3ed63 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.042s
I20260809 15:47:15.164171 28302 raft_consensus.cc:3037] T 3924b648461e463b9d7925e5671d68ae P 61e69695258f48baa984c9d395e8ce3f [term 9 FOLLOWER]: Advancing to term 10
I20260809 15:47:15.148343 28560 raft_consensus.cc:482] T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:47:15.166921 28580 raft_consensus.cc:482] T 998c6cee72cf40228c6dba064c0db2da P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:47:15.179232 28560 raft_consensus.cc:3037] T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d [term 9 FOLLOWER]: Advancing to term 10
W20260809 15:47:15.167834 28601 log.cc:924] Time spent T d2d4f749fcfd4fb3bfb8115af3471598 P c0f38c1908414566aa9115bb2cbdd85d: Append to log took a long time: real 0.054s user 0.000s sys 0.000s
I20260809 15:47:15.159986 27493 raft_consensus.cc:3037] T 73f6f44c259547f1b05a3a434616fa54 P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Advancing to term 9
I20260809 15:47:15.180390 28523 consensus_queue.cc:1046] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20260809 15:47:15.146973 27751 leader_election.cc:288] T df20671a7b874dbb98c0ef28371c1308 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:15.145869 28546 raft_consensus.cc:504] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
W20260809 15:47:15.168591 28604 log.cc:924] Time spent T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 65c36d164b0540f099d592f38307ee6c: Append to log took a long time: real 0.067s user 0.000s sys 0.000s
I20260809 15:47:15.183645 28546 leader_election.cc:288] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:15.184562 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f9d6565aa4c45a3b3f01210d0eb71f7" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:47:15.174180 28602 raft_consensus.cc:686] T 7baa2492df0340c4a5ac241959ce1ac3 P c0f38c1908414566aa9115bb2cbdd85d [term 6 LEADER]: Becoming Leader. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:47:15.156786 28621 raft_consensus.cc:482] T 2db057fd10e149a9895dff6ef8c9f038 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:15.187098 28621 raft_consensus.cc:504] T 2db057fd10e149a9895dff6ef8c9f038 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:15.188340 28586 consensus_queue.cc:237] T ae6d2c08a7a94f58ac8dd698e3fd456c P 65c36d164b0540f099d592f38307ee6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:15.189234 28210 raft_consensus.cc:482] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 61e69695258f48baa984c9d395e8ce3f [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:47:15.189724 28210 raft_consensus.cc:504] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 61e69695258f48baa984c9d395e8ce3f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:15.160358 28383 raft_consensus.cc:504] T 763e58d6ba644eea830a6c1664d39c9b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:15.194183 23519 raft_consensus.cc:1261] T d2d4f749fcfd4fb3bfb8115af3471598 P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260809 15:47:15.169869 28255 leader_election.cc:288] T 3d880de657e544468c82157769622a26 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:15.186213 28602 consensus_queue.cc:237] T 7baa2492df0340c4a5ac241959ce1ac3 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:15.156576 28532 raft_consensus.cc:3037] T d3e94b7e50754a81b3d72a07476cb7b9 P 61e69695258f48baa984c9d395e8ce3f [term 8 FOLLOWER]: Advancing to term 9
I20260809 15:47:15.200966 23393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad8a3b25d7b49e5a2142d5709edd0c7" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:47:15.157294 28319 raft_consensus.cc:482] T a88421dba80d4d3388aa5eb62e23853b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:15.207307 28629 raft_consensus.cc:2726] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260809 15:47:15.211750 23472 leader_election.cc:398] T 4b30f28311964a0b9149a50495e83d8f P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 5 election: Vote denied by peer c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263) with higher term. Message: Invalid argument: T 4b30f28311964a0b9149a50495e83d8f P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c for earlier term 5. Current term is 6.
W20260809 15:47:15.215231 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 69922.2us
I20260809 15:47:15.215947 23228 leader_election.cc:302] T 07fcabb569414bddbd39df6cedd24e64 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c
I20260809 15:47:15.197885 28538 raft_consensus.cc:2726] T 73f6f44c259547f1b05a3a434616fa54 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260809 15:47:15.221803 23393 raft_consensus.cc:1226] T 4b30f28311964a0b9149a50495e83d8f P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 5. Current term is 6. Ops: [5.5-5.5]
I20260809 15:47:15.230240 23393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472816994d814cb198b69e5248697440" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:15.247159 23273 raft_consensus.cc:2454] T 73f6f44c259547f1b05a3a434616fa54 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 8.
W20260809 15:47:15.252393 23223 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.68:34689, user_credentials={real_user=slave}} blocked reactor thread for 53460.3us
I20260809 15:47:15.200191 28546 consensus_queue.cc:1035] T d2d4f749fcfd4fb3bfb8115af3471598 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.048s
I20260809 15:47:15.255836 23228 leader_election.cc:302] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters: 61e69695258f48baa984c9d395e8ce3f
W20260809 15:47:15.262295 24410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 122333us
I20260809 15:47:15.264662 23519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d880de657e544468c82157769622a26" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:15.265359 23519 raft_consensus.cc:2379] T 3d880de657e544468c82157769622a26 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 61e69695258f48baa984c9d395e8ce3f in current term 5: Already voted for candidate 65c36d164b0540f099d592f38307ee6c in this term.
I20260809 15:47:15.246953 28210 leader_election.cc:288] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:15.211594 28319 raft_consensus.cc:504] T a88421dba80d4d3388aa5eb62e23853b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:15.270090 23520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3aa6a9a52804df0ba9b0ca3c82f38c3" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:15.254083 28383 leader_election.cc:288] T 763e58d6ba644eea830a6c1664d39c9b P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 8 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:15.200083 28580 raft_consensus.cc:504] T 998c6cee72cf40228c6dba064c0db2da P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:15.288677 23395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3aa6a9a52804df0ba9b0ca3c82f38c3" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:15.254487 27892 consensus_queue.cc:1046] T 30c1740ef6874df0a332fbd6fd4b03ce P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.022s
W20260809 15:47:15.303359 24406 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 70845.8us
I20260809 15:47:15.257499 28522 raft_consensus.cc:504] T 0a51c0baaa6e432997c9d3552af4f254 P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:15.293184 28284 consensus_queue.cc:1046] T 4b30f28311964a0b9149a50495e83d8f P 65c36d164b0540f099d592f38307ee6c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: INVALID_TERM, Last received: 3.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.017s
I20260809 15:47:15.301070 28383 raft_consensus.cc:2781] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260809 15:47:15.289904 28621 leader_election.cc:288] T 2db057fd10e149a9895dff6ef8c9f038 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:15.297806 28345 leader_election.cc:288] T a88421dba80d4d3388aa5eb62e23853b P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 7 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:15.367911 23397 raft_consensus.cc:2379] T 0a51c0baaa6e432997c9d3552af4f254 P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 61e69695258f48baa984c9d395e8ce3f in current term 8: Already voted for candidate c0f38c1908414566aa9115bb2cbdd85d in this term.
I20260809 15:47:15.368284 23277 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53554 (request call id 1816) took 2183 ms. Trace:
I20260809 15:47:15.369095 23395 raft_consensus.cc:1226] T 0da4d5283c8a46878bd39b85e5c71f66 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 5. Current term is 6. Ops: []
W20260809 15:47:15.373113 24410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 98977.5us
I20260809 15:47:15.370621 23393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bcac4b24e7a4448abdc934bba5a624e" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:15.371426 23277 rpcz_store.cc:276] 0809 15:47:13.133435 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:13.133581 (+ 146us) service_pool.cc:227] Handling call
0809 15:47:13.154977 (+ 21396us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:47:13.644633 (+489656us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:47:15.316994 (+1672361us) spinlock_profiling.cc:243] Waited 472 ms on lock 0x7b54001540d4. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e9220f51 00007f34e921f99e 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772 00007f34e764a94c 00007f34e7649d86
0809 15:47:15.317162 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1085970304}
I20260809 15:47:15.371618 28383 raft_consensus.cc:482] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:47:15.371546 27892 raft_consensus.cc:3032] T 30c1740ef6874df0a332fbd6fd4b03ce P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 LEADER]: Stepping down as leader of term 7
I20260809 15:47:15.369431 28612 raft_consensus.cc:482] T f7715d71dbac447b9351fc5eb36c5e80 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
W20260809 15:47:15.395192 28549 consensus_meta.cc:287] T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d: Time spent flushing consensus metadata: real 0.575s user 0.007s sys 0.002s
I20260809 15:47:15.420899 23277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f9d6565aa4c45a3b3f01210d0eb71f7" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:15.408056 27892 raft_consensus.cc:729] T 30c1740ef6874df0a332fbd6fd4b03ce P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Running, Role: LEADER
I20260809 15:47:15.388247 28431 raft_consensus.cc:504] T 538a228ddfb54bc6a36528d5ff07597b P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:15.397147 28631 consensus_queue.cc:1046] T ba56725e42f74a1baa4c04d9660727f9 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } attrs { replace: true }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.329s
I20260809 15:47:15.408300 28383 raft_consensus.cc:3037] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Advancing to term 9
I20260809 15:47:16.248062 24410 leader_election.cc:302] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:15.198634 28293 rpcz_store.cc:276] 0809 15:47:11.701565 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:11.701731 (+ 166us) service_pool.cc:227] Handling call
0809 15:47:11.826770 (+125039us) log.cc:842] Serialized 506 byte log entry
0809 15:47:15.091986 (+3265216us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":5099,"mutex_wait_us":47587,"raft.queue_time_us":48592,"raft.run_cpu_time_us":820,"raft.run_wall_time_us":819,"thread_start_us":358008,"threads_started":1,"wal-append.queue_time_us":494010,"wal-append.run_cpu_time_us":1593,"wal-append.run_wall_time_us":1003864}
I20260809 15:47:16.251591 23522 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56766 (request call id 2170) took 1059 ms. Trace:
I20260809 15:47:15.565578 23277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a88421dba80d4d3388aa5eb62e23853b" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
W20260809 15:47:15.862931 24404 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.667s user 0.006s sys 0.006s
W20260809 15:47:16.296774 24404 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.101s user 0.011s sys 0.011s
W20260809 15:47:15.711337 23472 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 338240us
W20260809 15:47:15.614107 28634 log.cc:924] Time spent T 907bacfa79d04a55a68c8dbe60056462 P c0f38c1908414566aa9115bb2cbdd85d: Append to log took a long time: real 0.323s user 0.000s sys 0.001s
I20260809 15:47:16.291476 24410 leader_election.cc:302] T 472816994d814cb198b69e5248697440 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
W20260809 15:47:15.786589 28657 log.cc:924] Time spent T d0721a4403a440f383318a3cea9d1442 P c0f38c1908414566aa9115bb2cbdd85d: Append to log took a long time: real 0.470s user 0.000s sys 0.001s
I20260809 15:47:16.296066 23522 rpcz_store.cc:276] 0809 15:47:15.191698 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:15.192121 (+ 423us) service_pool.cc:227] Handling call
0809 15:47:15.192435 (+ 314us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 15:47:15.192758 (+ 323us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:47:15.192783 (+ 25us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 15:47:15.193187 (+ 404us) log.cc:842] Serialized 26 byte log entry
0809 15:47:15.294225 (+101038us) raft_consensus.cc:1631] Marking committed up to 3
0809 15:47:15.294282 (+ 57us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:47:15.294385 (+ 103us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 15:47:15.379699 (+ 85314us) raft_consensus.cc:1671] finished
0809 15:47:15.379719 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:47:16.251524 (+871805us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":342,"raft.run_cpu_time_us":63,"raft.run_wall_time_us":62,"thread_start_us":100591,"threads_started":1,"wal-append.queue_time_us":184637}
I20260809 15:47:16.306121 24410 leader_election.cc:302] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:15.536880 27957 raft_consensus.cc:482] T 213543cdbe814c89a17a963469922c1b P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:16.311854 28476 raft_consensus.cc:3032] T d1a134ba33ea4076bbaac0ae460c17ba P 65c36d164b0540f099d592f38307ee6c [term 6 LEADER]: Stepping down as leader of term 6
I20260809 15:47:16.312353 28431 leader_election.cc:288] T 538a228ddfb54bc6a36528d5ff07597b P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 8 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:16.312183 27957 raft_consensus.cc:504] T 213543cdbe814c89a17a963469922c1b P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:16.312929 28431 raft_consensus.cc:482] T a3622f6e3e3641cea1960e3a31498bb7 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 15:47:15.985340 23228 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.725s user 0.000s sys 0.016s
W20260809 15:47:16.240953 28598 consensus_meta.cc:287] T c5c2e425a74b4e64ae70b5def750434c P 65c36d164b0540f099d592f38307ee6c: Time spent flushing consensus metadata: real 1.123s user 0.008s sys 0.000s
I20260809 15:47:16.313208 23283 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53554 (request call id 1833) took 1927 ms. Trace:
I20260809 15:47:16.313550 28598 raft_consensus.cc:504] T c5c2e425a74b4e64ae70b5def750434c P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:16.314504 28650 raft_consensus.cc:2726] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 61e69695258f48baa984c9d395e8ce3f [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260809 15:47:16.315567 28598 leader_election.cc:288] T c5c2e425a74b4e64ae70b5def750434c P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 7 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:47:15.417023 28576 leader_election.cc:288] T 4bcac4b24e7a4448abdc934bba5a624e P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:16.315800 28440 raft_consensus.cc:482] T e14d118b2eaa42aa8a7cd4025aba7128 P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
W20260809 15:47:16.316392 23228 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.056s user 0.000s sys 0.019s
W20260809 15:47:16.316963 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 1.06112e+06us
I20260809 15:47:16.317297 28210 raft_consensus.cc:2726] T 472816994d814cb198b69e5248697440 P 61e69695258f48baa984c9d395e8ce3f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260809 15:47:16.317613 23228 leader_election.cc:302] T 99bdb93503db4de4a1cfb0161ea0c07c P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:16.317883 28210 raft_consensus.cc:2726] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 61e69695258f48baa984c9d395e8ce3f [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260809 15:47:16.318392 28629 raft_consensus.cc:2726] T 99bdb93503db4de4a1cfb0161ea0c07c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260809 15:47:16.301695 28538 raft_consensus.cc:482] T 05ad7d8fdca64e3da527280ca5be9280 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:47:16.319725 28538 raft_consensus.cc:504] T 05ad7d8fdca64e3da527280ca5be9280 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:16.319914 24453 raft_consensus.cc:1226] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 61e69695258f48baa984c9d395e8ce3f [term 6 FOLLOWER]: Rejecting Update request from peer c0f38c1908414566aa9115bb2cbdd85d for earlier term 5. Current term is 6. Ops: []
I20260809 15:47:16.321719 28538 leader_election.cc:288] T 05ad7d8fdca64e3da527280ca5be9280 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:16.312234 28476 raft_consensus.cc:729] T d1a134ba33ea4076bbaac0ae460c17ba P 65c36d164b0540f099d592f38307ee6c [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:47:16.319715 23228 leader_election.cc:398] T d2d4f749fcfd4fb3bfb8115af3471598 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 7 election: Vote denied by peer c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263) with higher term. Message: Invalid argument: T d2d4f749fcfd4fb3bfb8115af3471598 P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 7. Current term is 8.
I20260809 15:47:16.322548 23228 leader_election.cc:401] T d2d4f749fcfd4fb3bfb8115af3471598 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 7 election: Cancelling election due to peer responding with higher term
I20260809 15:47:15.410069 28626 raft_consensus.cc:2781] T 07fcabb569414bddbd39df6cedd24e64 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260809 15:47:16.313568 23283 rpcz_store.cc:276] 0809 15:47:14.386162 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:14.386276 (+ 114us) service_pool.cc:227] Handling call
0809 15:47:14.386532 (+ 256us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:47:14.851083 (+464551us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:47:16.312976 (+1461893us) spinlock_profiling.cc:243] Waited 464 ms on lock 0x7b54002f49d4. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e9220f51 00007f34e921f99e 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772 00007f34e764a94c 00007f34e7649d86
0809 15:47:16.313130 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1067169664}
I20260809 15:47:16.323088 28626 raft_consensus.cc:482] T 07fcabb569414bddbd39df6cedd24e64 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:47:16.323556 28626 raft_consensus.cc:3037] T 07fcabb569414bddbd39df6cedd24e64 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Advancing to term 8
I20260809 15:47:16.323850 23283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df20671a7b874dbb98c0ef28371c1308" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
W20260809 15:47:15.882877 28662 log.cc:924] Time spent T ae6d2c08a7a94f58ac8dd698e3fd456c P 65c36d164b0540f099d592f38307ee6c: Append to log took a long time: real 0.528s user 0.001s sys 0.000s
I20260809 15:47:16.324597 28538 raft_consensus.cc:2726] T d2d4f749fcfd4fb3bfb8115af3471598 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Leader election lost for term 7. Reason: Vote denied by peer c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263) with higher term. Message: Invalid argument: T d2d4f749fcfd4fb3bfb8115af3471598 P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 7. Current term is 8.
I20260809 15:47:16.324764 23283 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53562 (request call id 1990) took 1214 ms. Trace:
W20260809 15:47:15.841410 27493 consensus_meta.cc:287] T 73f6f44c259547f1b05a3a434616fa54 P c0f38c1908414566aa9115bb2cbdd85d: Time spent flushing consensus metadata: real 0.660s user 0.006s sys 0.002s
W20260809 15:47:15.869184 28302 consensus_meta.cc:287] T 3924b648461e463b9d7925e5671d68ae P 61e69695258f48baa984c9d395e8ce3f: Time spent flushing consensus metadata: real 0.693s user 0.005s sys 0.004s
W20260809 15:47:15.784797 28542 log.cc:924] Time spent T 0a51c0baaa6e432997c9d3552af4f254 P 67b3aeadfa7a432abe8af4d292e04ab9: Append to log took a long time: real 0.137s user 0.000s sys 0.000s
I20260809 15:47:16.325465 28302 raft_consensus.cc:504] T 3924b648461e463b9d7925e5671d68ae P 61e69695258f48baa984c9d395e8ce3f [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:16.325838 28538 consensus_queue.cc:1035] T cee02fff2c9c48a19f43eb358a89e7b0 P 67b3aeadfa7a432abe8af4d292e04ab9 [NON_LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260809 15:47:16.326603 24468 raft_consensus.cc:2379] T 3924b648461e463b9d7925e5671d68ae P 61e69695258f48baa984c9d395e8ce3f [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d in current term 10: Already voted for candidate 61e69695258f48baa984c9d395e8ce3f in this term.
I20260809 15:47:16.303140 23277 raft_consensus.cc:3037] T a88421dba80d4d3388aa5eb62e23853b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Advancing to term 7
W20260809 15:47:16.251019 24406 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.20.202.65:35459 timed out after 1.575s (SENT) [suppressed 36 similar messages]
I20260809 15:47:16.325174 23283 rpcz_store.cc:276] 0809 15:47:15.110635 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:16.323575 (+1212940us) service_pool.cc:227] Handling call
0809 15:47:16.324553 (+ 978us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:47:15.801265 28580 leader_election.cc:288] T 998c6cee72cf40228c6dba064c0db2da P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:47:15.421950 28636 consensus_queue.cc:1046] T f9e7cadcd24d45fcbfa84209225e8f2d P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 2, Time since last communication: 1.054s
I20260809 15:47:16.327057 28631 raft_consensus.cc:482] T 30c1740ef6874df0a332fbd6fd4b03ce P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:16.340811 28302 leader_election.cc:288] T 3924b648461e463b9d7925e5671d68ae P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 10 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:15.584218 28319 leader_election.cc:288] T a88421dba80d4d3388aa5eb62e23853b P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:16.308736 27892 consensus_queue.cc:260] T 30c1740ef6874df0a332fbd6fd4b03ce P 67b3aeadfa7a432abe8af4d292e04ab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:16.343950 27892 raft_consensus.cc:3037] T 30c1740ef6874df0a332fbd6fd4b03ce P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Advancing to term 8
I20260809 15:47:15.420558 28549 raft_consensus.cc:504] T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:16.347817 23404 raft_consensus.cc:2379] T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 61e69695258f48baa984c9d395e8ce3f in current term 9: Already voted for candidate c0f38c1908414566aa9115bb2cbdd85d in this term.
I20260809 15:47:16.322198 28587 raft_consensus.cc:482] T d0721a4403a440f383318a3cea9d1442 P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Starting leader election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:47:16.348382 28587 raft_consensus.cc:3037] T d0721a4403a440f383318a3cea9d1442 P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Advancing to term 9
I20260809 15:47:16.349400 28549 leader_election.cc:288] T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 9 election: Requested vote from peers 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:16.309741 28293 raft_consensus.cc:482] T 99bdb93503db4de4a1cfb0161ea0c07c P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:47:16.350140 28293 raft_consensus.cc:504] T 99bdb93503db4de4a1cfb0161ea0c07c P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:16.352035 28293 leader_election.cc:288] T 99bdb93503db4de4a1cfb0161ea0c07c P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:47:16.310788 23402 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52856 (request call id 411) took 1330 ms. Trace:
I20260809 15:47:16.229410 28425 raft_consensus.cc:482] T 3d880de657e544468c82157769622a26 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 15:47:15.872366 28656 log.cc:924] Time spent T 4bcac4b24e7a4448abdc934bba5a624e P 61e69695258f48baa984c9d395e8ce3f: Append to log took a long time: real 0.517s user 0.001s sys 0.000s
W20260809 15:47:16.243786 28560 consensus_meta.cc:287] T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d: Time spent flushing consensus metadata: real 1.064s user 0.007s sys 0.002s
W20260809 15:47:16.327977 24406 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 76986.5us
I20260809 15:47:16.353523 28425 raft_consensus.cc:504] T 3d880de657e544468c82157769622a26 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:16.338689 24468 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38724 (request call id 575) took 1215 ms. Trace:
I20260809 15:47:16.354507 24468 rpcz_store.cc:276] 0809 15:47:15.122854 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:15.122981 (+ 127us) service_pool.cc:227] Handling call
0809 15:47:16.338520 (+1215539us) spinlock_profiling.cc:243] Waited 1.18 s on lock 0x7b54005a05d4. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e92242ae 00007f34ee161c27 00007f34e7849c04 00007f34e7849b92 00007f34e7849ac1 00007f34e7849a3c 00007f34e7848cf2 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:47:16.338637 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2719105792}
I20260809 15:47:16.355418 28425 leader_election.cc:288] T 3d880de657e544468c82157769622a26 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:16.352622 23402 rpcz_store.cc:276] 0809 15:47:14.980321 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:14.980461 (+ 140us) service_pool.cc:227] Handling call
0809 15:47:14.987531 (+ 7070us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 15:47:14.998865 (+ 11334us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:47:14.998890 (+ 25us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0809 15:47:14.999307 (+ 417us) log.cc:842] Serialized 52 byte log entry
0809 15:47:15.140314 (+141007us) raft_consensus.cc:1631] Marking committed up to 6
0809 15:47:15.140348 (+ 34us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:47:15.140409 (+ 61us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 15:47:16.310599 (+1170190us) raft_consensus.cc:1671] finished
0809 15:47:16.310618 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:47:16.310736 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":338,"raft.run_cpu_time_us":92,"raft.run_wall_time_us":93,"thread_start_us":76400,"threads_started":1,"wal-append.queue_time_us":244398}
I20260809 15:47:16.339495 27493 raft_consensus.cc:504] T 73f6f44c259547f1b05a3a434616fa54 P c0f38c1908414566aa9115bb2cbdd85d [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:16.357314 28290 raft_consensus.cc:482] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:47:16.322832 28476 consensus_queue.cc:260] T d1a134ba33ea4076bbaac0ae460c17ba P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:16.357812 28290 raft_consensus.cc:504] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:16.358546 28476 raft_consensus.cc:3037] T d1a134ba33ea4076bbaac0ae460c17ba P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Advancing to term 7
I20260809 15:47:16.353259 28560 raft_consensus.cc:504] T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:16.360940 28290 leader_election.cc:288] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
W20260809 15:47:16.353390 28419 consensus_meta.cc:287] T 297c82ed906d4b48b184985eed3de601 P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent flushing consensus metadata: real 1.386s user 0.006s sys 0.003s
I20260809 15:47:16.308100 28522 leader_election.cc:288] T 0a51c0baaa6e432997c9d3552af4f254 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 8 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:16.342404 23283 raft_consensus.cc:1261] T 0acc47ce21c3422698d1ef7277c3ed63 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 9 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 7 index: 3. Preceding OpId from leader: term: 9 index: 4. (index mismatch)
I20260809 15:47:16.342314 28631 raft_consensus.cc:504] T 30c1740ef6874df0a332fbd6fd4b03ce P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:16.316352 28440 raft_consensus.cc:504] T e14d118b2eaa42aa8a7cd4025aba7128 P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:16.358924 27493 leader_election.cc:288] T 73f6f44c259547f1b05a3a434616fa54 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 9 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:16.366101 23283 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53562 (request call id 1991) took 1222 ms. Trace:
I20260809 15:47:16.367519 23283 rpcz_store.cc:276] 0809 15:47:15.143844 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:16.341846 (+1198002us) service_pool.cc:227] Handling call
0809 15:47:16.342131 (+ 285us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:47:16.365774 (+ 23643us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:47:16.366045 (+ 271us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:47:16.368237 28440 leader_election.cc:288] T e14d118b2eaa42aa8a7cd4025aba7128 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:16.362159 23399 raft_consensus.cc:2379] T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 61e69695258f48baa984c9d395e8ce3f in current term 10: Already voted for candidate c0f38c1908414566aa9115bb2cbdd85d in this term.
I20260809 15:47:16.341008 28636 raft_consensus.cc:3032] T f9e7cadcd24d45fcbfa84209225e8f2d P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 LEADER]: Stepping down as leader of term 7
I20260809 15:47:16.368649 28631 leader_election.cc:288] T 30c1740ef6874df0a332fbd6fd4b03ce P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
W20260809 15:47:16.352789 28532 consensus_meta.cc:287] T d3e94b7e50754a81b3d72a07476cb7b9 P 61e69695258f48baa984c9d395e8ce3f: Time spent flushing consensus metadata: real 1.136s user 0.000s sys 0.008s
I20260809 15:47:16.369022 28636 raft_consensus.cc:729] T f9e7cadcd24d45fcbfa84209225e8f2d P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Running, Role: LEADER
I20260809 15:47:16.369189 28419 raft_consensus.cc:2726] T 297c82ed906d4b48b184985eed3de601 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260809 15:47:16.368924 23283 raft_consensus.cc:1226] T 763e58d6ba644eea830a6c1664d39c9b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 7. Current term is 8. Ops: []
I20260809 15:47:15.424175 28597 consensus_queue.cc:1046] T 0da4d5283c8a46878bd39b85e5c71f66 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.905s
I20260809 15:47:16.292536 28563 raft_consensus.cc:482] T f9e7cadcd24d45fcbfa84209225e8f2d P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:16.327069 23228 leader_election.cc:302] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c
I20260809 15:47:16.188352 24455 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38734 (request call id 425) took 1070 ms. Trace:
I20260809 15:47:16.372154 23283 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53554 (request call id 1841) took 1203 ms. Trace:
I20260809 15:47:16.314141 27957 leader_election.cc:288] T 213543cdbe814c89a17a963469922c1b P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:15.424487 23351 leader_election.cc:302] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 61e69695258f48baa984c9d395e8ce3f, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:47:16.380223 23283 rpcz_store.cc:276] 0809 15:47:15.168282 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:16.368308 (+1200026us) service_pool.cc:227] Handling call
0809 15:47:16.368615 (+ 307us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:47:16.371925 (+ 3310us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:47:16.372105 (+ 180us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 15:47:16.377306 24403 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.526s user 0.006s sys 0.006s
W20260809 15:47:16.381733 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 54758.8us
W20260809 15:47:16.382131 24403 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.793s user 0.006s sys 0.006s
I20260809 15:47:16.382638 23283 raft_consensus.cc:1226] T d1a134ba33ea4076bbaac0ae460c17ba P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 6. Current term is 7. Ops: []
I20260809 15:47:16.369385 28532 raft_consensus.cc:504] T d3e94b7e50754a81b3d72a07476cb7b9 P 61e69695258f48baa984c9d395e8ce3f [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:15.421938 28624 consensus_queue.cc:1046] T 0c29937a80724ddb9d07a27534a146f3 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: INVALID_TERM, Last received: 5.2, Next index: 3, Last known committed idx: 2, Time since last communication: 1.117s
W20260809 15:47:16.384258 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 959826us
I20260809 15:47:16.374709 23277 raft_consensus.cc:2454] T a88421dba80d4d3388aa5eb62e23853b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 7.
I20260809 15:47:16.404573 23277 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53562 (request call id 1993) took 1133 ms. Trace:
I20260809 15:47:16.371452 28560 leader_election.cc:288] T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 10 election: Requested vote from peers 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:16.400667 23283 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53554 (request call id 1842) took 1200 ms. Trace:
I20260809 15:47:16.404798 23228 leader_election.cc:302] T 0c48a3049897452383257aa757834fb5 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c
I20260809 15:47:16.407835 23283 rpcz_store.cc:276] 0809 15:47:15.200423 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:16.382248 (+1181825us) service_pool.cc:227] Handling call
0809 15:47:16.382488 (+ 240us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:47:16.383039 (+ 551us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:47:16.400613 (+ 17574us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:47:16.405000 23277 rpcz_store.cc:276] 0809 15:47:15.270711 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:15.446439 (+175728us) service_pool.cc:227] Handling call
0809 15:47:16.404519 (+958080us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:47:15.419767 28612 raft_consensus.cc:504] T f7715d71dbac447b9351fc5eb36c5e80 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:16.327226 28431 raft_consensus.cc:504] T a3622f6e3e3641cea1960e3a31498bb7 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:16.415632 23277 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53562 (request call id 1995) took 1101 ms. Trace:
I20260809 15:47:16.421169 23277 rpcz_store.cc:276] 0809 15:47:15.314478 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:16.414896 (+1100418us) service_pool.cc:227] Handling call
0809 15:47:16.415179 (+ 283us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:47:16.415354 (+ 175us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:47:16.415378 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 15:47:16.415415 (+ 37us) raft_consensus.cc:1631] Marking committed up to 4
0809 15:47:16.415450 (+ 35us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:47:16.415497 (+ 47us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:47:16.415582 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:47:16.383636 28419 raft_consensus.cc:2726] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260809 15:47:16.414529 23283 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53562 (request call id 1994) took 1101 ms. Trace:
I20260809 15:47:16.417145 28626 raft_consensus.cc:504] T 07fcabb569414bddbd39df6cedd24e64 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:16.409333 23228 leader_election.cc:302] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:16.402635 27493 raft_consensus.cc:2726] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260809 15:47:16.429183 23283 rpcz_store.cc:276] 0809 15:47:15.312795 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:16.413822 (+1101027us) service_pool.cc:227] Handling call
0809 15:47:16.414097 (+ 275us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:47:16.414249 (+ 152us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:47:16.414273 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 15:47:16.414310 (+ 37us) raft_consensus.cc:1631] Marking committed up to 3
0809 15:47:16.414339 (+ 29us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:47:16.414382 (+ 43us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:47:16.414485 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:47:16.400524 28290 raft_consensus.cc:482] T 7660dfa2ef694927b8eda4100a2bcd9b P 61e69695258f48baa984c9d395e8ce3f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
W20260809 15:47:16.430557 23286 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.235s user 0.004s sys 0.027s
I20260809 15:47:16.407620 27957 raft_consensus.cc:482] T f7715d71dbac447b9351fc5eb36c5e80 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:47:16.444267 23277 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53554 (request call id 1843) took 1084 ms. Trace:
W20260809 15:47:16.440212 23286 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.245s user 0.004s sys 0.027s
W20260809 15:47:16.445693 24406 leader_election.cc:334] T 294d4bcc1e7a477cb56455581277a60f P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459): Timed out: RequestConsensusVote RPC to 127.20.202.65:35459 timed out after 1.945s (SENT)
I20260809 15:47:16.381387 28563 raft_consensus.cc:504] T f9e7cadcd24d45fcbfa84209225e8f2d P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:16.444702 23277 rpcz_store.cc:276] 0809 15:47:15.360099 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:16.422110 (+1062011us) service_pool.cc:227] Handling call
0809 15:47:16.443811 (+ 21701us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:47:16.443971 (+ 160us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:47:16.443998 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 15:47:16.444032 (+ 34us) raft_consensus.cc:1631] Marking committed up to 6
0809 15:47:16.444067 (+ 35us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:47:16.444117 (+ 50us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:47:16.444222 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:47:16.408295 28319 consensus_queue.cc:1035] T 7baa2492df0340c4a5ac241959ce1ac3 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.035s
W20260809 15:47:16.447153 24451 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.147s user 0.014s sys 0.010s
W20260809 15:47:16.459703 24406 leader_election.cc:334] T 907bacfa79d04a55a68c8dbe60056462 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459): Timed out: RequestConsensusVote RPC to 127.20.202.65:35459 timed out after 2.054s (SENT)
I20260809 15:47:16.370385 28636 consensus_queue.cc:260] T f9e7cadcd24d45fcbfa84209225e8f2d P 67b3aeadfa7a432abe8af4d292e04ab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:16.444310 28521 raft_consensus.cc:482] T d1a134ba33ea4076bbaac0ae460c17ba P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:47:16.438903 28667 raft_consensus.cc:2726] T 0c48a3049897452383257aa757834fb5 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20260809 15:47:16.465766 24406 leader_election.cc:334] T 0c48a3049897452383257aa757834fb5 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459): Timed out: RequestConsensusVote RPC to 127.20.202.65:35459 timed out after 2.148s (SENT)
I20260809 15:47:16.449129 28419 raft_consensus.cc:2726] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20260809 15:47:16.460665 24451 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.160s user 0.015s sys 0.010s
W20260809 15:47:16.494576 23517 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44018 (request call id 333) took 2263 ms (client timeout 1575 ms). Trace:
I20260809 15:47:16.493515 23274 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53554 (request call id 1815) took 3365 ms. Trace:
I20260809 15:47:16.495460 23274 rpcz_store.cc:276] 0809 15:47:13.128295 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:13.128406 (+ 111us) service_pool.cc:227] Handling call
0809 15:47:13.128668 (+ 262us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:47:13.410300 (+281632us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:47:16.470642 (+3060342us) spinlock_profiling.cc:243] Waited 281 ms on lock 0x7b540030eb54. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e9220f51 00007f34e921f99e 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772 00007f34e764a94c 00007f34e7649d86
0809 15:47:16.493402 (+ 22760us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":646477184}
W20260809 15:47:16.494946 23517 rpcz_store.cc:269] 0809 15:47:14.207173 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:14.207297 (+ 124us) service_pool.cc:227] Handling call
0809 15:47:16.470262 (+2262965us) spinlock_profiling.cc:243] Waited 13 ms on lock 0x7b54000f8cd4. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e92242ae 00007f34ee161c27 00007f34e7849c04 00007f34e7849b92 00007f34e7849ac1 00007f34e7849a3c 00007f34e7848cf2 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:47:16.470360 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":29822336}
W20260809 15:47:16.496542 23223 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.764s user 0.000s sys 0.021s
W20260809 15:47:16.498247 23223 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.766s user 0.000s sys 0.021s
I20260809 15:47:16.447840 27957 raft_consensus.cc:504] T f7715d71dbac447b9351fc5eb36c5e80 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:16.498423 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a51c0baaa6e432997c9d3552af4f254" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f"
I20260809 15:47:16.445971 28290 raft_consensus.cc:504] T 7660dfa2ef694927b8eda4100a2bcd9b P 61e69695258f48baa984c9d395e8ce3f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:16.465755 28631 raft_consensus.cc:482] T 213543cdbe814c89a17a963469922c1b P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:16.444216 28663 consensus_queue.cc:1046] T 0c29937a80724ddb9d07a27534a146f3 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: INVALID_TERM, Last received: 7.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.049s
I20260809 15:47:16.504877 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3924b648461e463b9d7925e5671d68ae" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f"
I20260809 15:47:16.505668 24468 raft_consensus.cc:2379] T 3924b648461e463b9d7925e5671d68ae P 61e69695258f48baa984c9d395e8ce3f [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d in current term 10: Already voted for candidate 61e69695258f48baa984c9d395e8ce3f in this term.
I20260809 15:47:16.469446 28669 raft_consensus.cc:482] T a88421dba80d4d3388aa5eb62e23853b P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:16.535251 24475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3e94b7e50754a81b3d72a07476cb7b9" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f"
W20260809 15:47:16.537088 23468 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.259s user 0.011s sys 0.018s
W20260809 15:47:16.540788 23468 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.263s user 0.011s sys 0.018s
I20260809 15:47:16.550036 24452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d880de657e544468c82157769622a26" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:47:16.550868 24452 raft_consensus.cc:2396] T 3d880de657e544468c82157769622a26 P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260809 15:47:16.592840 24475 raft_consensus.cc:2379] T d3e94b7e50754a81b3d72a07476cb7b9 P 61e69695258f48baa984c9d395e8ce3f [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d in current term 9: Already voted for candidate 61e69695258f48baa984c9d395e8ce3f in this term.
I20260809 15:47:16.494663 28667 raft_consensus.cc:482] T 00a1881267a14ba8b78c81040f17a014 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:47:16.498361 28419 raft_consensus.cc:482] T 0c29937a80724ddb9d07a27534a146f3 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:16.543258 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bcac4b24e7a4448abdc934bba5a624e" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
W20260809 15:47:16.612874 23227 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.742s user 0.002s sys 0.023s
W20260809 15:47:16.619550 23227 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.914s user 0.002s sys 0.023s
I20260809 15:47:16.620720 23351 leader_election.cc:302] T df20671a7b874dbb98c0ef28371c1308 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:47:16.642745 24448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e14d118b2eaa42aa8a7cd4025aba7128" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:47:16.649807 28676 raft_consensus.cc:482] T ae6d2c08a7a94f58ac8dd698e3fd456c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:16.650465 28676 raft_consensus.cc:504] T ae6d2c08a7a94f58ac8dd698e3fd456c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:16.469738 28521 raft_consensus.cc:504] T d1a134ba33ea4076bbaac0ae460c17ba P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:16.593554 28636 raft_consensus.cc:3037] T f9e7cadcd24d45fcbfa84209225e8f2d P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Advancing to term 8
I20260809 15:47:16.536953 28440 raft_consensus.cc:482] T df20671a7b874dbb98c0ef28371c1308 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
I20260809 15:47:16.644022 28580 raft_consensus.cc:482] T 2424fd9bd7f14e508fd106cbfda786a4 P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:16.690090 28690 raft_consensus.cc:482] T 6ae14a74a26f434db08c652e097385c8 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
I20260809 15:47:16.696717 23282 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53554 (request call id 1838) took 1788 ms. Trace:
I20260809 15:47:16.706395 23282 rpcz_store.cc:276] 0809 15:47:14.907995 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:14.908204 (+ 209us) service_pool.cc:227] Handling call
0809 15:47:14.919093 (+ 10889us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:47:15.159583 (+240490us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:47:16.696483 (+1536900us) spinlock_profiling.cc:243] Waited 206 ms on lock 0x7b540030f554. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e9220f51 00007f34e921f99e 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772 00007f34e764a94c 00007f34e7649d86
0809 15:47:16.696655 (+ 172us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":473411840}
W20260809 15:47:16.707063 28671 log.cc:924] Time spent T 294d4bcc1e7a477cb56455581277a60f P 65c36d164b0540f099d592f38307ee6c: Append to log took a long time: real 0.200s user 0.000s sys 0.000s
I20260809 15:47:16.644259 28598 raft_consensus.cc:482] T 0c48a3049897452383257aa757834fb5 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
I20260809 15:47:16.671947 28624 raft_consensus.cc:482] T f9e7cadcd24d45fcbfa84209225e8f2d P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:16.534487 28631 raft_consensus.cc:504] T 213543cdbe814c89a17a963469922c1b P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:16.654625 28319 raft_consensus.cc:482] T ba56725e42f74a1baa4c04d9660727f9 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:16.721794 23526 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56766 (request call id 2148) took 2333 ms. Trace:
I20260809 15:47:16.672618 28425 raft_consensus.cc:482] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:47:16.674131 28532 leader_election.cc:288] T d3e94b7e50754a81b3d72a07476cb7b9 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 9 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
W20260809 15:47:16.469071 28538 consensus_queue.cc:1159] T f9e7cadcd24d45fcbfa84209225e8f2d P 67b3aeadfa7a432abe8af4d292e04ab9 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "65c36d164b0540f099d592f38307ee6c" responder_term: 8 status { last_received { term: 7 index: 6 } last_committed_idx: 3 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 7. Current term is 8. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260809 15:47:16.736572 28690 raft_consensus.cc:504] T 6ae14a74a26f434db08c652e097385c8 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:47:16.720784 23469 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.683s user 0.002s sys 0.040s
W20260809 15:47:16.740590 23469 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.856s user 0.002s sys 0.040s
I20260809 15:47:16.655151 28419 raft_consensus.cc:504] T 0c29937a80724ddb9d07a27534a146f3 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:16.727787 24472 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38724 (request call id 568) took 3105 ms. Trace:
I20260809 15:47:16.753959 24472 rpcz_store.cc:276] 0809 15:47:13.622591 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:13.622707 (+ 116us) service_pool.cc:227] Handling call
0809 15:47:14.507988 (+885281us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:47:14.508647 (+ 659us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:47:16.727587 (+2218940us) spinlock_profiling.cc:243] Waited 874 ms on lock 0x7b5400720450. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e921fa09 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:47:16.727731 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2011282304}
I20260809 15:47:16.752070 28686 raft_consensus.cc:482] T 763e58d6ba644eea830a6c1664d39c9b P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:47:16.732625 23280 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53562 (request call id 1981) took 1939 ms. Trace:
I20260809 15:47:16.722904 23526 rpcz_store.cc:276] 0809 15:47:14.388565 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:14.388813 (+ 248us) service_pool.cc:227] Handling call
0809 15:47:16.721609 (+2332796us) spinlock_profiling.cc:243] Waited 25.5 ms on lock 0x7b540038e354. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e92242ae 00007f34ee161c27 00007f34e7849c04 00007f34e7849b92 00007f34e7849ac1 00007f34e7849a3c 00007f34e7848cf2 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:47:16.721738 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":58659200}
W20260809 15:47:16.725775 28675 log.cc:924] Time spent T d0721a4403a440f383318a3cea9d1442 P 67b3aeadfa7a432abe8af4d292e04ab9: Append to log took a long time: real 0.107s user 0.000s sys 0.000s
I20260809 15:47:16.743096 28681 consensus_queue.cc:1046] T 763e58d6ba644eea830a6c1664d39c9b P 65c36d164b0540f099d592f38307ee6c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.004s
I20260809 15:47:16.715680 28626 leader_election.cc:288] T 07fcabb569414bddbd39df6cedd24e64 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 8 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:16.721390 28449 raft_consensus.cc:979] T 763e58d6ba644eea830a6c1664d39c9b P 65c36d164b0540f099d592f38307ee6c: : Instructing follower c0f38c1908414566aa9115bb2cbdd85d to start an election
I20260809 15:47:16.380172 24455 rpcz_store.cc:276] 0809 15:47:15.114030 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:15.116169 (+ 2139us) service_pool.cc:227] Handling call
0809 15:47:15.150512 (+ 34343us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:47:15.165071 (+ 14559us) raft_consensus.cc:1520] Early marking committed up to index 8
0809 15:47:15.194072 (+ 29001us) log.cc:842] Serialized 14 byte log entry
0809 15:47:15.194377 (+ 305us) log.cc:842] Serialized 14 byte log entry
0809 15:47:15.813697 (+619320us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 15:47:15.839358 (+ 25661us) raft_consensus.cc:1631] Marking committed up to 8
0809 15:47:16.069118 (+229760us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:47:16.128420 (+ 59302us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:47:16.184115 (+ 55695us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":4000,"prepare.run_cpu_time_us":64,"prepare.run_wall_time_us":56,"raft.queue_time_us":24128,"raft.run_cpu_time_us":96,"raft.run_wall_time_us":5339,"spinlock_wait_cycles":67676416,"thread_start_us":451860,"threads_started":1}
I20260809 15:47:16.766644 23519 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44018 (request call id 343) took 1498 ms. Trace:
I20260809 15:47:16.767035 23519 rpcz_store.cc:276] 0809 15:47:15.268298 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:15.268508 (+ 210us) service_pool.cc:227] Handling call
0809 15:47:15.291953 (+ 23445us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 15:47:15.292246 (+ 293us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:47:15.292270 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 15:47:15.295672 (+ 3402us) log.cc:842] Serialized 26 byte log entry
0809 15:47:15.621056 (+325384us) raft_consensus.cc:1631] Marking committed up to 6
0809 15:47:15.621093 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:47:15.621669 (+ 576us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 15:47:16.729213 (+1107544us) spinlock_profiling.cc:243] Waited 53.9 ms on lock 0x7b4800280758. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34ee0c77f1 00007f34ee0be8e7 00007f34ee0be77a 00007f34ee1b295f 00007f34ee1b2627 00007f34ee1c6b2c 00007f34ee1c6ad8 00007f34ee1c6a58 00007f34ee1c6a13 00007f34ee1c5cbc 00007f34ed147be6
0809 15:47:16.757897 (+ 28684us) raft_consensus.cc:1671] finished
0809 15:47:16.757921 (+ 24us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:47:16.766571 (+ 8650us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":78502,"raft.run_cpu_time_us":11558,"raft.run_wall_time_us":1351613,"spinlock_wait_cycles":123957632,"thread_start_us":238129,"threads_started":1,"wal-append.queue_time_us":1137087}
I20260809 15:47:16.713266 28674 consensus_queue.cc:1035] T 907bacfa79d04a55a68c8dbe60056462 P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.304s
I20260809 15:47:16.747246 28290 leader_election.cc:288] T 7660dfa2ef694927b8eda4100a2bcd9b P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:16.722755 28624 raft_consensus.cc:504] T f9e7cadcd24d45fcbfa84209225e8f2d P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:16.741612 28319 raft_consensus.cc:504] T ba56725e42f74a1baa4c04d9660727f9 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } attrs { replace: true } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } }
I20260809 15:47:16.748919 28588 raft_consensus.cc:482] T 4564dfc6d4094c1b9240b367ccecf3b1 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:16.750499 28348 raft_consensus.cc:482] T 6ced718c1a5c4da79cb8c1ebd1435b87 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:47:16.752677 28431 leader_election.cc:288] T a3622f6e3e3641cea1960e3a31498bb7 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:16.671651 28302 raft_consensus.cc:482] T c0741ab44d99439abc3aa3bdd27ee160 P 61e69695258f48baa984c9d395e8ce3f [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:47:16.755630 23280 rpcz_store.cc:276] 0809 15:47:14.793151 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:14.793288 (+ 137us) service_pool.cc:227] Handling call
0809 15:47:14.793540 (+ 252us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:47:14.793715 (+ 175us) raft_consensus.cc:1520] Early marking committed up to index 8
0809 15:47:14.958709 (+164994us) log.cc:842] Serialized 14 byte log entry
0809 15:47:16.732100 (+1773391us) spinlock_profiling.cc:243] Waited 1.14 ms on lock 0x7b180003e920. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34ef1978b1 00007f34ef1a4661 00007f34e8eaf82c 00007f34e62da9dc 00007f34ef111a39 00007f34e62f78bb 00007f34e62f5cbf 00007f34e62f5155 00007f34e92183d0 00007f34e922a576 00007f34e923b0d7
0809 15:47:16.732229 (+ 129us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 15:47:16.732279 (+ 50us) raft_consensus.cc:1631] Marking committed up to 8
0809 15:47:16.732328 (+ 49us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:47:16.732443 (+ 115us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:47:16.732562 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":185443,"prepare.run_cpu_time_us":59,"prepare.run_wall_time_us":2572,"raft.queue_time_us":145881,"raft.run_cpu_time_us":154,"raft.run_wall_time_us":24506,"spinlock_wait_cycles":2624896,"thread_start_us":184264,"threads_started":2}
I20260809 15:47:16.693279 28440 raft_consensus.cc:504] T df20671a7b874dbb98c0ef28371c1308 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:16.793864 23528 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44018 (request call id 335) took 1822 ms. Trace:
W20260809 15:47:16.796454 23273 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53562 (request call id 1980) took 2004 ms (client timeout 2055 ms). Trace:
W20260809 15:47:16.804423 23273 rpcz_store.cc:269] 0809 15:47:14.792456 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:14.792603 (+ 147us) service_pool.cc:227] Handling call
0809 15:47:16.796228 (+2003625us) spinlock_profiling.cc:243] Waited 422 ms on lock 0x7b5400366d54. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e92242ae 00007f34ee161c27 00007f34e7849c04 00007f34e7849b92 00007f34e7849ac1 00007f34e7849a3c 00007f34e7848cf2 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:47:16.796330 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":970355584}
I20260809 15:47:16.802917 23528 rpcz_store.cc:276] 0809 15:47:14.971384 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:14.971488 (+ 104us) service_pool.cc:227] Handling call
0809 15:47:14.971735 (+ 247us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:47:15.161108 (+189373us) raft_consensus.cc:1520] Early marking committed up to index 7
0809 15:47:15.161339 (+ 231us) log.cc:842] Serialized 14 byte log entry
0809 15:47:15.166918 (+ 5579us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 15:47:15.166965 (+ 47us) raft_consensus.cc:1631] Marking committed up to 7
0809 15:47:15.167010 (+ 45us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:47:16.793640 (+1626630us) spinlock_profiling.cc:243] Waited 189 ms on lock 0x7b54000564d4. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e9220f51 00007f34e921f99e 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772 00007f34e764a94c 00007f34e7649d86
0809 15:47:16.793686 (+ 46us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:47:16.793806 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3381,"prepare.queue_time_us":1315,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":56,"raft.queue_time_us":16320,"raft.run_cpu_time_us":65,"raft.run_wall_time_us":65,"spinlock_wait_cycles":435066624}
I20260809 15:47:16.807210 23270 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55434 (request call id 314) took 2625 ms. Trace:
I20260809 15:47:16.723727 28598 raft_consensus.cc:504] T 0c48a3049897452383257aa757834fb5 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:16.643400 28667 raft_consensus.cc:504] T 00a1881267a14ba8b78c81040f17a014 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:16.755755 28612 leader_election.cc:288] T f7715d71dbac447b9351fc5eb36c5e80 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:16.740729 28425 raft_consensus.cc:504] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:16.782650 28588 raft_consensus.cc:504] T 4564dfc6d4094c1b9240b367ccecf3b1 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:16.786372 28348 raft_consensus.cc:504] T 6ced718c1a5c4da79cb8c1ebd1435b87 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:16.824995 27957 leader_election.cc:288] T f7715d71dbac447b9351fc5eb36c5e80 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:16.825695 27957 raft_consensus.cc:482] T d3be1e356b254d22b995be32f1c12f6f P c0f38c1908414566aa9115bb2cbdd85d [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
I20260809 15:47:16.826638 28348 leader_election.cc:288] T 6ced718c1a5c4da79cb8c1ebd1435b87 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:16.786937 28663 raft_consensus.cc:1067] T 763e58d6ba644eea830a6c1664d39c9b P 65c36d164b0540f099d592f38307ee6c [term 7 LEADER]: Signalling peer c0f38c1908414566aa9115bb2cbdd85d to start an election
I20260809 15:47:16.800776 28319 leader_election.cc:288] T ba56725e42f74a1baa4c04d9660727f9 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:16.829128 28667 leader_election.cc:288] T 00a1881267a14ba8b78c81040f17a014 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:16.829900 28588 leader_election.cc:288] T 4564dfc6d4094c1b9240b367ccecf3b1 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:16.613581 28669 raft_consensus.cc:504] T a88421dba80d4d3388aa5eb62e23853b P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:16.826368 27957 raft_consensus.cc:504] T d3be1e356b254d22b995be32f1c12f6f P c0f38c1908414566aa9115bb2cbdd85d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:16.758623 28415 raft_consensus.cc:482] T 953eea268e17491ca76f3a555860a512 P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:16.712121 28580 raft_consensus.cc:504] T 2424fd9bd7f14e508fd106cbfda786a4 P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:16.794185 28230 raft_consensus.cc:2726] T df20671a7b874dbb98c0ef28371c1308 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260809 15:47:16.786412 28680 raft_consensus.cc:482] T 0acc47ce21c3422698d1ef7277c3ed63 P 65c36d164b0540f099d592f38307ee6c [term 9 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:47:16.808033 23270 rpcz_store.cc:276] 0809 15:47:14.181827 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:14.183372 (+ 1545us) service_pool.cc:227] Handling call
0809 15:47:14.267454 (+ 84082us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:47:14.268086 (+ 632us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:47:16.807026 (+2538940us) spinlock_profiling.cc:243] Waited 83.7 ms on lock 0x7b54002409d0. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e921fa09 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:47:16.807151 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":192621952}
I20260809 15:47:16.812440 23393 raft_consensus.cc:1226] T e71a05ba1d4d42bd95c3f85f9313399e P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Rejecting Update request from peer 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 6. Current term is 7. Ops: []
I20260809 15:47:16.820752 28445 raft_consensus.cc:482] T c5c2e425a74b4e64ae70b5def750434c P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:16.793779 28521 leader_election.cc:288] T d1a134ba33ea4076bbaac0ae460c17ba P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:16.730818 28586 consensus_queue.cc:1046] T ba56725e42f74a1baa4c04d9660727f9 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.011s
I20260809 15:47:16.822017 28688 raft_consensus.cc:482] T 6ae14a74a26f434db08c652e097385c8 P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:16.756820 28563 leader_election.cc:288] T f9e7cadcd24d45fcbfa84209225e8f2d P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:16.835343 28624 leader_election.cc:288] T f9e7cadcd24d45fcbfa84209225e8f2d P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:47:16.794044 28431 raft_consensus.cc:482] T 0da4d5283c8a46878bd39b85e5c71f66 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:16.750524 28695 raft_consensus.cc:482] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
I20260809 15:47:16.754957 28686 raft_consensus.cc:504] T 763e58d6ba644eea830a6c1664d39c9b P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:16.830752 28440 leader_election.cc:288] T df20671a7b874dbb98c0ef28371c1308 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:47:16.820295 28302 raft_consensus.cc:504] T c0741ab44d99439abc3aa3bdd27ee160 P 61e69695258f48baa984c9d395e8ce3f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:16.833791 28588 raft_consensus.cc:482] T 3296ef50be254fc29cfc77d22a8f7a23 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
W20260809 15:47:17.010963 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 390319us
I20260809 15:47:17.110950 23273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "907bacfa79d04a55a68c8dbe60056462" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
W20260809 15:47:17.112478 28587 consensus_meta.cc:287] T d0721a4403a440f383318a3cea9d1442 P 65c36d164b0540f099d592f38307ee6c: Time spent flushing consensus metadata: real 0.764s user 0.009s sys 0.000s
W20260809 15:47:17.117343 28383 consensus_meta.cc:287] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent flushing consensus metadata: real 0.816s user 0.011s sys 0.000s
W20260809 15:47:17.118556 28476 consensus_meta.cc:287] T d1a134ba33ea4076bbaac0ae460c17ba P 65c36d164b0540f099d592f38307ee6c: Time spent flushing consensus metadata: real 0.759s user 0.012s sys 0.000s
I20260809 15:47:17.109833 28588 raft_consensus.cc:504] T 3296ef50be254fc29cfc77d22a8f7a23 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:16.834021 28348 raft_consensus.cc:482] T 8e4ffcf83c234ae7910ddef3ec0b4bda P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
W20260809 15:47:16.862298 23472 leader_election.cc:334] T 28e17dcb5efe45b8b6a8c42da03b4819 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263): Timed out: RequestConsensusVote RPC to 127.20.202.66:36263 timed out after 2.078s (SENT)
I20260809 15:47:16.870330 24471 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38722 (request call id 563) took 1286 ms. Trace:
I20260809 15:47:17.191984 28631 leader_election.cc:288] T 213543cdbe814c89a17a963469922c1b P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:47:17.196918 23280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "294d4bcc1e7a477cb56455581277a60f" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:17.197564 23282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c48a3049897452383257aa757834fb5" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:47:17.224411 23282 raft_consensus.cc:2379] T 0c48a3049897452383257aa757834fb5 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 61e69695258f48baa984c9d395e8ce3f in current term 6: Already voted for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in this term.
I20260809 15:47:17.197808 23283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d880de657e544468c82157769622a26" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:17.198335 23277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a51c0baaa6e432997c9d3552af4f254" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
W20260809 15:47:17.046327 24451 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.511s user 0.008s sys 0.008s
I20260809 15:47:16.937721 28280 raft_consensus.cc:482] T e71a05ba1d4d42bd95c3f85f9313399e P 61e69695258f48baa984c9d395e8ce3f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
I20260809 15:47:16.958794 28419 leader_election.cc:288] T 0c29937a80724ddb9d07a27534a146f3 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:17.074133 27957 leader_election.cc:288] T d3be1e356b254d22b995be32f1c12f6f P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:17.199308 28431 raft_consensus.cc:504] T 0da4d5283c8a46878bd39b85e5c71f66 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:47:16.881999 27892 consensus_meta.cc:287] T 30c1740ef6874df0a332fbd6fd4b03ce P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent flushing consensus metadata: real 0.525s user 0.005s sys 0.006s
I20260809 15:47:17.200702 28580 leader_election.cc:288] T 2424fd9bd7f14e508fd106cbfda786a4 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:47:17.199045 28425 leader_election.cc:288] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:17.204053 28248 raft_consensus.cc:482] T 4bcac4b24e7a4448abdc934bba5a624e P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
I20260809 15:47:16.914938 23286 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53562 (request call id 1989) took 1922 ms. Trace:
W20260809 15:47:17.210204 23394 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.596s user 0.001s sys 0.021s
W20260809 15:47:17.248231 23394 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.653s user 0.002s sys 0.022s
I20260809 15:47:16.936060 28679 raft_consensus.cc:482] T 213543cdbe814c89a17a963469922c1b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:47:17.253458 28679 raft_consensus.cc:504] T 213543cdbe814c89a17a963469922c1b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:17.255445 28679 leader_election.cc:288] T 213543cdbe814c89a17a963469922c1b P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:16.799448 28271 raft_consensus.cc:3037] T a3622f6e3e3641cea1960e3a31498bb7 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Advancing to term 6
I20260809 15:47:17.109756 24455 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3be1e356b254d22b995be32f1c12f6f" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:47:17.219687 23472 leader_election.cc:302] T 28e17dcb5efe45b8b6a8c42da03b4819 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:17.193126 28676 leader_election.cc:288] T ae6d2c08a7a94f58ac8dd698e3fd456c P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:17.221539 23351 leader_election.cc:302] T a88421dba80d4d3388aa5eb62e23853b P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:47:16.959357 28691 raft_consensus.cc:482] T ba56725e42f74a1baa4c04d9660727f9 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 15:47:17.194659 24410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 262054us
W20260809 15:47:17.263684 23469 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.507s user 0.009s sys 0.032s
W20260809 15:47:17.264271 23469 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.508s user 0.009s sys 0.032s
I20260809 15:47:17.248039 23286 rpcz_store.cc:276] 0809 15:47:14.992777 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:14.992924 (+ 147us) service_pool.cc:227] Handling call
0809 15:47:14.993230 (+ 306us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 15:47:15.111207 (+117977us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:47:15.111235 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 15:47:15.125848 (+ 14613us) log.cc:842] Serialized 26 byte log entry
0809 15:47:16.446314 (+1320466us) raft_consensus.cc:1631] Marking committed up to 4
0809 15:47:16.446354 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:47:16.446414 (+ 60us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 15:47:16.914728 (+468314us) raft_consensus.cc:1671] finished
0809 15:47:16.914749 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:47:16.914881 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":33556,"raft.queue_time_us":156923,"raft.run_cpu_time_us":95,"raft.run_wall_time_us":5906,"thread_start_us":1363783,"threads_started":2,"wal-append.queue_time_us":1311317}
I20260809 15:47:17.266657 23286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee02fff2c9c48a19f43eb358a89e7b0" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:17.216771 23270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee02fff2c9c48a19f43eb358a89e7b0" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:47:17.210736 28521 raft_consensus.cc:482] T 0c29937a80724ddb9d07a27534a146f3 P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:17.222877 28348 raft_consensus.cc:504] T 8e4ffcf83c234ae7910ddef3ec0b4bda P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
W20260809 15:47:17.274737 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 53284.1us
I20260809 15:47:17.209307 28325 raft_consensus.cc:482] T ae6d2c08a7a94f58ac8dd698e3fd456c P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:17.228768 23280 raft_consensus.cc:2379] T 294d4bcc1e7a477cb56455581277a60f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 61e69695258f48baa984c9d395e8ce3f in current term 6: Already voted for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in this term.
W20260809 15:47:17.231514 23400 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.524s user 0.004s sys 0.025s
W20260809 15:47:17.276970 23400 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.569s user 0.004s sys 0.025s
I20260809 15:47:17.219738 24471 rpcz_store.cc:276] 0809 15:47:15.583572 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:15.583718 (+ 146us) service_pool.cc:227] Handling call
0809 15:47:15.583995 (+ 277us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:47:16.248062 (+664067us) raft_consensus.cc:1520] Early marking committed up to index 7
0809 15:47:16.503976 (+255914us) log.cc:842] Serialized 14 byte log entry
0809 15:47:16.870010 (+366034us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 15:47:16.870065 (+ 55us) raft_consensus.cc:1631] Marking committed up to 7
0809 15:47:16.870110 (+ 45us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:47:16.870174 (+ 64us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:47:16.870275 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1238,"prepare.queue_time_us":244680,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":34,"raft.queue_time_us":18507,"raft.run_cpu_time_us":119,"raft.run_wall_time_us":17637,"thread_start_us":602872,"threads_started":2}
I20260809 15:47:17.232836 23393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30c1740ef6874df0a332fbd6fd4b03ce" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:17.212157 28445 raft_consensus.cc:504] T c5c2e425a74b4e64ae70b5def750434c P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:17.213063 28563 raft_consensus.cc:482] T 07fcabb569414bddbd39df6cedd24e64 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:47:16.840898 28415 raft_consensus.cc:504] T 953eea268e17491ca76f3a555860a512 P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:47:17.212229 28602 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.540s user 0.009s sys 0.015s
I20260809 15:47:17.238912 23282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "294d4bcc1e7a477cb56455581277a60f" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:47:17.082454 28707 consensus_queue.cc:1035] T 472816994d814cb198b69e5248697440 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.162s
I20260809 15:47:17.244241 23277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d880de657e544468c82157769622a26" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:17.281281 23394 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55320 (request call id 1628) took 2744 ms. Trace:
I20260809 15:47:17.282032 23282 raft_consensus.cc:2379] T 294d4bcc1e7a477cb56455581277a60f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 61e69695258f48baa984c9d395e8ce3f in current term 6: Already voted for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in this term.
W20260809 15:47:17.245810 23223 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.630s user 0.015s sys 0.012s
W20260809 15:47:17.284497 23223 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.669s user 0.015s sys 0.012s
I20260809 15:47:17.248548 23283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5712f6b6fffc4cc5a8ddfbe63e4097df" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:17.250397 23396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e17dcb5efe45b8b6a8c42da03b4819" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:17.252148 28302 leader_election.cc:288] T c0741ab44d99439abc3aa3bdd27ee160 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:17.280707 28720 consensus_queue.cc:1035] T 0acc47ce21c3422698d1ef7277c3ed63 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.016s
W20260809 15:47:17.292413 23396 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55320 (request call id 1621) took 2776 ms (client timeout 2078 ms). Trace:
I20260809 15:47:17.040030 28702 consensus_queue.cc:1046] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.236s
I20260809 15:47:17.214705 28383 raft_consensus.cc:504] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:17.293498 28712 raft_consensus.cc:2781] T a88421dba80d4d3388aa5eb62e23853b P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Leader election won for term 7
I20260809 15:47:17.294064 28712 raft_consensus.cc:686] T a88421dba80d4d3388aa5eb62e23853b P c0f38c1908414566aa9115bb2cbdd85d [term 7 LEADER]: Becoming Leader. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:47:17.294473 28383 leader_election.cc:288] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 9 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:17.282454 28325 raft_consensus.cc:504] T ae6d2c08a7a94f58ac8dd698e3fd456c P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:17.281525 28631 raft_consensus.cc:482] T d3e94b7e50754a81b3d72a07476cb7b9 P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:17.295572 28631 raft_consensus.cc:504] T d3e94b7e50754a81b3d72a07476cb7b9 P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:17.295296 28712 consensus_queue.cc:237] T a88421dba80d4d3388aa5eb62e23853b P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:17.297183 28348 leader_election.cc:288] T 8e4ffcf83c234ae7910ddef3ec0b4bda P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:17.298429 28631 leader_election.cc:288] T d3e94b7e50754a81b3d72a07476cb7b9 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
W20260809 15:47:17.277494 23227 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.631s user 0.005s sys 0.034s
I20260809 15:47:17.298893 28445 leader_election.cc:288] T c5c2e425a74b4e64ae70b5def750434c P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
W20260809 15:47:17.299219 23227 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.653s user 0.005s sys 0.034s
I20260809 15:47:17.299788 23270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a51c0baaa6e432997c9d3552af4f254" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
W20260809 15:47:17.292793 23396 rpcz_store.cc:269] 0809 15:47:14.510096 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:14.510201 (+ 105us) service_pool.cc:227] Handling call
0809 15:47:17.250241 (+2740040us) spinlock_profiling.cc:243] Waited 7.84 ms on lock 0x7b58001d7c98. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e613cf11 00007f34e62c0851 00007f34e62c0702 00007f34ef1b0b09 00007f34ee0ec730 00007f34ee1b279b 00007f34ee1b2821 00007f34ee1621b0 00007f34ee161a94 00007f34e7849c04 00007f34e7849b92
0809 15:47:17.286211 (+ 35970us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":18020736}
I20260809 15:47:17.212592 28587 raft_consensus.cc:504] T d0721a4403a440f383318a3cea9d1442 P 65c36d164b0540f099d592f38307ee6c [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:17.262112 28290 raft_consensus.cc:482] T 563daf42d7784db195ea99c3cb21b370 P 61e69695258f48baa984c9d395e8ce3f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:47:17.263747 28248 raft_consensus.cc:504] T 4bcac4b24e7a4448abdc934bba5a624e P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:17.283491 23394 rpcz_store.cc:276] 0809 15:47:14.532020 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:14.600356 (+ 68336us) service_pool.cc:227] Handling call
0809 15:47:14.647696 (+ 47340us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 15:47:14.651353 (+ 3657us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:47:14.655670 (+ 4317us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 15:47:14.656117 (+ 447us) log.cc:842] Serialized 26 byte log entry
0809 15:47:16.369918 (+1713801us) spinlock_profiling.cc:243] Waited 4.34 ms on lock 0x7b18004a3140. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34ef1978b1 00007f34ef1a4661 00007f34e8eaf82c 00007f34e62da9dc 00007f34ef111a39 00007f34e62f78bb 00007f34e62f5cbf 00007f34e62f7e22 00007f34e8ea15bb 00007f34e8ea7063 00007f34e8ea7411
0809 15:47:16.370052 (+ 134us) raft_consensus.cc:1631] Marking committed up to 6
0809 15:47:16.595514 (+225462us) log.cc:842] Serialized 14 byte log entry
0809 15:47:17.265455 (+669941us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:47:17.265543 (+ 88us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 15:47:17.265585 (+ 42us) raft_consensus.cc:1671] finished
0809 15:47:17.275988 (+ 10403us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:47:17.276112 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":26198,"prepare.queue_time_us":319494,"prepare.run_cpu_time_us":25,"prepare.run_wall_time_us":26,"raft.queue_time_us":43612,"raft.run_cpu_time_us":280,"raft.run_wall_time_us":29364,"spinlock_wait_cycles":9984768,"thread_start_us":937240,"threads_started":3,"wal-append.queue_time_us":95203,"wal-append.run_cpu_time_us":2004,"wal-append.run_wall_time_us":1096742}
I20260809 15:47:17.276680 24451 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38724 (request call id 573) took 2779 ms. Trace:
I20260809 15:47:17.328168 23394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05ad7d8fdca64e3da527280ca5be9280" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:17.328286 23396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1a134ba33ea4076bbaac0ae460c17ba" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:47:17.328858 23396 raft_consensus.cc:3037] T d1a134ba33ea4076bbaac0ae460c17ba P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Advancing to term 7
I20260809 15:47:17.276434 28425 raft_consensus.cc:482] T 00a1881267a14ba8b78c81040f17a014 P 65c36d164b0540f099d592f38307ee6c [term 9 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:47:17.296479 28325 leader_election.cc:288] T ae6d2c08a7a94f58ac8dd698e3fd456c P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:17.299937 23273 raft_consensus.cc:2354] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 61e69695258f48baa984c9d395e8ce3f for earlier term 8. Current term is 9.
I20260809 15:47:17.277945 23393 raft_consensus.cc:2454] T 30c1740ef6874df0a332fbd6fd4b03ce P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 8.
W20260809 15:47:17.276047 23472 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 413763us
I20260809 15:47:17.301137 23282 raft_consensus.cc:3032] T 7baa2492df0340c4a5ac241959ce1ac3 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 LEADER]: Stepping down as leader of term 5
I20260809 15:47:17.265658 28522 raft_consensus.cc:482] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:47:17.275106 28580 raft_consensus.cc:2726] T 28e17dcb5efe45b8b6a8c42da03b4819 P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260809 15:47:17.236140 28280 raft_consensus.cc:504] T e71a05ba1d4d42bd95c3f85f9313399e P 61e69695258f48baa984c9d395e8ce3f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
W20260809 15:47:17.284018 28602 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.610s user 0.010s sys 0.016s
I20260809 15:47:17.278761 28521 raft_consensus.cc:504] T 0c29937a80724ddb9d07a27534a146f3 P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:17.324215 23528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1a134ba33ea4076bbaac0ae460c17ba" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:47:17.326509 23517 raft_consensus.cc:1261] T 538a228ddfb54bc6a36528d5ff07597b P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Refusing update from remote peer 67b3aeadfa7a432abe8af4d292e04ab9: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
W20260809 15:47:17.309736 23470 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.638s user 0.005s sys 0.029s
W20260809 15:47:17.327385 23468 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.670s user 0.009s sys 0.028s
I20260809 15:47:17.329339 23394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2424fd9bd7f14e508fd106cbfda786a4" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:47:17.331180 23270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "538a228ddfb54bc6a36528d5ff07597b" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:47:17.237690 28685 consensus_queue.cc:1046] T e71a05ba1d4d42bd95c3f85f9313399e P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.065s
I20260809 15:47:17.335654 23283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a51c0baaa6e432997c9d3552af4f254" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
W20260809 15:47:17.335855 24406 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 108443us
I20260809 15:47:17.343798 23518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "953eea268e17491ca76f3a555860a512" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:17.346637 23520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "763e58d6ba644eea830a6c1664d39c9b" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:47:17.397946 23522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4564dfc6d4094c1b9240b367ccecf3b1" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:17.337690 23282 raft_consensus.cc:729] T 7baa2492df0340c4a5ac241959ce1ac3 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Running, Role: LEADER
I20260809 15:47:17.376678 23519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "294d4bcc1e7a477cb56455581277a60f" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:17.283939 28563 raft_consensus.cc:504] T 07fcabb569414bddbd39df6cedd24e64 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:17.329716 27892 raft_consensus.cc:482] T 28e17dcb5efe45b8b6a8c42da03b4819 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:47:17.326382 28586 raft_consensus.cc:482] T 472816994d814cb198b69e5248697440 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:47:17.318858 28714 raft_consensus.cc:482] T 563daf42d7784db195ea99c3cb21b370 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:47:17.217623 28695 raft_consensus.cc:504] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:17.215376 28688 raft_consensus.cc:504] T 6ae14a74a26f434db08c652e097385c8 P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:17.215700 28680 raft_consensus.cc:504] T 0acc47ce21c3422698d1ef7277c3ed63 P 65c36d164b0540f099d592f38307ee6c [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:17.303246 27493 raft_consensus.cc:482] T 7660dfa2ef694927b8eda4100a2bcd9b P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:47:17.283703 28691 raft_consensus.cc:504] T ba56725e42f74a1baa4c04d9660727f9 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:17.627148 23521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2db057fd10e149a9895dff6ef8c9f038" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:17.641225 23394 raft_consensus.cc:3037] T 2424fd9bd7f14e508fd106cbfda786a4 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Advancing to term 8
W20260809 15:47:17.610404 23397 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52856 (request call id 416) took 2542 ms (client timeout 2234 ms). Trace:
W20260809 15:47:17.642098 23468 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.985s user 0.014s sys 0.035s
I20260809 15:47:17.406011 23393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99bdb93503db4de4a1cfb0161ea0c07c" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
W20260809 15:47:17.641808 23397 rpcz_store.cc:269] 0809 15:47:15.068453 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:15.068714 (+ 261us) service_pool.cc:227] Handling call
0809 15:47:17.610189 (+2541475us) spinlock_profiling.cc:243] Waited 261 ms on lock 0x7b5400374bd4. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e92242ae 00007f34ee161c27 00007f34e7849c04 00007f34e7849b92 00007f34e7849ac1 00007f34e7849a3c 00007f34e7848cf2 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:47:17.610330 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":599850240}
W20260809 15:47:17.641031 23470 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.981s user 0.005s sys 0.040s
W20260809 15:47:17.647871 23470 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 988221us
W20260809 15:47:17.587430 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 308850us
I20260809 15:47:17.323377 28679 raft_consensus.cc:482] T 998c6cee72cf40228c6dba064c0db2da P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
W20260809 15:47:17.767694 24410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 278662us
I20260809 15:47:17.646474 23282 consensus_queue.cc:260] T 7baa2492df0340c4a5ac241959ce1ac3 P 67b3aeadfa7a432abe8af4d292e04ab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:47:17.616489 28721 log.cc:924] Time spent T 7660dfa2ef694927b8eda4100a2bcd9b P c0f38c1908414566aa9115bb2cbdd85d: Append to log took a long time: real 0.173s user 0.000s sys 0.000s
I20260809 15:47:17.656976 28411 raft_consensus.cc:3037] T 2db057fd10e149a9895dff6ef8c9f038 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Advancing to term 7
I20260809 15:47:17.698081 23402 raft_consensus.cc:1226] T 7baa2492df0340c4a5ac241959ce1ac3 P c0f38c1908414566aa9115bb2cbdd85d [term 6 LEADER]: Rejecting Update request from peer 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 5. Current term is 6. Ops: []
I20260809 15:47:17.433925 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a88421dba80d4d3388aa5eb62e23853b" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:17.608953 23396 raft_consensus.cc:2454] T d1a134ba33ea4076bbaac0ae460c17ba P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 7.
I20260809 15:47:17.655958 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2424fd9bd7f14e508fd106cbfda786a4" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
W20260809 15:47:17.615883 28708 log.cc:924] Time spent T 0a51c0baaa6e432997c9d3552af4f254 P 61e69695258f48baa984c9d395e8ce3f: Append to log took a long time: real 0.335s user 0.000s sys 0.001s
I20260809 15:47:17.649731 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e4ffcf83c234ae7910ddef3ec0b4bda" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:47:17.657655 23520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07fcabb569414bddbd39df6cedd24e64" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:47:17.783943 23396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5c2e425a74b4e64ae70b5def750434c" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:47:17.645171 23522 raft_consensus.cc:2454] T 4564dfc6d4094c1b9240b367ccecf3b1 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 7.
I20260809 15:47:17.785038 23522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7715d71dbac447b9351fc5eb36c5e80" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:17.647162 23518 raft_consensus.cc:2454] T 953eea268e17491ca76f3a555860a512 P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 8.
I20260809 15:47:17.759755 23517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05ad7d8fdca64e3da527280ca5be9280" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
W20260809 15:47:17.759083 28271 consensus_meta.cc:287] T a3622f6e3e3641cea1960e3a31498bb7 P 65c36d164b0540f099d592f38307ee6c: Time spent flushing consensus metadata: real 0.502s user 0.006s sys 0.003s
I20260809 15:47:17.361120 28319 raft_consensus.cc:482] T d0721a4403a440f383318a3cea9d1442 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:47:17.769203 23282 raft_consensus.cc:3037] T 7baa2492df0340c4a5ac241959ce1ac3 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Advancing to term 6
W20260809 15:47:17.771557 23472 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 90874us
I20260809 15:47:17.770831 23397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a88421dba80d4d3388aa5eb62e23853b" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:17.765007 23528 raft_consensus.cc:2454] T d1a134ba33ea4076bbaac0ae460c17ba P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 7.
I20260809 15:47:17.760928 23274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3be1e356b254d22b995be32f1c12f6f" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:17.619791 28274 raft_consensus.cc:3037] T 297c82ed906d4b48b184985eed3de601 P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Advancing to term 9
W20260809 15:47:17.767423 23470 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 108075us
W20260809 15:47:17.415125 28727 log.cc:924] Time spent T 7baa2492df0340c4a5ac241959ce1ac3 P c0f38c1908414566aa9115bb2cbdd85d: Append to log took a long time: real 0.080s user 0.000s sys 0.000s
I20260809 15:47:17.774281 23527 raft_consensus.cc:2454] T a88421dba80d4d3388aa5eb62e23853b P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 6.
I20260809 15:47:17.802799 23518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "538a228ddfb54bc6a36528d5ff07597b" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:47:17.803352 23518 raft_consensus.cc:3037] T 538a228ddfb54bc6a36528d5ff07597b P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Advancing to term 8
I20260809 15:47:17.767241 28631 raft_consensus.cc:482] T 73f6f44c259547f1b05a3a434616fa54 P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:17.782292 23228 leader_election.cc:302] T d1a134ba33ea4076bbaac0ae460c17ba P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:47:17.806694 23397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "953eea268e17491ca76f3a555860a512" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:17.808378 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "213543cdbe814c89a17a963469922c1b" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:17.810285 23397 raft_consensus.cc:2454] T 953eea268e17491ca76f3a555860a512 P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 8.
I20260809 15:47:17.783886 23393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4564dfc6d4094c1b9240b367ccecf3b1" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:17.811445 23393 raft_consensus.cc:2454] T 4564dfc6d4094c1b9240b367ccecf3b1 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 7.
I20260809 15:47:17.787099 23517 raft_consensus.cc:2454] T 05ad7d8fdca64e3da527280ca5be9280 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 7.
W20260809 15:47:17.812436 23399 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52856 (request call id 419) took 2683 ms (client timeout 1580 ms). Trace:
W20260809 15:47:17.812793 23399 rpcz_store.cc:269] 0809 15:47:15.129473 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:15.129607 (+ 134us) service_pool.cc:227] Handling call
0809 15:47:17.812274 (+2682667us) spinlock_profiling.cc:243] Waited 1.21 s on lock 0x7b54002206d4. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e92242ae 00007f34ee161c27 00007f34e7849c04 00007f34e7849b92 00007f34e7849ac1 00007f34e7849a3c 00007f34e7848cf2 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:47:17.812377 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2790659968}
I20260809 15:47:17.810334 23527 raft_consensus.cc:2454] T 213543cdbe814c89a17a963469922c1b P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 4.
I20260809 15:47:17.813687 23399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba56725e42f74a1baa4c04d9660727f9" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:17.814257 23399 raft_consensus.cc:2454] T ba56725e42f74a1baa4c04d9660727f9 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 6.
I20260809 15:47:17.775667 28580 raft_consensus.cc:482] T d2d4f749fcfd4fb3bfb8115af3471598 P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:17.818725 23396 raft_consensus.cc:2454] T c5c2e425a74b4e64ae70b5def750434c P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 7.
I20260809 15:47:17.789456 23520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a88421dba80d4d3388aa5eb62e23853b" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:47:17.539155 28588 leader_election.cc:288] T 3296ef50be254fc29cfc77d22a8f7a23 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:17.605451 28415 leader_election.cc:288] T 953eea268e17491ca76f3a555860a512 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:47:17.619207 28383 raft_consensus.cc:482] T a3622f6e3e3641cea1960e3a31498bb7 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:17.775876 28440 raft_consensus.cc:482] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 65c36d164b0540f099d592f38307ee6c [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
I20260809 15:47:17.604056 28676 raft_consensus.cc:482] T 0da4d5283c8a46878bd39b85e5c71f66 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:17.795337 23528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba56725e42f74a1baa4c04d9660727f9" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:17.414690 28686 leader_election.cc:288] T 763e58d6ba644eea830a6c1664d39c9b P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:17.768872 28564 raft_consensus.cc:482] T 7baa2492df0340c4a5ac241959ce1ac3 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:17.797758 28597 raft_consensus.cc:482] T a3622f6e3e3641cea1960e3a31498bb7 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:17.332438 28290 raft_consensus.cc:504] T 563daf42d7784db195ea99c3cb21b370 P 61e69695258f48baa984c9d395e8ce3f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:17.762053 28725 raft_consensus.cc:482] T 16ca412a0dae4feebe85dd01bf1a9173 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
I20260809 15:47:17.802512 23274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99bdb93503db4de4a1cfb0161ea0c07c" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:17.780040 23270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4564dfc6d4094c1b9240b367ccecf3b1" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:17.878075 23270 raft_consensus.cc:2396] T 4564dfc6d4094c1b9240b367ccecf3b1 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d for term 8 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20260809 15:47:17.768827 28719 raft_consensus.cc:482] T cef3e7277d7d4760824a3436ed4a847d P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:47:17.819509 23520 raft_consensus.cc:3037] T a88421dba80d4d3388aa5eb62e23853b P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Advancing to term 7
I20260809 15:47:17.820441 23228 leader_election.cc:302] T 953eea268e17491ca76f3a555860a512 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:47:17.815058 23399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "763e58d6ba644eea830a6c1664d39c9b" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
W20260809 15:47:17.895519 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 75152.2us
I20260809 15:47:17.895692 23399 raft_consensus.cc:3037] T 763e58d6ba644eea830a6c1664d39c9b P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Advancing to term 8
I20260809 15:47:17.826195 23526 raft_consensus.cc:2454] T 2424fd9bd7f14e508fd106cbfda786a4 P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 8.
I20260809 15:47:17.896196 23228 leader_election.cc:302] T 4564dfc6d4094c1b9240b367ccecf3b1 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:47:17.896596 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "213543cdbe814c89a17a963469922c1b" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:17.827356 23396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998c6cee72cf40228c6dba064c0db2da" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:17.828506 23283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ced718c1a5c4da79cb8c1ebd1435b87" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:17.897660 23283 raft_consensus.cc:2379] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d in current term 5: Already voted for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in this term.
I20260809 15:47:17.779800 28457 raft_consensus.cc:482] T 907bacfa79d04a55a68c8dbe60056462 P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
I20260809 15:47:17.376250 28425 raft_consensus.cc:504] T 00a1881267a14ba8b78c81040f17a014 P 65c36d164b0540f099d592f38307ee6c [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:17.849303 23528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c29937a80724ddb9d07a27534a146f3" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 9 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:17.851009 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae6d2c08a7a94f58ac8dd698e3fd456c" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:17.789412 28587 leader_election.cc:288] T d0721a4403a440f383318a3cea9d1442 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 9 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:47:17.856530 23517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73f6f44c259547f1b05a3a434616fa54" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:47:17.794422 23472 leader_election.cc:302] T 30c1740ef6874df0a332fbd6fd4b03ce P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:47:17.758208 27493 raft_consensus.cc:504] T 7660dfa2ef694927b8eda4100a2bcd9b P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:17.861294 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bcac4b24e7a4448abdc934bba5a624e" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:47:17.861847 23393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7715d71dbac447b9351fc5eb36c5e80" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:17.399430 28522 raft_consensus.cc:504] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:17.800535 23522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00a1881267a14ba8b78c81040f17a014" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:17.864207 23397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9e7cadcd24d45fcbfa84209225e8f2d" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:17.806753 28431 leader_election.cc:288] T 0da4d5283c8a46878bd39b85e5c71f66 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
W20260809 15:47:17.865756 28709 log.cc:924] Time spent T 5ad8a3b25d7b49e5a2142d5709edd0c7 P c0f38c1908414566aa9115bb2cbdd85d: Append to log took a long time: real 0.128s user 0.000s sys 0.001s
I20260809 15:47:17.330053 24451 rpcz_store.cc:276] 0809 15:47:14.497751 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:14.500010 (+ 2259us) service_pool.cc:227] Handling call
0809 15:47:15.299510 (+799500us) spinlock_profiling.cc:243] Waited 2.04 ms on lock 0x7b5800005498. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e613cf11 00007f34e62c0851 00007f34e62c0702 00007f34ef1b0b09 00007f34ee0ec730 00007f34ee1b279b 00007f34ee1b2821 00007f34ee1762b0 00007f34ee1617f9 00007f34ee1612a2 00007f34e7847684
0809 15:47:15.299718 (+ 208us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:47:15.299913 (+ 195us) raft_consensus.cc:1520] Early marking committed up to index 8
0809 15:47:16.497275 (+1197362us) log.cc:842] Serialized 14 byte log entry
0809 15:47:17.236293 (+739018us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 15:47:17.237279 (+ 986us) raft_consensus.cc:1631] Marking committed up to 8
0809 15:47:17.246153 (+ 8874us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:47:17.262043 (+ 15890us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:47:17.274923 (+ 12880us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":1149906,"prepare.run_cpu_time_us":365,"prepare.run_wall_time_us":502,"raft.queue_time_us":60153,"raft.run_cpu_time_us":72,"raft.run_wall_time_us":72,"spinlock_wait_cycles":4702080,"thread_start_us":1877183,"threads_started":2,"wal-append.queue_time_us":755747}
I20260809 15:47:17.866276 28564 raft_consensus.cc:504] T 7baa2492df0340c4a5ac241959ce1ac3 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:17.807300 28246 consensus_queue.cc:1046] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.060s
I20260809 15:47:17.863252 28248 leader_election.cc:288] T 4bcac4b24e7a4448abdc934bba5a624e P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:17.849398 28383 raft_consensus.cc:504] T a3622f6e3e3641cea1960e3a31498bb7 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:17.873687 23518 raft_consensus.cc:2454] T 538a228ddfb54bc6a36528d5ff07597b P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 8.
I20260809 15:47:17.829314 28685 raft_consensus.cc:2781] T d1a134ba33ea4076bbaac0ae460c17ba P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Leader election won for term 7
I20260809 15:47:17.898401 23274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5c2e425a74b4e64ae70b5def750434c" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:47:17.937472 28419 raft_consensus.cc:2781] T 4564dfc6d4094c1b9240b367ccecf3b1 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260809 15:47:17.807099 24453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "907bacfa79d04a55a68c8dbe60056462" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f"
I20260809 15:47:17.759384 28714 raft_consensus.cc:504] T 563daf42d7784db195ea99c3cb21b370 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:17.773196 28586 raft_consensus.cc:504] T 472816994d814cb198b69e5248697440 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:17.903036 23396 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55320 (request call id 1642) took 1049 ms. Trace:
I20260809 15:47:17.945161 28587 raft_consensus.cc:2781] T 30c1740ef6874df0a332fbd6fd4b03ce P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260809 15:47:17.977293 23396 rpcz_store.cc:276] 0809 15:47:16.853525 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:17.826701 (+973176us) service_pool.cc:227] Handling call
0809 15:47:17.902976 (+ 76275us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:47:17.934559 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3e94b7e50754a81b3d72a07476cb7b9" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:47:17.935968 23397 raft_consensus.cc:2454] T f9e7cadcd24d45fcbfa84209225e8f2d P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 8.
W20260809 15:47:17.897122 28721 log.cc:924] Time spent T 7660dfa2ef694927b8eda4100a2bcd9b P c0f38c1908414566aa9115bb2cbdd85d: Append to log took a long time: real 0.071s user 0.000s sys 0.000s
I20260809 15:47:17.936623 23393 raft_consensus.cc:2454] T f7715d71dbac447b9351fc5eb36c5e80 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 6.
I20260809 15:47:17.802202 28319 raft_consensus.cc:504] T d0721a4403a440f383318a3cea9d1442 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:17.938838 23528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3622f6e3e3641cea1960e3a31498bb7" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
W20260809 15:47:17.937831 23472 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 143481us
I20260809 15:47:17.985296 23472 leader_election.cc:302] T c5c2e425a74b4e64ae70b5def750434c P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:47:17.944200 23270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998c6cee72cf40228c6dba064c0db2da" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:17.944377 23522 raft_consensus.cc:2454] T 00a1881267a14ba8b78c81040f17a014 P 65c36d164b0540f099d592f38307ee6c [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 9.
I20260809 15:47:17.946825 23520 raft_consensus.cc:2454] T a88421dba80d4d3388aa5eb62e23853b P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 7.
I20260809 15:47:17.847327 28631 raft_consensus.cc:504] T 73f6f44c259547f1b05a3a434616fa54 P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:17.949707 23399 raft_consensus.cc:2454] T 763e58d6ba644eea830a6c1664d39c9b P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 8.
I20260809 15:47:17.771965 27892 raft_consensus.cc:504] T 28e17dcb5efe45b8b6a8c42da03b4819 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:17.846292 28580 raft_consensus.cc:504] T d2d4f749fcfd4fb3bfb8115af3471598 P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:17.908314 28326 raft_consensus.cc:2781] T 953eea268e17491ca76f3a555860a512 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260809 15:47:17.955698 23283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0741ab44d99439abc3aa3bdd27ee160" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:17.957175 23518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3924b648461e463b9d7925e5671d68ae" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:47:17.972358 28419 raft_consensus.cc:482] T 4564dfc6d4094c1b9240b367ccecf3b1 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:47:17.857409 28676 raft_consensus.cc:504] T 0da4d5283c8a46878bd39b85e5c71f66 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:17.857071 28440 raft_consensus.cc:504] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 65c36d164b0540f099d592f38307ee6c [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:17.960403 23274 raft_consensus.cc:2379] T c5c2e425a74b4e64ae70b5def750434c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c in current term 7: Already voted for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in this term.
I20260809 15:47:17.788354 28679 raft_consensus.cc:504] T 998c6cee72cf40228c6dba064c0db2da P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:17.972733 23519 raft_consensus.cc:1226] T 7baa2492df0340c4a5ac241959ce1ac3 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Rejecting Update request from peer 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
W20260809 15:47:17.908310 24406 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 251475us
I20260809 15:47:17.979619 23527 raft_consensus.cc:3037] T d3e94b7e50754a81b3d72a07476cb7b9 P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Advancing to term 9
I20260809 15:47:17.980266 23397 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55320 (request call id 1653) took 1082 ms. Trace:
I20260809 15:47:18.002130 23397 rpcz_store.cc:276] 0809 15:47:16.897543 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:17.844978 (+947435us) service_pool.cc:227] Handling call
0809 15:47:17.980215 (+135237us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:47:17.867676 28597 raft_consensus.cc:504] T a3622f6e3e3641cea1960e3a31498bb7 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:17.987200 23399 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55332 (request call id 1518) took 1148 ms. Trace:
I20260809 15:47:17.934764 28457 raft_consensus.cc:504] T 907bacfa79d04a55a68c8dbe60056462 P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:18.052444 28563 leader_election.cc:288] T 07fcabb569414bddbd39df6cedd24e64 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:18.053295 23283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3622f6e3e3641cea1960e3a31498bb7" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:17.990127 23518 raft_consensus.cc:3037] T 3924b648461e463b9d7925e5671d68ae P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Advancing to term 10
I20260809 15:47:17.897181 23526 raft_consensus.cc:2454] T 213543cdbe814c89a17a963469922c1b P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 4.
I20260809 15:47:17.970062 28280 leader_election.cc:288] T e71a05ba1d4d42bd95c3f85f9313399e P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:47:18.053930 23283 raft_consensus.cc:2454] T a3622f6e3e3641cea1960e3a31498bb7 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 6.
I20260809 15:47:17.899770 28725 raft_consensus.cc:504] T 16ca412a0dae4feebe85dd01bf1a9173 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:18.033838 23399 rpcz_store.cc:276] 0809 15:47:16.838394 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:17.814845 (+976451us) service_pool.cc:227] Handling call
0809 15:47:17.987145 (+172300us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:47:18.082904 23399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00a1881267a14ba8b78c81040f17a014" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:17.987243 28462 raft_consensus.cc:3032] T 0c29937a80724ddb9d07a27534a146f3 P 65c36d164b0540f099d592f38307ee6c [term 7 LEADER]: Stepping down as leader of term 7
I20260809 15:47:17.999344 28326 raft_consensus.cc:482] T 953eea268e17491ca76f3a555860a512 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:47:18.050999 28691 leader_election.cc:288] T ba56725e42f74a1baa4c04d9660727f9 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:18.051810 28688 leader_election.cc:288] T 6ae14a74a26f434db08c652e097385c8 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:18.052260 28383 leader_election.cc:288] T a3622f6e3e3641cea1960e3a31498bb7 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:17.978673 28587 raft_consensus.cc:482] T 30c1740ef6874df0a332fbd6fd4b03ce P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:47:17.984298 28521 leader_election.cc:288] T 0c29937a80724ddb9d07a27534a146f3 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:17.902776 28719 raft_consensus.cc:504] T cef3e7277d7d4760824a3436ed4a847d P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:18.040007 23527 raft_consensus.cc:2454] T d3e94b7e50754a81b3d72a07476cb7b9 P 65c36d164b0540f099d592f38307ee6c [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 9.
I20260809 15:47:18.139778 23400 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55320 (request call id 1631) took 3607 ms. Trace:
W20260809 15:47:18.167461 24410 leader_election.cc:334] T cee02fff2c9c48a19f43eb358a89e7b0 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 9 election: RPC error from VoteRequest() call to peer 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553): Timed out: RequestConsensusVote RPC to 127.20.202.67:40553 timed out after 1.575s (SENT)
W20260809 15:47:18.393723 23394 consensus_meta.cc:287] T 2424fd9bd7f14e508fd106cbfda786a4 P c0f38c1908414566aa9115bb2cbdd85d: Time spent flushing consensus metadata: real 0.700s user 0.004s sys 0.004s
I20260809 15:47:18.485117 23394 raft_consensus.cc:2454] T 2424fd9bd7f14e508fd106cbfda786a4 P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 8.
W20260809 15:47:18.485807 23394 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55332 (request call id 1513) took 1670 ms (client timeout 1687 ms). Trace:
W20260809 15:47:18.486249 23394 rpcz_store.cc:269] 0809 15:47:16.819235 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:17.329116 (+509881us) service_pool.cc:227] Handling call
0809 15:47:18.485749 (+1156633us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:47:18.085683 23517 raft_consensus.cc:3037] T 73f6f44c259547f1b05a3a434616fa54 P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Advancing to term 9
I20260809 15:47:18.487416 23394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ae14a74a26f434db08c652e097385c8" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:18.488197 23394 raft_consensus.cc:2379] T 6ae14a74a26f434db08c652e097385c8 P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c in current term 8: Already voted for candidate c0f38c1908414566aa9115bb2cbdd85d in this term.
W20260809 15:47:18.484565 24410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 317136us
I20260809 15:47:18.263751 28745 consensus_queue.cc:1035] T 294d4bcc1e7a477cb56455581277a60f P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.225s
I20260809 15:47:18.084625 23393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3924b648461e463b9d7925e5671d68ae" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:18.490103 23393 raft_consensus.cc:2379] T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 61e69695258f48baa984c9d395e8ce3f in current term 10: Already voted for candidate c0f38c1908414566aa9115bb2cbdd85d in this term.
I20260809 15:47:18.481514 23270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30c1740ef6874df0a332fbd6fd4b03ce" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:18.491098 23393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7660dfa2ef694927b8eda4100a2bcd9b" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:18.492350 23393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c29937a80724ddb9d07a27534a146f3" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 9 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:18.481768 23283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e14d118b2eaa42aa8a7cd4025aba7128" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:18.089336 23518 raft_consensus.cc:2454] T 3924b648461e463b9d7925e5671d68ae P 65c36d164b0540f099d592f38307ee6c [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 10.
I20260809 15:47:18.138790 23528 raft_consensus.cc:2454] T a3622f6e3e3641cea1960e3a31498bb7 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 6.
I20260809 15:47:18.552462 23393 raft_consensus.cc:2454] T 0c29937a80724ddb9d07a27534a146f3 P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 8.
I20260809 15:47:18.553015 23393 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55332 (request call id 1528) took 1271 ms. Trace:
I20260809 15:47:18.481215 23396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "213543cdbe814c89a17a963469922c1b" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
W20260809 15:47:18.167517 23353 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.548s user 0.007s sys 0.024s
I20260809 15:47:17.999630 28680 leader_election.cc:288] T 0acc47ce21c3422698d1ef7277c3ed63 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:47:18.549849 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5712f6b6fffc4cc5a8ddfbe63e4097df" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:18.540087 23517 raft_consensus.cc:2454] T 73f6f44c259547f1b05a3a434616fa54 P 65c36d164b0540f099d592f38307ee6c [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 9.
I20260809 15:47:18.555828 23396 raft_consensus.cc:2379] T 213543cdbe814c89a17a963469922c1b P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 in current term 4: Already voted for candidate c0f38c1908414566aa9115bb2cbdd85d in this term.
I20260809 15:47:18.557144 23396 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55332 (request call id 1530) took 1160 ms. Trace:
I20260809 15:47:18.557479 23396 rpcz_store.cc:276] 0809 15:47:17.397234 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:17.978089 (+580855us) service_pool.cc:227] Handling call
0809 15:47:18.557087 (+578998us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:47:18.482702 23399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "907bacfa79d04a55a68c8dbe60056462" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:47:18.558687 23399 raft_consensus.cc:3037] T 907bacfa79d04a55a68c8dbe60056462 P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Advancing to term 9
I20260809 15:47:18.489017 23394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0721a4403a440f383318a3cea9d1442" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
W20260809 15:47:18.554178 23353 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.935s user 0.008s sys 0.025s
I20260809 15:47:18.563609 23397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3924b648461e463b9d7925e5671d68ae" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
W20260809 15:47:18.563827 23353 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 944939us
I20260809 15:47:18.553761 23393 rpcz_store.cc:276] 0809 15:47:17.281209 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:18.492100 (+1210891us) service_pool.cc:227] Handling call
0809 15:47:18.552963 (+ 60863us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 15:47:18.155987 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 259840us
I20260809 15:47:18.564256 23518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ced718c1a5c4da79cb8c1ebd1435b87" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:18.564865 23228 leader_election.cc:302] T ba56725e42f74a1baa4c04d9660727f9 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:47:18.565726 23228 leader_election.cc:302] T f7715d71dbac447b9351fc5eb36c5e80 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:47:17.989964 28246 raft_consensus.cc:2781] T c5c2e425a74b4e64ae70b5def750434c P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Leader election won for term 7
I20260809 15:47:18.033663 28419 raft_consensus.cc:3037] T 4564dfc6d4094c1b9240b367ccecf3b1 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Advancing to term 8
I20260809 15:47:18.484153 23400 rpcz_store.cc:276] 0809 15:47:14.532722 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:14.648754 (+116032us) service_pool.cc:227] Handling call
0809 15:47:14.672928 (+ 24174us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 15:47:14.674549 (+ 1621us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:47:14.674586 (+ 37us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 15:47:16.692138 (+2017552us) spinlock_profiling.cc:243] Waited 4.06 ms on lock 0x7b2c0005c9b8. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e7c03e11 00007f34e7c13442 00007f34e925bc11 00007f34e922058a 00007f34e921f99e 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772
0809 15:47:16.692487 (+ 349us) log.cc:842] Serialized 26 byte log entry
0809 15:47:17.277418 (+584931us) raft_consensus.cc:1631] Marking committed up to 8
0809 15:47:17.301210 (+ 23792us) log.cc:842] Serialized 14 byte log entry
0809 15:47:17.301362 (+ 152us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:47:17.301435 (+ 73us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 15:47:18.082408 (+780973us) raft_consensus.cc:1671] finished
0809 15:47:18.139579 (+ 57171us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:47:18.139718 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":15436,"prepare.queue_time_us":25529,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":15875,"raft.queue_time_us":74355,"raft.run_cpu_time_us":104,"raft.run_wall_time_us":8506,"spinlock_wait_cycles":21529216,"thread_start_us":569702,"threads_started":1,"wal-append.queue_time_us":593170}
I20260809 15:47:18.138337 23274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1a134ba33ea4076bbaac0ae460c17ba" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:18.569197 23399 raft_consensus.cc:2396] T 907bacfa79d04a55a68c8dbe60056462 P c0f38c1908414566aa9115bb2cbdd85d [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 for term 9 because replica has last-logged OpId of term: 8 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
I20260809 15:47:18.574165 23399 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55332 (request call id 1531) took 1123 ms. Trace:
I20260809 15:47:18.572212 23397 raft_consensus.cc:2379] T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 61e69695258f48baa984c9d395e8ce3f in current term 10: Already voted for candidate c0f38c1908414566aa9115bb2cbdd85d in this term.
I20260809 15:47:18.574728 23397 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52856 (request call id 427) took 1103 ms. Trace:
I20260809 15:47:18.575001 23397 rpcz_store.cc:276] 0809 15:47:17.470816 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:18.002784 (+531968us) service_pool.cc:227] Handling call
0809 15:47:18.574678 (+571894us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 15:47:18.569949 24406 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.566s user 0.012s sys 0.012s
W20260809 15:47:18.575803 24406 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.572s user 0.013s sys 0.013s
W20260809 15:47:18.576138 24406 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 575349us
I20260809 15:47:18.577006 24406 leader_election.cc:302] T cee02fff2c9c48a19f43eb358a89e7b0 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:47:18.577220 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4564dfc6d4094c1b9240b367ccecf3b1" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:18.577710 23399 rpcz_store.cc:276] 0809 15:47:17.451102 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:18.083762 (+632660us) service_pool.cc:227] Handling call
0809 15:47:18.574115 (+490353us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:47:18.578896 23399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "953eea268e17491ca76f3a555860a512" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:18.579586 23399 raft_consensus.cc:2454] T 953eea268e17491ca76f3a555860a512 P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 8.
W20260809 15:47:18.579640 23470 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.615s user 0.009s sys 0.025s
I20260809 15:47:18.580667 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1a134ba33ea4076bbaac0ae460c17ba" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:18.581102 23399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3e94b7e50754a81b3d72a07476cb7b9" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
W20260809 15:47:18.581259 23470 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.616s user 0.009s sys 0.025s
W20260809 15:47:18.582077 23470 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 617290us
I20260809 15:47:18.583443 23399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad8a3b25d7b49e5a2142d5709edd0c7" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:18.581439 23527 raft_consensus.cc:2379] T d1a134ba33ea4076bbaac0ae460c17ba P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d in current term 7: Already voted for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in this term.
I20260809 15:47:18.584635 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9e7cadcd24d45fcbfa84209225e8f2d" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 9 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:18.585388 23527 raft_consensus.cc:2396] T f9e7cadcd24d45fcbfa84209225e8f2d P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d for term 9 because replica has last-logged OpId of term: 7 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 4.
W20260809 15:47:18.583662 23282 consensus_meta.cc:287] T 7baa2492df0340c4a5ac241959ce1ac3 P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent flushing consensus metadata: real 0.790s user 0.008s sys 0.000s
I20260809 15:47:18.490706 23270 raft_consensus.cc:2454] T 30c1740ef6874df0a332fbd6fd4b03ce P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 8.
I20260809 15:47:18.587742 23526 raft_consensus.cc:2396] T 4564dfc6d4094c1b9240b367ccecf3b1 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d for term 8 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20260809 15:47:18.590260 24406 leader_election.cc:302] T 3d880de657e544468c82157769622a26 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:47:18.591853 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7715d71dbac447b9351fc5eb36c5e80" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:18.593892 23283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ae14a74a26f434db08c652e097385c8" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:18.597990 23394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2db057fd10e149a9895dff6ef8c9f038" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:18.598665 23394 raft_consensus.cc:2454] T 2db057fd10e149a9895dff6ef8c9f038 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 7.
I20260809 15:47:18.599190 23394 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55332 (request call id 1519) took 1759 ms. Trace:
I20260809 15:47:18.599491 23394 rpcz_store.cc:276] 0809 15:47:16.841630 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:18.597743 (+1756113us) service_pool.cc:227] Handling call
0809 15:47:18.599142 (+ 1399us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:47:18.600107 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7660dfa2ef694927b8eda4100a2bcd9b" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:18.600664 23394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2424fd9bd7f14e508fd106cbfda786a4" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 9 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:18.601270 23394 raft_consensus.cc:2454] T 2424fd9bd7f14e508fd106cbfda786a4 P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 8.
I20260809 15:47:18.270784 28763 raft_consensus.cc:482] T e14d118b2eaa42aa8a7cd4025aba7128 P 61e69695258f48baa984c9d395e8ce3f [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
I20260809 15:47:18.493795 28587 raft_consensus.cc:3037] T 30c1740ef6874df0a332fbd6fd4b03ce P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Advancing to term 9
I20260809 15:47:18.265934 28695 leader_election.cc:288] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:18.494581 28326 raft_consensus.cc:3037] T 953eea268e17491ca76f3a555860a512 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Advancing to term 9
I20260809 15:47:18.483754 28462 raft_consensus.cc:729] T 0c29937a80724ddb9d07a27534a146f3 P 65c36d164b0540f099d592f38307ee6c [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:47:18.602613 28425 leader_election.cc:288] T 00a1881267a14ba8b78c81040f17a014 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:47:18.613771 23400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e71a05ba1d4d42bd95c3f85f9313399e" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:18.601389 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3e94b7e50754a81b3d72a07476cb7b9" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:47:18.604254 23518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae6d2c08a7a94f58ac8dd698e3fd456c" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
W20260809 15:47:18.585703 28685 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.604s user 0.003s sys 0.022s
I20260809 15:47:18.610472 23393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3e94b7e50754a81b3d72a07476cb7b9" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:18.604308 23394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae6d2c08a7a94f58ac8dd698e3fd456c" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:18.607144 23397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "213543cdbe814c89a17a963469922c1b" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:18.615061 23527 raft_consensus.cc:2379] T d3e94b7e50754a81b3d72a07476cb7b9 P 65c36d164b0540f099d592f38307ee6c [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 61e69695258f48baa984c9d395e8ce3f in current term 9: Already voted for candidate c0f38c1908414566aa9115bb2cbdd85d in this term.
I20260809 15:47:18.619876 23397 raft_consensus.cc:2454] T 213543cdbe814c89a17a963469922c1b P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 4.
I20260809 15:47:18.593901 23399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a88421dba80d4d3388aa5eb62e23853b" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:18.549079 28383 raft_consensus.cc:482] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:47:18.616809 23396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07fcabb569414bddbd39df6cedd24e64" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:47:18.618073 23400 raft_consensus.cc:2379] T e71a05ba1d4d42bd95c3f85f9313399e P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 61e69695258f48baa984c9d395e8ce3f in current term 7: Already voted for candidate c0f38c1908414566aa9115bb2cbdd85d in this term.
W20260809 15:47:18.616078 23353 connection.cc:232] RPC call timeout handler was delayed by -1.2330567319877446s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.23170000000000002s for a response
I20260809 15:47:18.625174 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3924b648461e463b9d7925e5671d68ae" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:47:18.625720 23396 raft_consensus.cc:3037] T 07fcabb569414bddbd39df6cedd24e64 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Advancing to term 8
I20260809 15:47:18.625886 23526 raft_consensus.cc:2379] T 3924b648461e463b9d7925e5671d68ae P 65c36d164b0540f099d592f38307ee6c [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d in current term 10: Already voted for candidate 61e69695258f48baa984c9d395e8ce3f in this term.
I20260809 15:47:18.619808 23394 raft_consensus.cc:2454] T ae6d2c08a7a94f58ac8dd698e3fd456c P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 8.
I20260809 15:47:18.631193 23283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9e7cadcd24d45fcbfa84209225e8f2d" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 9 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:18.631492 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0741ab44d99439abc3aa3bdd27ee160" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:18.632020 23353 leader_election.cc:302] T 213543cdbe814c89a17a963469922c1b P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
W20260809 15:47:18.618716 28685 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.640s user 0.004s sys 0.022s
I20260809 15:47:18.603392 28758 raft_consensus.cc:2726] T cee02fff2c9c48a19f43eb358a89e7b0 P 61e69695258f48baa984c9d395e8ce3f [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260809 15:47:18.640915 28774 raft_consensus.cc:2781] T ba56725e42f74a1baa4c04d9660727f9 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260809 15:47:18.632026 23283 raft_consensus.cc:2396] T f9e7cadcd24d45fcbfa84209225e8f2d P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d for term 9 because replica has last-logged OpId of term: 7 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 4.
I20260809 15:47:18.641309 28774 raft_consensus.cc:482] T ba56725e42f74a1baa4c04d9660727f9 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:47:18.641597 23283 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53562 (request call id 2003) took 1339 ms. Trace:
I20260809 15:47:18.641654 28774 raft_consensus.cc:3037] T ba56725e42f74a1baa4c04d9660727f9 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Advancing to term 7
I20260809 15:47:18.641994 23283 rpcz_store.cc:276] 0809 15:47:17.302039 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:18.603166 (+1301127us) service_pool.cc:227] Handling call
0809 15:47:18.641547 (+ 38381us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:47:18.642891 23283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "213543cdbe814c89a17a963469922c1b" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:18.635743 23270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9e7cadcd24d45fcbfa84209225e8f2d" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
W20260809 15:47:18.637252 28755 log.cc:924] Time spent T a88421dba80d4d3388aa5eb62e23853b P c0f38c1908414566aa9115bb2cbdd85d: Append to log took a long time: real 0.499s user 0.000s sys 0.001s
I20260809 15:47:18.637791 28440 leader_election.cc:288] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:18.624701 28564 leader_election.cc:288] T 7baa2492df0340c4a5ac241959ce1ac3 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:47:18.639608 28691 raft_consensus.cc:2781] T 213543cdbe814c89a17a963469922c1b P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260809 15:47:18.633528 28248 raft_consensus.cc:482] T e71a05ba1d4d42bd95c3f85f9313399e P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:18.640702 28676 leader_election.cc:288] T 0da4d5283c8a46878bd39b85e5c71f66 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:18.644294 23270 raft_consensus.cc:2454] T f9e7cadcd24d45fcbfa84209225e8f2d P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 8.
I20260809 15:47:18.643442 23283 raft_consensus.cc:2454] T 213543cdbe814c89a17a963469922c1b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 3.
I20260809 15:47:18.648371 23397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0acc47ce21c3422698d1ef7277c3ed63" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 10 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:18.604390 28246 raft_consensus.cc:686] T c5c2e425a74b4e64ae70b5def750434c P 65c36d164b0540f099d592f38307ee6c [term 7 LEADER]: Becoming Leader. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:47:18.650245 23518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5c2e425a74b4e64ae70b5def750434c" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:18.651202 23393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df20671a7b874dbb98c0ef28371c1308" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:18.654099 23400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a51c0baaa6e432997c9d3552af4f254" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:18.665551 23400 raft_consensus.cc:2379] T 0a51c0baaa6e432997c9d3552af4f254 P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 61e69695258f48baa984c9d395e8ce3f in current term 8: Already voted for candidate c0f38c1908414566aa9115bb2cbdd85d in this term.
I20260809 15:47:18.646840 28685 raft_consensus.cc:686] T d1a134ba33ea4076bbaac0ae460c17ba P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 LEADER]: Becoming Leader. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Running, Role: LEADER
I20260809 15:47:18.618014 28586 leader_election.cc:288] T 472816994d814cb198b69e5248697440 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:18.645087 28457 leader_election.cc:288] T 907bacfa79d04a55a68c8dbe60056462 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:18.644579 24453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a823d35f4f348d1b52ba9e6d80b00d2" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:47:18.668912 23397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3e94b7e50754a81b3d72a07476cb7b9" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:47:18.671900 23397 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52856 (request call id 430) took 1145 ms. Trace:
I20260809 15:47:18.672219 23397 rpcz_store.cc:276] 0809 15:47:17.526639 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:18.668679 (+1142040us) service_pool.cc:227] Handling call
0809 15:47:18.671847 (+ 3168us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 15:47:18.663074 28752 log.cc:924] Time spent T c0741ab44d99439abc3aa3bdd27ee160 P 67b3aeadfa7a432abe8af4d292e04ab9: Append to log took a long time: real 0.675s user 0.001s sys 0.000s
I20260809 15:47:18.647898 28319 leader_election.cc:288] T d0721a4403a440f383318a3cea9d1442 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:18.630596 23394 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55332 (request call id 1529) took 1303 ms. Trace:
I20260809 15:47:18.673074 24448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3e94b7e50754a81b3d72a07476cb7b9" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:47:18.657704 28691 raft_consensus.cc:482] T 213543cdbe814c89a17a963469922c1b P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:47:18.674961 23397 raft_consensus.cc:1226] T 0a51c0baaa6e432997c9d3552af4f254 P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Rejecting Update request from peer 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 7. Current term is 8. Ops: []
I20260809 15:47:18.685950 23397 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55332 (request call id 1516) took 1865 ms. Trace:
I20260809 15:47:18.686254 23397 rpcz_store.cc:276] 0809 15:47:16.820030 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:18.674428 (+1854398us) service_pool.cc:227] Handling call
0809 15:47:18.674791 (+ 363us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:47:18.685718 (+ 10927us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:47:18.685891 (+ 173us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:47:18.690876 23397 raft_consensus.cc:1226] T 30c1740ef6874df0a332fbd6fd4b03ce P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Rejecting Update request from peer 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20260809 15:47:18.616415 28763 raft_consensus.cc:504] T e14d118b2eaa42aa8a7cd4025aba7128 P 61e69695258f48baa984c9d395e8ce3f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:18.657958 27892 leader_election.cc:288] T 28e17dcb5efe45b8b6a8c42da03b4819 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:18.676367 23394 rpcz_store.cc:276] 0809 15:47:17.327357 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:18.602225 (+1274868us) service_pool.cc:227] Handling call
0809 15:47:18.630546 (+ 28321us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:47:18.677090 24448 raft_consensus.cc:2379] T d3e94b7e50754a81b3d72a07476cb7b9 P 61e69695258f48baa984c9d395e8ce3f [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c in current term 9: Already voted for candidate 61e69695258f48baa984c9d395e8ce3f in this term.
I20260809 15:47:18.709211 23283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73f6f44c259547f1b05a3a434616fa54" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:47:18.634653 28383 raft_consensus.cc:504] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:18.709769 23270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "213543cdbe814c89a17a963469922c1b" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:18.678514 23399 raft_consensus.cc:1226] T 6ae14a74a26f434db08c652e097385c8 P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Rejecting Update request from peer 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 7. Current term is 8. Ops: []
I20260809 15:47:18.666320 23393 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55320 (request call id 1652) took 1769 ms. Trace:
W20260809 15:47:18.675307 24463 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.511s user 0.009s sys 0.047s
I20260809 15:47:18.710368 23394 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55320 (request call id 1640) took 1857 ms. Trace:
I20260809 15:47:18.710767 23394 rpcz_store.cc:276] 0809 15:47:16.853020 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:18.709567 (+1856547us) service_pool.cc:227] Handling call
0809 15:47:18.709875 (+ 308us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:47:18.710092 (+ 217us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:47:18.710116 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 15:47:18.710151 (+ 35us) raft_consensus.cc:1631] Marking committed up to 3
0809 15:47:18.710181 (+ 30us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:47:18.710229 (+ 48us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:47:18.710325 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:47:18.711232 23393 rpcz_store.cc:276] 0809 15:47:16.897272 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:18.619374 (+1722102us) service_pool.cc:227] Handling call
0809 15:47:18.666267 (+ 46893us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:47:18.710426 23270 raft_consensus.cc:2454] T 213543cdbe814c89a17a963469922c1b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 3.
I20260809 15:47:18.711133 23399 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55332 (request call id 1521) took 1869 ms. Trace:
I20260809 15:47:18.712548 23270 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53562 (request call id 2000) took 1411 ms. Trace:
I20260809 15:47:18.713219 23394 raft_consensus.cc:1226] T d1a134ba33ea4076bbaac0ae460c17ba P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 6. Current term is 7. Ops: []
I20260809 15:47:18.709797 23283 raft_consensus.cc:3037] T 73f6f44c259547f1b05a3a434616fa54 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Advancing to term 9
I20260809 15:47:18.706765 23397 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55332 (request call id 1522) took 1864 ms. Trace:
I20260809 15:47:18.713879 23394 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55320 (request call id 1643) took 1860 ms. Trace:
I20260809 15:47:18.665357 28768 raft_consensus.cc:2781] T f7715d71dbac447b9351fc5eb36c5e80 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260809 15:47:18.712630 23399 rpcz_store.cc:276] 0809 15:47:16.842195 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:18.678074 (+1835879us) service_pool.cc:227] Handling call
0809 15:47:18.678347 (+ 273us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:47:18.710919 (+ 32572us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:47:18.711087 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:47:18.716171 23400 raft_consensus.cc:1226] T 538a228ddfb54bc6a36528d5ff07597b P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Rejecting Update request from peer 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 7. Current term is 8. Ops: []
I20260809 15:47:18.665553 24451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad8a3b25d7b49e5a2142d5709edd0c7" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:47:18.720059 23400 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55332 (request call id 1514) took 1900 ms. Trace:
I20260809 15:47:18.664707 28248 raft_consensus.cc:504] T e71a05ba1d4d42bd95c3f85f9313399e P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:18.713248 23270 rpcz_store.cc:276] 0809 15:47:17.301228 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:18.669640 (+1368412us) service_pool.cc:227] Handling call
0809 15:47:18.712495 (+ 42855us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:47:18.670536 28631 leader_election.cc:288] T 73f6f44c259547f1b05a3a434616fa54 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:47:18.720386 23400 rpcz_store.cc:276] 0809 15:47:16.819525 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:18.674211 (+1854686us) service_pool.cc:227] Handling call
0809 15:47:18.716011 (+ 41800us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:47:18.719823 (+ 3812us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:47:18.719991 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:47:18.673106 28679 leader_election.cc:288] T 998c6cee72cf40228c6dba064c0db2da P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
W20260809 15:47:18.716720 24410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 227741us
I20260809 15:47:18.705622 28676 raft_consensus.cc:482] T d2d4f749fcfd4fb3bfb8115af3471598 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:18.722656 23399 raft_consensus.cc:1261] T ae6d2c08a7a94f58ac8dd698e3fd456c P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260809 15:47:18.731300 23399 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55320 (request call id 1645) took 1877 ms. Trace:
I20260809 15:47:18.724651 24451 raft_consensus.cc:2379] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 61e69695258f48baa984c9d395e8ce3f [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c in current term 6: Already voted for candidate 61e69695258f48baa984c9d395e8ce3f in this term.
I20260809 15:47:18.725765 28248 leader_election.cc:288] T e71a05ba1d4d42bd95c3f85f9313399e P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:18.705406 28691 raft_consensus.cc:3037] T 213543cdbe814c89a17a963469922c1b P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Advancing to term 5
I20260809 15:47:18.727653 23393 raft_consensus.cc:1226] T 763e58d6ba644eea830a6c1664d39c9b P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 7. Current term is 8. Ops: []
I20260809 15:47:18.721782 28768 raft_consensus.cc:482] T f7715d71dbac447b9351fc5eb36c5e80 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:47:18.674832 28580 leader_election.cc:288] T d2d4f749fcfd4fb3bfb8115af3471598 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
W20260809 15:47:18.730006 24410 leader_election.cc:334] T 3924b648461e463b9d7925e5671d68ae P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263): Timed out: RequestConsensusVote RPC to 127.20.202.66:36263 timed out after 1.580s (SENT)
I20260809 15:47:18.731604 23399 rpcz_store.cc:276] 0809 15:47:16.854194 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:18.721566 (+1867372us) service_pool.cc:227] Handling call
0809 15:47:18.721815 (+ 249us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:47:18.731069 (+ 9254us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:47:18.731233 (+ 164us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:47:18.755191 23399 raft_consensus.cc:1226] T 0da4d5283c8a46878bd39b85e5c71f66 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 5. Current term is 6. Ops: []
I20260809 15:47:18.755728 23399 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55320 (request call id 1649) took 1900 ms. Trace:
I20260809 15:47:18.756011 23399 rpcz_store.cc:276] 0809 15:47:16.855099 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:18.754754 (+1899655us) service_pool.cc:227] Handling call
0809 15:47:18.755036 (+ 282us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:47:18.755538 (+ 502us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:47:18.755684 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:47:18.757268 23399 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55320 (request call id 1650) took 1888 ms. Trace:
I20260809 15:47:18.757545 23399 rpcz_store.cc:276] 0809 15:47:16.868937 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:18.756615 (+1887678us) service_pool.cc:227] Handling call
0809 15:47:18.756857 (+ 242us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:47:18.757016 (+ 159us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:47:18.757039 (+ 23us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 15:47:18.757076 (+ 37us) raft_consensus.cc:1631] Marking committed up to 4
0809 15:47:18.757103 (+ 27us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:47:18.757146 (+ 43us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:47:18.757229 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:47:18.715641 28319 raft_consensus.cc:482] T 3296ef50be254fc29cfc77d22a8f7a23 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
I20260809 15:47:18.681559 28725 leader_election.cc:288] T 16ca412a0dae4feebe85dd01bf1a9173 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:18.738337 23393 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55320 (request call id 1641) took 1885 ms. Trace:
I20260809 15:47:18.760591 23393 rpcz_store.cc:276] 0809 15:47:16.853279 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:18.712475 (+1859196us) service_pool.cc:227] Handling call
0809 15:47:18.727472 (+ 14997us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:47:18.738125 (+ 10653us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:47:18.738289 (+ 164us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:47:18.739228 24453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e71a05ba1d4d42bd95c3f85f9313399e" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:47:18.740365 23285 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53554 (request call id 1834) took 4310 ms. Trace:
I20260809 15:47:18.749470 23270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0721a4403a440f383318a3cea9d1442" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:47:18.763645 23270 raft_consensus.cc:3037] T d0721a4403a440f383318a3cea9d1442 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Advancing to term 9
I20260809 15:47:18.714139 23397 rpcz_store.cc:276] 0809 15:47:16.842423 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:18.690367 (+1847944us) service_pool.cc:227] Handling call
0809 15:47:18.690676 (+ 309us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 15:47:18.706537 (+ 15861us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:47:18.706716 (+ 179us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:47:18.765259 23397 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52856 (request call id 425) took 1294 ms. Trace:
I20260809 15:47:18.765911 23397 rpcz_store.cc:276] 0809 15:47:17.470352 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:18.764420 (+1294068us) service_pool.cc:227] Handling call
0809 15:47:18.764811 (+ 391us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:47:18.764978 (+ 167us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:47:18.765000 (+ 22us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 15:47:18.765037 (+ 37us) raft_consensus.cc:1631] Marking committed up to 7
0809 15:47:18.765073 (+ 36us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:47:18.765120 (+ 47us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:47:18.765209 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:47:18.767557 23399 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "763e58d6ba644eea830a6c1664d39c9b"
dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
from {username='slave'} at 127.0.0.1:55320
I20260809 15:47:18.768128 23399 raft_consensus.cc:482] T 763e58d6ba644eea830a6c1664d39c9b P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 15:47:18.768455 23399 raft_consensus.cc:3037] T 763e58d6ba644eea830a6c1664d39c9b P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Advancing to term 9
I20260809 15:47:18.768606 23397 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52856 (request call id 426) took 1298 ms. Trace:
I20260809 15:47:18.769083 23397 rpcz_store.cc:276] 0809 15:47:17.470585 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:18.767732 (+1297147us) service_pool.cc:227] Handling call
0809 15:47:18.768058 (+ 326us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:47:18.768214 (+ 156us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:47:18.768237 (+ 23us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 15:47:18.768271 (+ 34us) raft_consensus.cc:1631] Marking committed up to 5
0809 15:47:18.768297 (+ 26us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:47:18.768342 (+ 45us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:47:18.768558 (+ 216us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:47:18.770780 23397 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52856 (request call id 428) took 1299 ms. Trace:
I20260809 15:47:18.762413 24410 leader_election.cc:302] T 7660dfa2ef694927b8eda4100a2bcd9b P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:18.775420 28255 raft_consensus.cc:2726] T 7660dfa2ef694927b8eda4100a2bcd9b P 61e69695258f48baa984c9d395e8ce3f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260809 15:47:18.776242 24410 leader_election.cc:302] T 0a51c0baaa6e432997c9d3552af4f254 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f; no voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:18.777333 24410 leader_election.cc:302] T d3e94b7e50754a81b3d72a07476cb7b9 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:18.778913 28255 raft_consensus.cc:3037] T 0a51c0baaa6e432997c9d3552af4f254 P 61e69695258f48baa984c9d395e8ce3f [term 7 FOLLOWER]: Advancing to term 8
I20260809 15:47:18.780303 28775 raft_consensus.cc:2726] T d3e94b7e50754a81b3d72a07476cb7b9 P 61e69695258f48baa984c9d395e8ce3f [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260809 15:47:18.748442 28246 consensus_queue.cc:237] T c5c2e425a74b4e64ae70b5def750434c P 65c36d164b0540f099d592f38307ee6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:18.752708 28685 consensus_queue.cc:237] T d1a134ba33ea4076bbaac0ae460c17ba P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:18.782625 28598 leader_election.cc:288] T 0c48a3049897452383257aa757834fb5 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:18.692282 28462 consensus_queue.cc:260] T 0c29937a80724ddb9d07a27534a146f3 P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.3, Last appended by leader: 3, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:18.763415 23285 rpcz_store.cc:276] 0809 15:47:14.429516 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:14.430661 (+ 1145us) service_pool.cc:227] Handling call
0809 15:47:14.439854 (+ 9193us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 15:47:14.440153 (+ 299us) raft_consensus.cc:1520] Early marking committed up to index 6
0809 15:47:14.440178 (+ 25us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 15:47:14.440530 (+ 352us) log.cc:842] Serialized 26 byte log entry
0809 15:47:17.304136 (+2863606us) spinlock_profiling.cc:243] Waited 7.46 ms on lock 0x7b18002651e0. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34ef1978b1 00007f34ef1a4661 00007f34e62da345 00007f34ef111a39 00007f34e62f78bb 00007f34e62f5cbf 00007f34e62f7e22 00007f34e8ea15bb 00007f34e8ea7063 00007f34e8ea7411 00007f34e91f45cf
0809 15:47:17.413029 (+108893us) raft_consensus.cc:1631] Marking committed up to 6
0809 15:47:17.413066 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:47:17.413132 (+ 66us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 15:47:18.730693 (+1317561us) raft_consensus.cc:1671] finished
0809 15:47:18.730711 (+ 18us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:47:18.740306 (+ 9595us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":88,"raft.run_cpu_time_us":106,"raft.run_wall_time_us":9522,"spinlock_wait_cycles":17157888,"thread_start_us":93818,"threads_started":1,"wal-append.queue_time_us":3510641}
I20260809 15:47:18.714296 23394 rpcz_store.cc:276] 0809 15:47:16.853747 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:18.712779 (+1859032us) service_pool.cc:227] Handling call
0809 15:47:18.713074 (+ 295us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:47:18.713672 (+ 598us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:47:18.713833 (+ 161us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 15:47:18.788442 23353 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 152976us
I20260809 15:47:18.789886 23394 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52856 (request call id 429) took 1318 ms. Trace:
I20260809 15:47:18.785324 23397 rpcz_store.cc:276] 0809 15:47:17.471079 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:18.770087 (+1299008us) service_pool.cc:227] Handling call
0809 15:47:18.770354 (+ 267us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:47:18.770516 (+ 162us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:47:18.770539 (+ 23us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 15:47:18.770576 (+ 37us) raft_consensus.cc:1631] Marking committed up to 6
0809 15:47:18.770604 (+ 28us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:47:18.770648 (+ 44us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:47:18.770737 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:47:18.786031 23393 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52856 (request call id 424) took 1316 ms. Trace:
I20260809 15:47:18.791481 23393 rpcz_store.cc:276] 0809 15:47:17.470086 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:18.761282 (+1291196us) service_pool.cc:227] Handling call
0809 15:47:18.785599 (+ 24317us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:47:18.785757 (+ 158us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:47:18.785779 (+ 22us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 15:47:18.785816 (+ 37us) raft_consensus.cc:1631] Marking committed up to 8
0809 15:47:18.785844 (+ 28us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:47:18.785888 (+ 44us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:47:18.785983 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:47:18.793217 23393 raft_consensus.cc:1226] T ba56725e42f74a1baa4c04d9660727f9 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20260809 15:47:18.793797 23393 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55320 (request call id 1661) took 1013 ms. Trace:
I20260809 15:47:18.789846 23353 leader_election.cc:302] T 538a228ddfb54bc6a36528d5ff07597b P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:47:18.794107 23393 rpcz_store.cc:276] 0809 15:47:17.780788 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:18.792412 (+1011624us) service_pool.cc:227] Handling call
0809 15:47:18.792905 (+ 493us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 15:47:18.793539 (+ 634us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:47:18.793751 (+ 212us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:47:18.794725 28248 raft_consensus.cc:2781] T 538a228ddfb54bc6a36528d5ff07597b P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Leader election won for term 8
I20260809 15:47:18.795130 23353 leader_election.cc:302] T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters: 61e69695258f48baa984c9d395e8ce3f
I20260809 15:47:18.753696 28768 raft_consensus.cc:3037] T f7715d71dbac447b9351fc5eb36c5e80 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Advancing to term 7
I20260809 15:47:18.795200 28248 raft_consensus.cc:686] T 538a228ddfb54bc6a36528d5ff07597b P c0f38c1908414566aa9115bb2cbdd85d [term 8 LEADER]: Becoming Leader. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:47:18.796178 23393 raft_consensus.cc:1226] T 907bacfa79d04a55a68c8dbe60056462 P c0f38c1908414566aa9115bb2cbdd85d [term 9 FOLLOWER]: Rejecting Update request from peer 61e69695258f48baa984c9d395e8ce3f for earlier term 8. Current term is 9. Ops: []
W20260809 15:47:18.790933 23468 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.68:34689, user_credentials={real_user=slave}} blocked reactor thread for 179922us
I20260809 15:47:18.807899 23282 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53562 (request call id 1998) took 1507 ms. Trace:
I20260809 15:47:18.797398 23397 raft_consensus.cc:1226] T 4b30f28311964a0b9149a50495e83d8f P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 5. Current term is 6. Ops: [5.5-5.5]
I20260809 15:47:18.798110 23399 raft_consensus.cc:504] T 763e58d6ba644eea830a6c1664d39c9b P c0f38c1908414566aa9115bb2cbdd85d [term 9 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:18.799106 23396 raft_consensus.cc:2454] T 07fcabb569414bddbd39df6cedd24e64 P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 8.
I20260809 15:47:18.805953 23285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0acc47ce21c3422698d1ef7277c3ed63" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 10 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:18.790441 23394 rpcz_store.cc:276] 0809 15:47:17.471322 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:18.789107 (+1317785us) service_pool.cc:227] Handling call
0809 15:47:18.789364 (+ 257us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:47:18.789531 (+ 167us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:47:18.789620 (+ 89us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 15:47:18.789664 (+ 44us) raft_consensus.cc:1631] Marking committed up to 7
0809 15:47:18.789695 (+ 31us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:47:18.789744 (+ 49us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:47:18.789836 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:47:18.734541 28676 raft_consensus.cc:504] T d2d4f749fcfd4fb3bfb8115af3471598 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:18.810963 23282 rpcz_store.cc:276] 0809 15:47:17.300671 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:17.300792 (+ 121us) service_pool.cc:227] Handling call
0809 15:47:17.301032 (+ 240us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:47:18.806751 (+1505719us) raft_consensus.cc:1520] Early marking committed up to index 3
0809 15:47:18.806778 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 15:47:18.806814 (+ 36us) raft_consensus.cc:1631] Marking committed up to 3
0809 15:47:18.806841 (+ 27us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:47:18.806900 (+ 59us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:47:18.807848 (+ 948us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":78492,"raft.run_cpu_time_us":110,"raft.run_wall_time_us":4811,"thread_start_us":219953,"threads_started":1}
I20260809 15:47:18.812692 23285 raft_consensus.cc:2454] T 0acc47ce21c3422698d1ef7277c3ed63 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 9.
I20260809 15:47:18.813413 23400 raft_consensus.cc:1226] T 07fcabb569414bddbd39df6cedd24e64 P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 7. Current term is 8. Ops: []
I20260809 15:47:18.817648 23283 raft_consensus.cc:2454] T 73f6f44c259547f1b05a3a434616fa54 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 9.
I20260809 15:47:18.819932 23283 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53562 (request call id 2001) took 1518 ms. Trace:
I20260809 15:47:18.813767 23396 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55332 (request call id 1525) took 1587 ms. Trace:
I20260809 15:47:18.811067 23468 leader_election.cc:302] T 4bcac4b24e7a4448abdc934bba5a624e P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:18.820463 23396 rpcz_store.cc:276] 0809 15:47:17.226219 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:18.586814 (+1360595us) service_pool.cc:227] Handling call
0809 15:47:18.813708 (+226894us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 15:47:18.808187 28274 consensus_meta.cc:287] T 297c82ed906d4b48b184985eed3de601 P 65c36d164b0540f099d592f38307ee6c: Time spent flushing consensus metadata: real 0.997s user 0.009s sys 0.005s
W20260809 15:47:18.823314 24406 leader_election.cc:334] T cee02fff2c9c48a19f43eb358a89e7b0 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459): Timed out: RequestConsensusVote RPC to 127.20.202.65:35459 timed out after 2.093s (SENT)
I20260809 15:47:18.791836 28719 leader_election.cc:288] T cef3e7277d7d4760824a3436ed4a847d P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:47:18.820304 23283 rpcz_store.cc:276] 0809 15:47:17.301481 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:18.668376 (+1366895us) service_pool.cc:227] Handling call
0809 15:47:18.819863 (+151487us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:47:18.825070 23285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7715d71dbac447b9351fc5eb36c5e80" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:18.766664 28319 raft_consensus.cc:504] T 3296ef50be254fc29cfc77d22a8f7a23 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
W20260809 15:47:18.828235 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 262568us
I20260809 15:47:18.808307 28208 raft_consensus.cc:2781] T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d [term 9 FOLLOWER]: Leader election won for term 9
I20260809 15:47:18.830683 23529 raft_consensus.cc:2396] T 297c82ed906d4b48b184985eed3de601 P 65c36d164b0540f099d592f38307ee6c [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d for term 10 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20260809 15:47:18.831037 23282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "953eea268e17491ca76f3a555860a512" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:18.832795 23353 leader_election.cc:302] T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c
I20260809 15:47:18.824314 23399 leader_election.cc:288] T 763e58d6ba644eea830a6c1664d39c9b P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 9 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:18.836892 23228 leader_election.cc:302] T 763e58d6ba644eea830a6c1664d39c9b P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:47:18.837385 24452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16ca412a0dae4feebe85dd01bf1a9173" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:47:18.838148 23228 leader_election.cc:302] T 2424fd9bd7f14e508fd106cbfda786a4 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:47:18.839359 23228 leader_election.cc:302] T 0c29937a80724ddb9d07a27534a146f3 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:47:18.841279 23283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00a1881267a14ba8b78c81040f17a014" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:18.843330 23399 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RunLeaderElection from 127.0.0.1:55320 (request call id 1651) took 1967 ms. Trace:
I20260809 15:47:18.841557 23228 leader_election.cc:302] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:18.841847 23283 raft_consensus.cc:2454] T 00a1881267a14ba8b78c81040f17a014 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 9.
I20260809 15:47:18.843748 23399 rpcz_store.cc:276] 0809 15:47:16.896887 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:18.758280 (+1861393us) service_pool.cc:227] Handling call
0809 15:47:18.836721 (+ 78441us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":159}
I20260809 15:47:18.846835 23228 leader_election.cc:302] T 2db057fd10e149a9895dff6ef8c9f038 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:47:18.848022 24448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c48a3049897452383257aa757834fb5" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:47:18.827662 28525 raft_consensus.cc:2726] T 4bcac4b24e7a4448abdc934bba5a624e P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260809 15:47:18.847327 24463 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38724 (request call id 583) took 1151 ms. Trace:
I20260809 15:47:18.872695 23472 leader_election.cc:302] T f9e7cadcd24d45fcbfa84209225e8f2d P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:47:18.840345 28474 raft_consensus.cc:482] T 297c82ed906d4b48b184985eed3de601 P 65c36d164b0540f099d592f38307ee6c [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:18.842509 28679 raft_consensus.cc:2781] T 763e58d6ba644eea830a6c1664d39c9b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Leader election won for term 8
I20260809 15:47:18.883550 23472 leader_election.cc:302] T 953eea268e17491ca76f3a555860a512 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:47:18.844587 27892 raft_consensus.cc:2781] T 0c29937a80724ddb9d07a27534a146f3 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260809 15:47:18.869086 24463 rpcz_store.cc:276] 0809 15:47:17.696143 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:17.696268 (+ 125us) service_pool.cc:227] Handling call
0809 15:47:17.771000 (+ 74732us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 15:47:18.089541 (+318541us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:47:18.089575 (+ 34us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 15:47:18.137688 (+ 48113us) log.cc:842] Serialized 26 byte log entry
0809 15:47:18.710734 (+573046us) raft_consensus.cc:1631] Marking committed up to 7
0809 15:47:18.846829 (+136095us) log.cc:842] Serialized 14 byte log entry
0809 15:47:18.847034 (+ 205us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:47:18.847102 (+ 68us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 15:47:18.847139 (+ 37us) raft_consensus.cc:1671] finished
0809 15:47:18.847149 (+ 10us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:47:18.847253 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":232075,"raft.run_cpu_time_us":153,"raft.run_wall_time_us":49509,"thread_start_us":979448,"threads_started":3,"wal-append.queue_time_us":586667}
I20260809 15:47:18.888417 23283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7baa2492df0340c4a5ac241959ce1ac3" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:18.842653 28779 raft_consensus.cc:2781] T 2424fd9bd7f14e508fd106cbfda786a4 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Leader election won for term 8
I20260809 15:47:18.898269 23472 leader_election.cc:302] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:18.853734 28290 leader_election.cc:288] T 563daf42d7784db195ea99c3cb21b370 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:47:18.901264 23472 leader_election.cc:302] T 2424fd9bd7f14e508fd106cbfda786a4 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:47:18.885214 28580 raft_consensus.cc:2781] T f9e7cadcd24d45fcbfa84209225e8f2d P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260809 15:47:18.890573 28474 raft_consensus.cc:504] T 297c82ed906d4b48b184985eed3de601 P 65c36d164b0540f099d592f38307ee6c [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:18.891095 24451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a823d35f4f348d1b52ba9e6d80b00d2" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:47:18.890789 27892 raft_consensus.cc:482] T 0c29937a80724ddb9d07a27534a146f3 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:47:18.908241 28598 raft_consensus.cc:3037] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Advancing to term 6
I20260809 15:47:18.910017 23472 leader_election.cc:302] T d3e94b7e50754a81b3d72a07476cb7b9 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d
W20260809 15:47:18.910131 24410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 124147us
I20260809 15:47:18.920090 23283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73f6f44c259547f1b05a3a434616fa54" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:18.926030 23283 raft_consensus.cc:2379] T 73f6f44c259547f1b05a3a434616fa54 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c in current term 9: Already voted for candidate c0f38c1908414566aa9115bb2cbdd85d in this term.
I20260809 15:47:18.891795 28383 leader_election.cc:288] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:18.931704 23470 leader_election.cc:302] T 6ae14a74a26f434db08c652e097385c8 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d
W20260809 15:47:18.904381 28577 thread.cc:636] tablet-download-pool [worker] (thread pool) Time spent creating pthread: real 3.784s user 0.057s sys 0.062s
I20260809 15:47:18.940965 28788 raft_consensus.cc:2726] T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260809 15:47:18.921049 28319 leader_election.cc:288] T 3296ef50be254fc29cfc77d22a8f7a23 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
W20260809 15:47:18.920636 28603 log.cc:924] Time spent T 472816994d814cb198b69e5248697440 P c0f38c1908414566aa9115bb2cbdd85d: Append to log took a long time: real 3.833s user 0.012s sys 0.015s
I20260809 15:47:18.915549 27892 raft_consensus.cc:3037] T 0c29937a80724ddb9d07a27534a146f3 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Advancing to term 9
W20260809 15:47:18.921783 28635 log.cc:924] Time spent T 4b30f28311964a0b9149a50495e83d8f P 67b3aeadfa7a432abe8af4d292e04ab9: Append to log took a long time: real 3.609s user 0.006s sys 0.015s
I20260809 15:47:18.908455 28725 raft_consensus.cc:2781] T 953eea268e17491ca76f3a555860a512 P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260809 15:47:18.908418 28782 raft_consensus.cc:2781] T 2db057fd10e149a9895dff6ef8c9f038 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260809 15:47:18.910717 28787 raft_consensus.cc:482] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
I20260809 15:47:18.943256 28676 leader_election.cc:288] T d2d4f749fcfd4fb3bfb8115af3471598 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
W20260809 15:47:18.944300 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 92519.5us
I20260809 15:47:18.919829 28763 leader_election.cc:288] T e14d118b2eaa42aa8a7cd4025aba7128 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
W20260809 15:47:18.946682 23345 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.68:34689, user_credentials={real_user=slave}} blocked reactor thread for 54031.9us
I20260809 15:47:18.905959 28208 raft_consensus.cc:686] T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d [term 9 LEADER]: Becoming Leader. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:47:18.971143 28208 consensus_queue.cc:237] T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:18.921052 27988 raft_consensus.cc:482] T df20671a7b874dbb98c0ef28371c1308 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
I20260809 15:47:18.973778 27988 raft_consensus.cc:504] T df20671a7b874dbb98c0ef28371c1308 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:18.976835 27988 leader_election.cc:288] T df20671a7b874dbb98c0ef28371c1308 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
W20260809 15:47:18.957077 23353 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 124347us
I20260809 15:47:18.943799 24451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3296ef50be254fc29cfc77d22a8f7a23" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:47:18.921988 28274 raft_consensus.cc:2781] T 2424fd9bd7f14e508fd106cbfda786a4 P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260809 15:47:18.981433 28274 raft_consensus.cc:482] T 2424fd9bd7f14e508fd106cbfda786a4 P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:47:18.982050 28274 raft_consensus.cc:3037] T 2424fd9bd7f14e508fd106cbfda786a4 P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Advancing to term 9
I20260809 15:47:18.903576 24453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472816994d814cb198b69e5248697440" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:47:18.911952 28580 raft_consensus.cc:482] T f9e7cadcd24d45fcbfa84209225e8f2d P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:47:18.964121 23283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30c1740ef6874df0a332fbd6fd4b03ce" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:18.998744 23283 raft_consensus.cc:2454] T 30c1740ef6874df0a332fbd6fd4b03ce P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 8.
I20260809 15:47:18.903254 28784 raft_consensus.cc:2726] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260809 15:47:18.931846 23472 leader_election.cc:302] T 213543cdbe814c89a17a963469922c1b P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:47:18.968165 23228 leader_election.cc:302] T ae6d2c08a7a94f58ac8dd698e3fd456c P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters:
W20260809 15:47:18.959715 28577 thread.cc:602] tablet-download-pool [worker] (thread pool) Time spent starting thread: real 3.849s user 0.057s sys 0.062s
I20260809 15:47:18.887666 28248 consensus_queue.cc:237] T 538a228ddfb54bc6a36528d5ff07597b P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:18.959970 28681 raft_consensus.cc:2726] T d3e94b7e50754a81b3d72a07476cb7b9 P 65c36d164b0540f099d592f38307ee6c [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260809 15:47:19.002882 23399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "563daf42d7784db195ea99c3cb21b370" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:19.003175 23396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3924b648461e463b9d7925e5671d68ae" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
W20260809 15:47:19.005112 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 75267.3us
I20260809 15:47:18.961278 28725 raft_consensus.cc:482] T 953eea268e17491ca76f3a555860a512 P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:47:18.917392 28326 raft_consensus.cc:504] T 953eea268e17491ca76f3a555860a512 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:18.966192 28782 raft_consensus.cc:482] T 2db057fd10e149a9895dff6ef8c9f038 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260809 15:47:19.003091 23472 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 69572.2us
I20260809 15:47:19.000437 28580 raft_consensus.cc:3037] T f9e7cadcd24d45fcbfa84209225e8f2d P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Advancing to term 9
I20260809 15:47:19.009967 23283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99bdb93503db4de4a1cfb0161ea0c07c" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:18.968314 28785 raft_consensus.cc:3037] T 6ae14a74a26f434db08c652e097385c8 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Advancing to term 8
I20260809 15:47:18.955991 28383 consensus_queue.cc:1046] T 0a51c0baaa6e432997c9d3552af4f254 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.051s
I20260809 15:47:19.018677 23283 raft_consensus.cc:2454] T 99bdb93503db4de4a1cfb0161ea0c07c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 4.
W20260809 15:47:19.012948 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 44879.2us
I20260809 15:47:19.021131 23283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c48a3049897452383257aa757834fb5" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:18.967067 28679 raft_consensus.cc:686] T 763e58d6ba644eea830a6c1664d39c9b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 LEADER]: Becoming Leader. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Running, Role: LEADER
I20260809 15:47:19.008783 23396 raft_consensus.cc:2379] T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 61e69695258f48baa984c9d395e8ce3f in current term 10: Already voted for candidate c0f38c1908414566aa9115bb2cbdd85d in this term.
I20260809 15:47:18.959029 28419 raft_consensus.cc:504] T 4564dfc6d4094c1b9240b367ccecf3b1 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:47:19.028105 23404 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52856 (request call id 418) took 3913 ms (client timeout 1961 ms). Trace:
W20260809 15:47:19.028493 23404 rpcz_store.cc:269] 0809 15:47:15.115078 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:15.115191 (+ 113us) service_pool.cc:227] Handling call
0809 15:47:19.027912 (+3912721us) spinlock_profiling.cc:243] Waited 1.23 s on lock 0x7b54000023d4. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e92242ae 00007f34ee161c27 00007f34e7849c04 00007f34e7849b92 00007f34e7849ac1 00007f34e7849a3c 00007f34e7848cf2 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:47:19.028035 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2827749632}
I20260809 15:47:19.011049 28631 raft_consensus.cc:2781] T 213543cdbe814c89a17a963469922c1b P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260809 15:47:19.029914 28631 raft_consensus.cc:482] T 213543cdbe814c89a17a963469922c1b P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:47:19.031553 23282 raft_consensus.cc:2379] T 953eea268e17491ca76f3a555860a512 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c in current term 9: Already voted for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in this term.
I20260809 15:47:19.031960 23283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a88421dba80d4d3388aa5eb62e23853b" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:19.014034 23351 leader_election.cc:302] T d3be1e356b254d22b995be32f1c12f6f P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 61e69695258f48baa984c9d395e8ce3f, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:47:18.955606 28783 consensus_queue.cc:1046] T 0da4d5283c8a46878bd39b85e5c71f66 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.069s
I20260809 15:47:19.018575 28782 raft_consensus.cc:3037] T 2db057fd10e149a9895dff6ef8c9f038 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Advancing to term 8
I20260809 15:47:19.026863 23519 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56770 (request call id 1491) took 1498 ms. Trace:
I20260809 15:47:19.034335 23351 leader_election.cc:302] T 4564dfc6d4094c1b9240b367ccecf3b1 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:47:19.013556 28725 raft_consensus.cc:3037] T 953eea268e17491ca76f3a555860a512 P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Advancing to term 9
I20260809 15:47:19.035239 23283 raft_consensus.cc:2379] T a88421dba80d4d3388aa5eb62e23853b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c in current term 7: Already voted for candidate c0f38c1908414566aa9115bb2cbdd85d in this term.
I20260809 15:47:19.036142 23519 rpcz_store.cc:276] 0809 15:47:17.528688 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:17.759511 (+230823us) service_pool.cc:227] Handling call
0809 15:47:17.779646 (+ 20135us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 15:47:18.055981 (+276335us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:47:19.026624 (+970643us) spinlock_profiling.cc:243] Waited 193 ms on lock 0x7b540038de54. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e9220f51 00007f34e921f99e 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772 00007f34e764a94c 00007f34e7649d86
0809 15:47:19.026808 (+ 184us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":443452672}
I20260809 15:47:19.014751 28319 raft_consensus.cc:482] T 0c48a3049897452383257aa757834fb5 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
I20260809 15:47:19.000056 28787 raft_consensus.cc:504] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:19.012327 28802 consensus_queue.cc:1046] T 907bacfa79d04a55a68c8dbe60056462 P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 6, Time since last communication: 0.069s
I20260809 15:47:19.568089 23280 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55434 (request call id 322) took 2407 ms. Trace:
I20260809 15:47:19.568369 23276 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53554 (request call id 1840) took 4613 ms. Trace:
I20260809 15:47:19.550050 23228 leader_election.cc:302] T 07fcabb569414bddbd39df6cedd24e64 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters:
W20260809 15:47:19.555485 23227 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.630s user 0.002s sys 0.000s
W20260809 15:47:19.595609 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 561179us
W20260809 15:47:19.620533 23227 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.695s user 0.010s sys 0.002s
I20260809 15:47:19.621484 23351 leader_election.cc:302] T 6ced718c1a5c4da79cb8c1ebd1435b87 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:47:19.605126 23519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "763e58d6ba644eea830a6c1664d39c9b" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:19.626420 23519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "763e58d6ba644eea830a6c1664d39c9b" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: true dest_uuid: "65c36d164b0540f099d592f38307ee6c"
W20260809 15:47:19.615481 23472 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.554s user 0.003s sys 0.003s
W20260809 15:47:19.627077 23472 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.566s user 0.003s sys 0.003s
W20260809 15:47:19.627442 23472 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 594382us
I20260809 15:47:19.598575 23280 rpcz_store.cc:276] 0809 15:47:17.160135 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:17.280835 (+120700us) service_pool.cc:227] Handling call
0809 15:47:17.292120 (+ 11285us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 15:47:17.318963 (+ 26843us) raft_consensus.cc:1520] Early marking committed up to index 7
0809 15:47:18.751057 (+1432094us) spinlock_profiling.cc:243] Waited 1.49 ms on lock 0x7b18003d5a60. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34ef1978b1 00007f34ef1a4661 00007f34e626f4ca 00007f34ef950f99 00007f34e6133990 00007f34e62f5864 00007f34e62f5155 00007f34ecbf5167 00007f34e922a807 00007f34e9239f0f 00007f34e9239e88
0809 15:47:18.878961 (+127904us) log.cc:842] Serialized 14 byte log entry
0809 15:47:18.999022 (+120061us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 15:47:18.999506 (+ 484us) log.cc:842] Serialized 26 byte log entry
0809 15:47:19.013394 (+ 13888us) raft_consensus.cc:1631] Marking committed up to 7
0809 15:47:19.013450 (+ 56us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:47:19.013516 (+ 66us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 15:47:19.567057 (+553541us) raft_consensus.cc:1671] finished
0809 15:47:19.567444 (+ 387us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:47:19.567831 (+ 387us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":82911,"prepare.queue_time_us":1433183,"prepare.run_cpu_time_us":31,"prepare.run_wall_time_us":32,"raft.queue_time_us":34130,"raft.run_cpu_time_us":270,"raft.run_wall_time_us":13216,"spinlock_wait_cycles":3425152,"thread_start_us":205823,"threads_started":2,"wal-append.queue_time_us":142995}
I20260809 15:47:19.126180 28762 raft_consensus.cc:482] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 61e69695258f48baa984c9d395e8ce3f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:19.673323 28762 raft_consensus.cc:504] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 61e69695258f48baa984c9d395e8ce3f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:19.339731 27892 raft_consensus.cc:504] T 0c29937a80724ddb9d07a27534a146f3 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:19.340071 28679 consensus_queue.cc:237] T 763e58d6ba644eea830a6c1664d39c9b P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:19.258836 28782 raft_consensus.cc:504] T 2db057fd10e149a9895dff6ef8c9f038 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:19.037061 28811 raft_consensus.cc:2726] T d3be1e356b254d22b995be32f1c12f6f P c0f38c1908414566aa9115bb2cbdd85d [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260809 15:47:19.450393 28274 raft_consensus.cc:504] T 2424fd9bd7f14e508fd106cbfda786a4 P 65c36d164b0540f099d592f38307ee6c [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:19.018294 28784 raft_consensus.cc:2781] T ae6d2c08a7a94f58ac8dd698e3fd456c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260809 15:47:19.346627 28580 raft_consensus.cc:504] T f9e7cadcd24d45fcbfa84209225e8f2d P 65c36d164b0540f099d592f38307ee6c [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:19.683988 23280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5c2e425a74b4e64ae70b5def750434c" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:47:19.249346 27493 leader_election.cc:288] T 7660dfa2ef694927b8eda4100a2bcd9b P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:19.286053 28779 raft_consensus.cc:686] T 2424fd9bd7f14e508fd106cbfda786a4 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 LEADER]: Becoming Leader. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Running, Role: LEADER
I20260809 15:47:19.689918 28508 raft_consensus.cc:2726] T 4564dfc6d4094c1b9240b367ccecf3b1 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260809 15:47:19.692505 28788 raft_consensus.cc:3037] T 6ced718c1a5c4da79cb8c1ebd1435b87 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Advancing to term 5
I20260809 15:47:19.703961 28550 raft_consensus.cc:3032] T 0da4d5283c8a46878bd39b85e5c71f66 P 65c36d164b0540f099d592f38307ee6c [term 5 LEADER]: Stepping down as leader of term 5
I20260809 15:47:19.347357 28785 raft_consensus.cc:2726] T 6ae14a74a26f434db08c652e097385c8 P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260809 15:47:19.704488 28550 raft_consensus.cc:729] T 0da4d5283c8a46878bd39b85e5c71f66 P 65c36d164b0540f099d592f38307ee6c [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:47:19.706105 28550 consensus_queue.cc:260] T 0da4d5283c8a46878bd39b85e5c71f66 P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:19.708248 24451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7660dfa2ef694927b8eda4100a2bcd9b" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:47:19.709246 24451 raft_consensus.cc:2454] T 7660dfa2ef694927b8eda4100a2bcd9b P 61e69695258f48baa984c9d395e8ce3f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 6.
I20260809 15:47:19.087299 28564 consensus_queue.cc:1035] T ae6d2c08a7a94f58ac8dd698e3fd456c P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.623s
I20260809 15:47:19.676141 28685 consensus_queue.cc:1046] T 538a228ddfb54bc6a36528d5ff07597b P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.035s
I20260809 15:47:19.711318 28410 raft_consensus.cc:3032] T 538a228ddfb54bc6a36528d5ff07597b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 LEADER]: Stepping down as leader of term 7
I20260809 15:47:19.128402 28255 raft_consensus.cc:2726] T 0a51c0baaa6e432997c9d3552af4f254 P 61e69695258f48baa984c9d395e8ce3f [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260809 15:47:19.712283 28550 raft_consensus.cc:3037] T 0da4d5283c8a46878bd39b85e5c71f66 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Advancing to term 6
I20260809 15:47:19.033258 28631 raft_consensus.cc:3037] T 213543cdbe814c89a17a963469922c1b P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Advancing to term 5
I20260809 15:47:19.713160 28669 leader_election.cc:288] T a88421dba80d4d3388aa5eb62e23853b P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:47:19.678577 27892 leader_election.cc:288] T 0c29937a80724ddb9d07a27534a146f3 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 9 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
W20260809 15:47:19.717119 28825 consensus_queue.cc:1159] T 0da4d5283c8a46878bd39b85e5c71f66 P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "c0f38c1908414566aa9115bb2cbdd85d" responder_term: 6 status { last_received { term: 5 index: 3 } last_committed_idx: 3 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 5. Current term is 6. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260809 15:47:19.129768 28774 raft_consensus.cc:504] T ba56725e42f74a1baa4c04d9660727f9 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } attrs { replace: true } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } }
W20260809 15:47:19.719476 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 98304.3us
I20260809 15:47:19.129822 28598 raft_consensus.cc:2726] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260809 15:47:19.711786 28410 raft_consensus.cc:729] T 538a228ddfb54bc6a36528d5ff07597b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Running, Role: LEADER
I20260809 15:47:19.721242 28404 consensus_queue.cc:1046] T 763e58d6ba644eea830a6c1664d39c9b P 65c36d164b0540f099d592f38307ee6c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.002s
W20260809 15:47:19.724189 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 174636us
I20260809 15:47:19.019821 28577 tablet_copy_client.cc:316] T ba56725e42f74a1baa4c04d9660727f9 P 61e69695258f48baa984c9d395e8ce3f: tablet copy: Beginning tablet copy session from remote peer at address 127.20.202.67:40553
I20260809 15:47:19.081185 28810 consensus_queue.cc:1046] T 6ae14a74a26f434db08c652e097385c8 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.645s
I20260809 15:47:19.732652 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3296ef50be254fc29cfc77d22a8f7a23" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:47:19.727442 23280 raft_consensus.cc:2379] T c5c2e425a74b4e64ae70b5def750434c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c in current term 7: Already voted for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in this term.
I20260809 15:47:19.723124 23519 raft_consensus.cc:3032] T 763e58d6ba644eea830a6c1664d39c9b P 65c36d164b0540f099d592f38307ee6c [term 7 LEADER]: Stepping down as leader of term 7
I20260809 15:47:19.727276 28685 raft_consensus.cc:2781] T 07fcabb569414bddbd39df6cedd24e64 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Leader election won for term 8
W20260809 15:47:19.733655 24468 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38724 (request call id 584) took 1945 ms (client timeout 1529 ms). Trace:
I20260809 15:47:19.728595 28784 raft_consensus.cc:482] T ae6d2c08a7a94f58ac8dd698e3fd456c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:47:19.080897 28326 leader_election.cc:288] T 953eea268e17491ca76f3a555860a512 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 9 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:19.743257 28513 leader_election.cc:288] T 4b30f28311964a0b9149a50495e83d8f P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 6 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:19.729070 28274 leader_election.cc:288] T 2424fd9bd7f14e508fd106cbfda786a4 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 9 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:47:19.730322 28802 raft_consensus.cc:3032] T 907bacfa79d04a55a68c8dbe60056462 P 61e69695258f48baa984c9d395e8ce3f [term 8 LEADER]: Stepping down as leader of term 8
I20260809 15:47:19.731846 28779 consensus_queue.cc:237] T 2424fd9bd7f14e508fd106cbfda786a4 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.3, Last appended by leader: 3, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:47:19.081723 28411 consensus_meta.cc:287] T 2db057fd10e149a9895dff6ef8c9f038 P 65c36d164b0540f099d592f38307ee6c: Time spent flushing consensus metadata: real 1.296s user 0.005s sys 0.005s
I20260809 15:47:19.728018 28803 raft_consensus.cc:482] T 472816994d814cb198b69e5248697440 P 61e69695258f48baa984c9d395e8ce3f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:47:19.602296 23276 rpcz_store.cc:276] 0809 15:47:14.954518 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:14.954746 (+ 228us) service_pool.cc:227] Handling call
0809 15:47:14.962326 (+ 7580us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 15:47:15.003033 (+ 40707us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:47:15.003062 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 15:47:15.003479 (+ 417us) log.cc:842] Serialized 26 byte log entry
0809 15:47:15.123140 (+119661us) raft_consensus.cc:1631] Marking committed up to 3
0809 15:47:15.123174 (+ 34us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:47:15.123237 (+ 63us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 15:47:18.921285 (+3798048us) spinlock_profiling.cc:243] Waited 13.5 ms on lock 0x7b44000232cc. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e613cf11 00007f34e620aa95 00007f34e620b4b2 00007f34e8eb3511 00007f34e8eaee1a 00007f34e8ea2bee 00007f34e8ea1f36 00007f34e8ea1b58 00007f34e8eab551 00007f34e8eab509 00007f34e8eab499
0809 15:47:19.567580 (+646295us) raft_consensus.cc:1671] finished
0809 15:47:19.567630 (+ 50us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:47:19.568155 (+ 525us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":154330,"raft.run_cpu_time_us":197,"raft.run_wall_time_us":18861,"spinlock_wait_cycles":31100672,"thread_start_us":65680,"threads_started":2,"wal-append.queue_time_us":208302}
I20260809 15:47:19.740214 28598 raft_consensus.cc:482] T d1a134ba33ea4076bbaac0ae460c17ba P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:19.740651 23345 leader_election.cc:302] T 7660dfa2ef694927b8eda4100a2bcd9b P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:47:19.742151 28580 leader_election.cc:288] T f9e7cadcd24d45fcbfa84209225e8f2d P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 9 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:47:19.746492 28802 raft_consensus.cc:729] T 907bacfa79d04a55a68c8dbe60056462 P 61e69695258f48baa984c9d395e8ce3f [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 61e69695258f48baa984c9d395e8ce3f, State: Running, Role: LEADER
I20260809 15:47:19.746604 28784 raft_consensus.cc:3037] T ae6d2c08a7a94f58ac8dd698e3fd456c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Advancing to term 9
I20260809 15:47:19.750206 28810 raft_consensus.cc:3032] T 6ae14a74a26f434db08c652e097385c8 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 LEADER]: Stepping down as leader of term 7
W20260809 15:47:19.739418 24468 rpcz_store.cc:269] 0809 15:47:17.787792 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:17.787935 (+ 143us) service_pool.cc:227] Handling call
0809 15:47:19.732254 (+1944319us) spinlock_profiling.cc:243] Waited 1.87 ms on lock 0x7b5800005498. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e613cf11 00007f34e62c0851 00007f34e62c0702 00007f34ef1b0b09 00007f34ee0ec730 00007f34ee1b279b 00007f34ee1b2821 00007f34ee1621b0 00007f34ee161a94 00007f34e7849c04 00007f34e7849b92
0809 15:47:19.733565 (+ 1311us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4305792}
I20260809 15:47:19.030277 28419 leader_election.cc:288] T 4564dfc6d4094c1b9240b367ccecf3b1 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 8 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:19.734725 23519 raft_consensus.cc:729] T 763e58d6ba644eea830a6c1664d39c9b P 65c36d164b0540f099d592f38307ee6c [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:47:19.754403 28598 raft_consensus.cc:504] T d1a134ba33ea4076bbaac0ae460c17ba P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:19.755127 28782 leader_election.cc:288] T 2db057fd10e149a9895dff6ef8c9f038 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 8 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:19.756290 28208 raft_consensus.cc:2781] T 7660dfa2ef694927b8eda4100a2bcd9b P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260809 15:47:19.755980 28803 raft_consensus.cc:504] T 472816994d814cb198b69e5248697440 P 61e69695258f48baa984c9d395e8ce3f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:19.752930 28685 raft_consensus.cc:686] T 07fcabb569414bddbd39df6cedd24e64 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 LEADER]: Becoming Leader. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Running, Role: LEADER
I20260809 15:47:19.733929 28762 leader_election.cc:288] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:19.758298 23280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d880de657e544468c82157769622a26" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:19.762234 24451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3296ef50be254fc29cfc77d22a8f7a23" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:47:19.764268 28802 consensus_queue.cc:260] T 907bacfa79d04a55a68c8dbe60056462 P 61e69695258f48baa984c9d395e8ce3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:19.782135 28774 leader_election.cc:288] T ba56725e42f74a1baa4c04d9660727f9 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 7 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:19.785436 23394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0721a4403a440f383318a3cea9d1442" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:19.788779 28598 leader_election.cc:288] T d1a134ba33ea4076bbaac0ae460c17ba P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:47:19.786319 23404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3622f6e3e3641cea1960e3a31498bb7" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:19.789469 28669 raft_consensus.cc:1013] T 4b30f28311964a0b9149a50495e83d8f P 65c36d164b0540f099d592f38307ee6c [term 5 LEADER]: attempt to promote peer 61e69695258f48baa984c9d395e8ce3f: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20260809 15:47:19.787667 23280 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53554 (request call id 1849) took 1990 ms. Trace:
I20260809 15:47:19.779492 23351 leader_election.cc:302] T a3622f6e3e3641cea1960e3a31498bb7 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:47:19.777567 28208 raft_consensus.cc:482] T 7660dfa2ef694927b8eda4100a2bcd9b P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Starting leader election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:47:19.787858 28410 consensus_queue.cc:260] T 538a228ddfb54bc6a36528d5ff07597b P 67b3aeadfa7a432abe8af4d292e04ab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:19.791013 28208 raft_consensus.cc:3037] T 7660dfa2ef694927b8eda4100a2bcd9b P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Advancing to term 7
I20260809 15:47:19.792254 28802 raft_consensus.cc:3037] T 907bacfa79d04a55a68c8dbe60056462 P 61e69695258f48baa984c9d395e8ce3f [term 8 FOLLOWER]: Advancing to term 9
I20260809 15:47:19.791363 28685 consensus_queue.cc:237] T 07fcabb569414bddbd39df6cedd24e64 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:19.790704 23280 rpcz_store.cc:276] 0809 15:47:17.797549 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:19.757927 (+1960378us) service_pool.cc:227] Handling call
0809 15:47:19.787604 (+ 29677us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:47:19.773205 28810 raft_consensus.cc:729] T 6ae14a74a26f434db08c652e097385c8 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Running, Role: LEADER
I20260809 15:47:19.791505 23404 raft_consensus.cc:2454] T a3622f6e3e3641cea1960e3a31498bb7 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 6.
I20260809 15:47:19.797279 23521 raft_consensus.cc:2454] T 2db057fd10e149a9895dff6ef8c9f038 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 7.
I20260809 15:47:19.798120 28449 raft_consensus.cc:482] T 2db057fd10e149a9895dff6ef8c9f038 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:19.798637 28449 raft_consensus.cc:504] T 2db057fd10e149a9895dff6ef8c9f038 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:19.798707 28810 consensus_queue.cc:260] T 6ae14a74a26f434db08c652e097385c8 P 67b3aeadfa7a432abe8af4d292e04ab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:19.804386 28785 raft_consensus.cc:482] T 28e17dcb5efe45b8b6a8c42da03b4819 P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:47:19.804620 28293 consensus_queue.cc:1046] T ba56725e42f74a1baa4c04d9660727f9 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } attrs { replace: true }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20260809 15:47:19.805186 28785 raft_consensus.cc:504] T 28e17dcb5efe45b8b6a8c42da03b4819 P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:19.805967 28293 raft_consensus.cc:482] T cee02fff2c9c48a19f43eb358a89e7b0 P 65c36d164b0540f099d592f38307ee6c [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:19.806833 28803 leader_election.cc:288] T 472816994d814cb198b69e5248697440 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:19.806557 28293 raft_consensus.cc:504] T cee02fff2c9c48a19f43eb358a89e7b0 P 65c36d164b0540f099d592f38307ee6c [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:19.807399 28284 raft_consensus.cc:482] T 294d4bcc1e7a477cb56455581277a60f P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
I20260809 15:47:19.811203 23280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2d4f749fcfd4fb3bfb8115af3471598" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:19.808030 28598 consensus_queue.cc:1046] T 4b30f28311964a0b9149a50495e83d8f P 65c36d164b0540f099d592f38307ee6c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: INVALID_TERM, Last received: 3.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260809 15:47:19.020921 28691 raft_consensus.cc:504] T 213543cdbe814c89a17a963469922c1b P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:19.810978 28284 raft_consensus.cc:504] T 294d4bcc1e7a477cb56455581277a60f P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:19.797207 23399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad8a3b25d7b49e5a2142d5709edd0c7" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
W20260809 15:47:19.799324 23353 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 114306us
I20260809 15:47:19.822153 28208 raft_consensus.cc:504] T 7660dfa2ef694927b8eda4100a2bcd9b P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:19.823611 23394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00a1881267a14ba8b78c81040f17a014" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:19.811558 28410 raft_consensus.cc:3037] T 538a228ddfb54bc6a36528d5ff07597b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Advancing to term 8
I20260809 15:47:19.805197 28810 raft_consensus.cc:3037] T 6ae14a74a26f434db08c652e097385c8 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Advancing to term 8
W20260809 15:47:19.825323 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 72650us
I20260809 15:47:19.832475 28248 raft_consensus.cc:2781] T a3622f6e3e3641cea1960e3a31498bb7 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260809 15:47:19.832924 28248 raft_consensus.cc:482] T a3622f6e3e3641cea1960e3a31498bb7 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:47:19.833410 28229 raft_consensus.cc:3032] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P c0f38c1908414566aa9115bb2cbdd85d [term 5 LEADER]: Stepping down as leader of term 5
I20260809 15:47:19.833838 28229 raft_consensus.cc:729] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P c0f38c1908414566aa9115bb2cbdd85d [term 5 LEADER]: Becoming Follower/Learner. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:47:19.833926 28284 leader_election.cc:288] T 294d4bcc1e7a477cb56455581277a60f P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:19.835029 23396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472816994d814cb198b69e5248697440" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:19.833316 28248 raft_consensus.cc:3037] T a3622f6e3e3641cea1960e3a31498bb7 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Advancing to term 7
W20260809 15:47:19.836616 28768 consensus_meta.cc:287] T f7715d71dbac447b9351fc5eb36c5e80 P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent flushing consensus metadata: real 0.981s user 0.021s sys 0.018s
I20260809 15:47:19.839808 28319 raft_consensus.cc:504] T 0c48a3049897452383257aa757834fb5 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:19.840313 28768 raft_consensus.cc:504] T f7715d71dbac447b9351fc5eb36c5e80 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:19.841850 28208 leader_election.cc:288] T 7660dfa2ef694927b8eda4100a2bcd9b P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 7 election: Requested vote from peers 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:19.842293 28768 leader_election.cc:288] T f7715d71dbac447b9351fc5eb36c5e80 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 7 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:19.830456 23283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2424fd9bd7f14e508fd106cbfda786a4" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 9 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:19.838053 28691 leader_election.cc:288] T 213543cdbe814c89a17a963469922c1b P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:19.838964 28676 raft_consensus.cc:3032] T 0a51c0baaa6e432997c9d3552af4f254 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 LEADER]: Stepping down as leader of term 7
I20260809 15:47:19.839250 23399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a823d35f4f348d1b52ba9e6d80b00d2" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:19.085628 28587 raft_consensus.cc:504] T 30c1740ef6874df0a332fbd6fd4b03ce P 65c36d164b0540f099d592f38307ee6c [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:19.846710 23394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7baa2492df0340c4a5ac241959ce1ac3" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:19.847189 23397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472816994d814cb198b69e5248697440" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:19.847452 23280 raft_consensus.cc:2454] T d2d4f749fcfd4fb3bfb8115af3471598 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 8.
I20260809 15:47:19.848315 23276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16ca412a0dae4feebe85dd01bf1a9173" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:19.845337 28676 raft_consensus.cc:729] T 0a51c0baaa6e432997c9d3552af4f254 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Running, Role: LEADER
I20260809 15:47:19.828629 28785 leader_election.cc:288] T 28e17dcb5efe45b8b6a8c42da03b4819 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:47:19.850678 23393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73f6f44c259547f1b05a3a434616fa54" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:19.857075 23283 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53554 (request call id 1859) took 1209 ms. Trace:
I20260809 15:47:19.845285 23519 consensus_queue.cc:260] T 763e58d6ba644eea830a6c1664d39c9b P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:19.859850 23283 rpcz_store.cc:276] 0809 15:47:18.643722 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:19.663789 (+1020067us) service_pool.cc:227] Handling call
0809 15:47:19.853003 (+189214us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:47:19.847209 28293 leader_election.cc:288] T cee02fff2c9c48a19f43eb358a89e7b0 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:19.867381 24406 leader_election.cc:302] T c0741ab44d99439abc3aa3bdd27ee160 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:47:19.868323 28803 raft_consensus.cc:2726] T c0741ab44d99439abc3aa3bdd27ee160 P 61e69695258f48baa984c9d395e8ce3f [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260809 15:47:19.869134 24406 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 81942.5us
I20260809 15:47:19.834959 28229 consensus_queue.cc:260] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P c0f38c1908414566aa9115bb2cbdd85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:19.877231 28788 raft_consensus.cc:2726] T 6ced718c1a5c4da79cb8c1ebd1435b87 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260809 15:47:19.878460 23351 leader_election.cc:302] T f9e7cadcd24d45fcbfa84209225e8f2d P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:47:19.879488 23393 raft_consensus.cc:2379] T 73f6f44c259547f1b05a3a434616fa54 P c0f38c1908414566aa9115bb2cbdd85d [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c in current term 9: Already voted for candidate c0f38c1908414566aa9115bb2cbdd85d in this term.
I20260809 15:47:19.850358 28676 consensus_queue.cc:260] T 0a51c0baaa6e432997c9d3552af4f254 P 67b3aeadfa7a432abe8af4d292e04ab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:19.877260 23285 raft_consensus.cc:2379] T f7715d71dbac447b9351fc5eb36c5e80 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d in current term 7: Already voted for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in this term.
I20260809 15:47:19.892119 28787 leader_election.cc:288] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:19.894186 28248 raft_consensus.cc:504] T a3622f6e3e3641cea1960e3a31498bb7 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:19.896018 23276 raft_consensus.cc:2396] T 16ca412a0dae4feebe85dd01bf1a9173 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c for term 8 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
I20260809 15:47:19.902912 23283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998c6cee72cf40228c6dba064c0db2da" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:19.907727 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7660dfa2ef694927b8eda4100a2bcd9b" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f"
I20260809 15:47:19.909822 24453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f9d6565aa4c45a3b3f01210d0eb71f7" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:47:19.909247 23404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "563daf42d7784db195ea99c3cb21b370" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:19.909512 23393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2d4f749fcfd4fb3bfb8115af3471598" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:19.917320 23472 leader_election.cc:302] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:19.909958 23276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cef3e7277d7d4760824a3436ed4a847d" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:19.918277 28293 raft_consensus.cc:2726] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 65c36d164b0540f099d592f38307ee6c [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260809 15:47:19.906085 23280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e14d118b2eaa42aa8a7cd4025aba7128" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
W20260809 15:47:19.911973 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 67319.7us
I20260809 15:47:19.919919 23519 raft_consensus.cc:3037] T 763e58d6ba644eea830a6c1664d39c9b P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Advancing to term 9
I20260809 15:47:19.911701 23283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3296ef50be254fc29cfc77d22a8f7a23" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
W20260809 15:47:19.894660 23228 connection.cc:232] RPC call timeout handler was delayed by -1.8590553388930857s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1654s for a response
I20260809 15:47:19.920368 23396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e17dcb5efe45b8b6a8c42da03b4819" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:19.920588 23393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998c6cee72cf40228c6dba064c0db2da" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:19.918282 23276 raft_consensus.cc:2454] T cef3e7277d7d4760824a3436ed4a847d P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 8.
I20260809 15:47:19.923448 23276 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53554 (request call id 1873) took 1006 ms. Trace:
I20260809 15:47:19.923799 23276 rpcz_store.cc:276] 0809 15:47:18.917170 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:19.902958 (+985788us) service_pool.cc:227] Handling call
0809 15:47:19.923399 (+ 20441us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:47:19.925118 23472 leader_election.cc:302] T 472816994d814cb198b69e5248697440 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:19.866918 28725 raft_consensus.cc:504] T 953eea268e17491ca76f3a555860a512 P 65c36d164b0540f099d592f38307ee6c [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:19.906152 28787 raft_consensus.cc:2726] T f9e7cadcd24d45fcbfa84209225e8f2d P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260809 15:47:19.929757 23397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3296ef50be254fc29cfc77d22a8f7a23" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:19.920521 28587 leader_election.cc:288] T 30c1740ef6874df0a332fbd6fd4b03ce P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 9 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
W20260809 15:47:19.926627 23228 connection.cc:232] RPC call timeout handler was delayed by -1.8428553389385343s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16720000000000002s for a response
W20260809 15:47:19.963760 23228 connection.cc:232] RPC call timeout handler was delayed by -1.8293553388211876s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16870000000000002s for a response
I20260809 15:47:19.930745 23280 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53554 (request call id 1876) took 1012 ms. Trace:
I20260809 15:47:19.933656 23404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0da4d5283c8a46878bd39b85e5c71f66" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:19.948076 23396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2d4f749fcfd4fb3bfb8115af3471598" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:19.949266 23399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4564dfc6d4094c1b9240b367ccecf3b1" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:47:19.951483 23283 raft_consensus.cc:2454] T 3296ef50be254fc29cfc77d22a8f7a23 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 2.
I20260809 15:47:19.965540 23280 rpcz_store.cc:276] 0809 15:47:18.918048 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:19.852593 (+934545us) service_pool.cc:227] Handling call
0809 15:47:19.930686 (+ 78093us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:47:19.953179 23351 leader_election.cc:302] T 73f6f44c259547f1b05a3a434616fa54 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters:
W20260809 15:47:19.953998 23270 consensus_meta.cc:287] T d0721a4403a440f383318a3cea9d1442 P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent flushing consensus metadata: real 1.117s user 0.011s sys 0.011s
I20260809 15:47:19.955176 23394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "953eea268e17491ca76f3a555860a512" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:47:19.956831 28631 raft_consensus.cc:504] T 213543cdbe814c89a17a963469922c1b P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:47:19.955050 28831 log.cc:924] Time spent T 538a228ddfb54bc6a36528d5ff07597b P c0f38c1908414566aa9115bb2cbdd85d: Append to log took a long time: real 0.103s user 0.000s sys 0.000s
I20260809 15:47:19.966485 23394 raft_consensus.cc:3037] T 953eea268e17491ca76f3a555860a512 P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Advancing to term 9
I20260809 15:47:19.931064 28248 leader_election.cc:288] T a3622f6e3e3641cea1960e3a31498bb7 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 7 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
W20260809 15:47:19.967680 23228 connection.cc:232] RPC call timeout handler was delayed by -1.7195553388446569s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1809s for a response
W20260809 15:47:19.968549 23228 connection.cc:232] RPC call timeout handler was delayed by -1.6826553388964385s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18500000000000003s for a response
I20260809 15:47:19.968793 24453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "294d4bcc1e7a477cb56455581277a60f" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
W20260809 15:47:19.969101 23228 connection.cc:232] RPC call timeout handler was delayed by -1.5791553389281034s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1965s for a response
W20260809 15:47:19.969471 23228 connection.cc:232] RPC call timeout handler was delayed by -1.4765553388278931s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20790000000000003s for a response
I20260809 15:47:19.967778 28293 raft_consensus.cc:2726] T 472816994d814cb198b69e5248697440 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260809 15:47:19.972024 28779 raft_consensus.cc:482] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:19.972554 28779 raft_consensus.cc:504] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:47:19.974098 23228 connection.cc:232] RPC call timeout handler was delayed by -1.3685553388204426s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21989999999999998s for a response
I20260809 15:47:19.967888 23404 raft_consensus.cc:2454] T 0da4d5283c8a46878bd39b85e5c71f66 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 6.
I20260809 15:47:19.977571 23404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba56725e42f74a1baa4c04d9660727f9" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:47:19.977852 23399 raft_consensus.cc:3037] T 4564dfc6d4094c1b9240b367ccecf3b1 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Advancing to term 8
I20260809 15:47:19.968824 23396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cef3e7277d7d4760824a3436ed4a847d" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:19.981980 23404 raft_consensus.cc:3037] T ba56725e42f74a1baa4c04d9660727f9 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Advancing to term 7
I20260809 15:47:19.972604 28578 raft_consensus.cc:3032] T ba56725e42f74a1baa4c04d9660727f9 P 65c36d164b0540f099d592f38307ee6c [term 5 LEADER]: Stepping down as leader of term 5
I20260809 15:47:19.983764 28578 raft_consensus.cc:729] T ba56725e42f74a1baa4c04d9660727f9 P 65c36d164b0540f099d592f38307ee6c [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:47:19.984537 23546 tablet_copy_service.cc:140] P 65c36d164b0540f099d592f38307ee6c: Received BeginTabletCopySession request for tablet ba56725e42f74a1baa4c04d9660727f9 from peer 61e69695258f48baa984c9d395e8ce3f ({username='slave'} at 127.0.0.1:44018)
I20260809 15:47:19.984890 24453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee02fff2c9c48a19f43eb358a89e7b0" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 10 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:47:19.984997 23395 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55332 (request call id 1511) took 3172 ms. Trace:
I20260809 15:47:19.982786 28676 raft_consensus.cc:3037] T 0a51c0baaa6e432997c9d3552af4f254 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Advancing to term 8
I20260809 15:47:19.985800 24453 raft_consensus.cc:2396] T cee02fff2c9c48a19f43eb358a89e7b0 P 61e69695258f48baa984c9d395e8ce3f [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c for term 10 because replica has last-logged OpId of term: 8 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 8.
I20260809 15:47:19.985487 23395 rpcz_store.cc:276] 0809 15:47:16.812141 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:16.814670 (+ 2529us) service_pool.cc:227] Handling call
0809 15:47:16.814988 (+ 318us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:47:17.207536 (+392548us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:47:17.207566 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 15:47:17.207603 (+ 37us) raft_consensus.cc:1631] Marking committed up to 3
0809 15:47:17.207630 (+ 27us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:47:19.984575 (+2776945us) spinlock_profiling.cc:243] Waited 392 ms on lock 0x7b540001dbd4. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e9220f51 00007f34e921f99e 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772 00007f34e764a94c 00007f34e7649d86
0809 15:47:19.984639 (+ 64us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:47:19.984823 (+ 184us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":902368000}
I20260809 15:47:19.985625 28612 raft_consensus.cc:3032] T e71a05ba1d4d42bd95c3f85f9313399e P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 LEADER]: Stepping down as leader of term 6
I20260809 15:47:19.988235 28612 raft_consensus.cc:729] T e71a05ba1d4d42bd95c3f85f9313399e P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Running, Role: LEADER
I20260809 15:47:19.987975 23396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c29937a80724ddb9d07a27534a146f3" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 9 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:47:19.988984 23396 raft_consensus.cc:3037] T 0c29937a80724ddb9d07a27534a146f3 P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Advancing to term 9
I20260809 15:47:19.989524 23517 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56766 (request call id 2181) took 2158 ms. Trace:
I20260809 15:47:19.989853 23517 rpcz_store.cc:276] 0809 15:47:17.831459 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:17.856302 (+ 24843us) service_pool.cc:227] Handling call
0809 15:47:19.989150 (+2132848us) spinlock_profiling.cc:243] Waited 177 ms on lock 0x7b54000f82d4. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e92242ae 00007f34ee161c27 00007f34e7849c04 00007f34e7849b92 00007f34e7849ac1 00007f34e7849a3c 00007f34e7848cf2 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:47:19.989339 (+ 189us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":407471360}
W20260809 15:47:19.990626 23472 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 71881us
W20260809 15:47:19.992065 23274 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53562 (request call id 2005) took 2663 ms (client timeout 2608 ms). Trace:
I20260809 15:47:19.989384 28612 consensus_queue.cc:260] T e71a05ba1d4d42bd95c3f85f9313399e P 67b3aeadfa7a432abe8af4d292e04ab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
W20260809 15:47:19.992412 23274 rpcz_store.cc:269] 0809 15:47:17.328464 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:18.056585 (+728121us) service_pool.cc:227] Handling call
0809 15:47:19.991685 (+1935100us) spinlock_profiling.cc:243] Waited 460 ms on lock 0x7b540008aa54. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e92242ae 00007f34ee161c27 00007f34e7849c04 00007f34e7849b92 00007f34e7849ac1 00007f34e7849a3c 00007f34e7848cf2 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:47:19.991866 (+ 181us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1057198464}
I20260809 15:47:19.993178 28612 raft_consensus.cc:3037] T e71a05ba1d4d42bd95c3f85f9313399e P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Advancing to term 7
I20260809 15:47:19.993500 23274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e4ffcf83c234ae7910ddef3ec0b4bda" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:19.995129 23270 raft_consensus.cc:2454] T d0721a4403a440f383318a3cea9d1442 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 9.
I20260809 15:47:19.991995 23546 tablet_copy_service.cc:161] P 65c36d164b0540f099d592f38307ee6c: Beginning new tablet copy session on tablet ba56725e42f74a1baa4c04d9660727f9 from peer 61e69695258f48baa984c9d395e8ce3f at {username='slave'} at 127.0.0.1:44018: session id = 61e69695258f48baa984c9d395e8ce3f-ba56725e42f74a1baa4c04d9660727f9
W20260809 15:47:19.997043 23270 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53554 (request call id 1860) took 1330 ms (client timeout 1539 ms). Trace:
W20260809 15:47:19.997424 23270 rpcz_store.cc:269] 0809 15:47:18.666367 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:18.728807 (+ 62440us) service_pool.cc:227] Handling call
0809 15:47:19.996979 (+1268172us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:47:19.998545 28447 raft_consensus.cc:2781] T 73f6f44c259547f1b05a3a434616fa54 P c0f38c1908414566aa9115bb2cbdd85d [term 9 FOLLOWER]: Leader election won for term 9
I20260809 15:47:19.996953 23394 raft_consensus.cc:2454] T 953eea268e17491ca76f3a555860a512 P c0f38c1908414566aa9115bb2cbdd85d [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 9.
I20260809 15:47:19.996851 23518 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56766 (request call id 2194) took 1388 ms. Trace:
I20260809 15:47:19.999724 23518 rpcz_store.cc:276] 0809 15:47:18.607938 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:18.629361 (+ 21423us) service_pool.cc:227] Handling call
0809 15:47:19.995490 (+1366129us) spinlock_profiling.cc:243] Waited 943 ms on lock 0x7b54001086d4. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e92242ae 00007f34ee161c27 00007f34e7849c04 00007f34e7849b92 00007f34e7849ac1 00007f34e7849a3c 00007f34e7848cf2 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:47:19.996664 (+ 1174us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2169009024}
I20260809 15:47:20.000279 23394 raft_consensus.cc:1261] T c5c2e425a74b4e64ae70b5def750434c P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260809 15:47:20.000783 23518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e14d118b2eaa42aa8a7cd4025aba7128" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:20.001541 23394 raft_consensus.cc:1226] T 6ced718c1a5c4da79cb8c1ebd1435b87 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 4. Current term is 5. Ops: []
I20260809 15:47:19.998402 23270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5c2e425a74b4e64ae70b5def750434c" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:20.002482 23270 raft_consensus.cc:2454] T c5c2e425a74b4e64ae70b5def750434c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 7.
I20260809 15:47:20.003326 23283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "763e58d6ba644eea830a6c1664d39c9b" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:20.003904 23276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df20671a7b874dbb98c0ef28371c1308" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:20.004736 23276 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53554 (request call id 1854) took 1919 ms. Trace:
I20260809 15:47:20.009461 23276 rpcz_store.cc:276] 0809 15:47:18.085049 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:19.924450 (+1839401us) service_pool.cc:227] Handling call
0809 15:47:20.004682 (+ 80232us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:47:19.999068 28447 raft_consensus.cc:686] T 73f6f44c259547f1b05a3a434616fa54 P c0f38c1908414566aa9115bb2cbdd85d [term 9 LEADER]: Becoming Leader. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:47:19.991211 23517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a823d35f4f348d1b52ba9e6d80b00d2" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:20.008213 23283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ae14a74a26f434db08c652e097385c8" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:20.003461 23270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07fcabb569414bddbd39df6cedd24e64" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:20.004046 23274 raft_consensus.cc:2454] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 4.
I20260809 15:47:20.001551 23518 raft_consensus.cc:2454] T e14d118b2eaa42aa8a7cd4025aba7128 P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 8.
I20260809 15:47:20.011641 23396 raft_consensus.cc:2454] T 0c29937a80724ddb9d07a27534a146f3 P c0f38c1908414566aa9115bb2cbdd85d [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 9.
I20260809 15:47:20.012535 28416 raft_consensus.cc:482] T d3be1e356b254d22b995be32f1c12f6f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
I20260809 15:47:20.012805 28447 consensus_queue.cc:237] T 73f6f44c259547f1b05a3a434616fa54 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:20.013118 23397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2db057fd10e149a9895dff6ef8c9f038" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:47:19.984975 28578 consensus_queue.cc:260] T ba56725e42f74a1baa4c04d9660727f9 P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } attrs { replace: true } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } }
I20260809 15:47:20.013346 23517 raft_consensus.cc:2454] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 65c36d164b0540f099d592f38307ee6c [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 9.
I20260809 15:47:20.026500 28416 raft_consensus.cc:504] T d3be1e356b254d22b995be32f1c12f6f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:20.027750 23517 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56766 (request call id 2200) took 1188 ms. Trace:
I20260809 15:47:20.028204 23517 rpcz_store.cc:276] 0809 15:47:18.838788 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:19.990890 (+1152102us) service_pool.cc:227] Handling call
0809 15:47:20.027629 (+ 36739us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:47:20.029422 23517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5712f6b6fffc4cc5a8ddfbe63e4097df" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:20.030321 23517 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44018 (request call id 354) took 1024 ms. Trace:
I20260809 15:47:20.013046 23518 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44018 (request call id 357) took 1004 ms. Trace:
I20260809 15:47:20.030948 23518 rpcz_store.cc:276] 0809 15:47:19.007091 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:20.000461 (+993370us) service_pool.cc:227] Handling call
0809 15:47:20.011349 (+ 10888us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:47:20.029623 23280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae6d2c08a7a94f58ac8dd698e3fd456c" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:20.013949 23274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c29937a80724ddb9d07a27534a146f3" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 9 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:20.017074 23399 raft_consensus.cc:2454] T 4564dfc6d4094c1b9240b367ccecf3b1 P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 8.
W20260809 15:47:20.012521 23228 connection.cc:232] RPC call timeout handler was delayed by -1.0420401338487864s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1683s for a response
I20260809 15:47:20.013002 28779 leader_election.cc:288] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:20.030519 23397 raft_consensus.cc:3037] T 2db057fd10e149a9895dff6ef8c9f038 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Advancing to term 8
I20260809 15:47:20.029624 28416 leader_election.cc:288] T d3be1e356b254d22b995be32f1c12f6f P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:20.043174 23518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3924b648461e463b9d7925e5671d68ae" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:47:20.043941 23518 raft_consensus.cc:2362] T 3924b648461e463b9d7925e5671d68ae P 65c36d164b0540f099d592f38307ee6c [term 10 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 10. Re-sending same reply.
I20260809 15:47:20.039814 23404 raft_consensus.cc:2454] T ba56725e42f74a1baa4c04d9660727f9 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 7.
W20260809 15:47:20.032912 23228 connection.cc:232] RPC call timeout handler was delayed by -1.0420401338487864s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1683s for a response
I20260809 15:47:20.045269 28631 leader_election.cc:288] T 213543cdbe814c89a17a963469922c1b P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 5 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:47:20.046829 23228 leader_election.cc:302] T 294d4bcc1e7a477cb56455581277a60f P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c
I20260809 15:47:20.034242 23395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9e7cadcd24d45fcbfa84209225e8f2d" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:47:20.047598 23399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1a134ba33ea4076bbaac0ae460c17ba" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:20.047848 23395 raft_consensus.cc:3037] T f9e7cadcd24d45fcbfa84209225e8f2d P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Advancing to term 9
I20260809 15:47:20.048317 23399 raft_consensus.cc:2454] T d1a134ba33ea4076bbaac0ae460c17ba P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 7.
I20260809 15:47:20.034704 28578 raft_consensus.cc:3037] T ba56725e42f74a1baa4c04d9660727f9 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Advancing to term 6
I20260809 15:47:19.999751 28725 leader_election.cc:288] T 953eea268e17491ca76f3a555860a512 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 9 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:47:20.035591 24410 leader_election.cc:302] T e14d118b2eaa42aa8a7cd4025aba7128 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c; no voters:
I20260809 15:47:20.036885 23283 raft_consensus.cc:2454] T 6ae14a74a26f434db08c652e097385c8 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 8.
I20260809 15:47:20.037653 24453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3be1e356b254d22b995be32f1c12f6f" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:47:20.041442 23353 leader_election.cc:302] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters: 61e69695258f48baa984c9d395e8ce3f
I20260809 15:47:20.042253 23397 raft_consensus.cc:2454] T 2db057fd10e149a9895dff6ef8c9f038 P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 8.
I20260809 15:47:20.044656 23274 raft_consensus.cc:2379] T 0c29937a80724ddb9d07a27534a146f3 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d in current term 9: Already voted for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in this term.
W20260809 15:47:20.043579 23285 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53562 (request call id 2004) took 2725 ms (client timeout 3226 ms). Trace:
I20260809 15:47:20.030696 23517 rpcz_store.cc:276] 0809 15:47:19.006281 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:20.029032 (+1022751us) service_pool.cc:227] Handling call
0809 15:47:20.030253 (+ 1221us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:47:20.049973 23518 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44018 (request call id 355) took 1043 ms. Trace:
I20260809 15:47:20.052556 28522 leader_election.cc:288] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:20.055806 23518 rpcz_store.cc:276] 0809 15:47:19.006571 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:20.042777 (+1036206us) service_pool.cc:227] Handling call
0809 15:47:20.049920 (+ 7143us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:47:20.057051 28416 raft_consensus.cc:482] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
I20260809 15:47:20.057379 28450 raft_consensus.cc:2781] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260809 15:47:20.059247 23274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba56725e42f74a1baa4c04d9660727f9" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:20.062546 23228 leader_election.cc:302] T a88421dba80d4d3388aa5eb62e23853b P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters: c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:20.063382 23517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bcac4b24e7a4448abdc934bba5a624e" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:20.064227 28758 raft_consensus.cc:2781] T e14d118b2eaa42aa8a7cd4025aba7128 P 61e69695258f48baa984c9d395e8ce3f [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260809 15:47:20.063066 23276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0da4d5283c8a46878bd39b85e5c71f66" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:20.065152 24410 leader_election.cc:302] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:47:20.070775 23404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e17dcb5efe45b8b6a8c42da03b4819" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
W20260809 15:47:20.065639 23285 rpcz_store.cc:269] 0809 15:47:17.318472 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:18.824812 (+1506340us) service_pool.cc:227] Handling call
0809 15:47:20.043359 (+1218547us) spinlock_profiling.cc:243] Waited 1.01 s on lock 0x7b54002f44d4. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e92242ae 00007f34ee161c27 00007f34e7849c04 00007f34e7849b92 00007f34e7849ac1 00007f34e7849a3c 00007f34e7848cf2 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:47:20.043527 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2331034368}
I20260809 15:47:20.072789 23395 raft_consensus.cc:2454] T f9e7cadcd24d45fcbfa84209225e8f2d P c0f38c1908414566aa9115bb2cbdd85d [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 9.
I20260809 15:47:20.069775 28450 raft_consensus.cc:482] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d [term 9 FOLLOWER]: Starting leader election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
I20260809 15:47:20.069947 28758 raft_consensus.cc:482] T e14d118b2eaa42aa8a7cd4025aba7128 P 61e69695258f48baa984c9d395e8ce3f [term 8 FOLLOWER]: Starting leader election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
I20260809 15:47:20.076668 28450 raft_consensus.cc:3037] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d [term 9 FOLLOWER]: Advancing to term 10
I20260809 15:47:20.076789 28758 raft_consensus.cc:3037] T e14d118b2eaa42aa8a7cd4025aba7128 P 61e69695258f48baa984c9d395e8ce3f [term 8 FOLLOWER]: Advancing to term 9
I20260809 15:47:20.079706 23274 raft_consensus.cc:2379] T ba56725e42f74a1baa4c04d9660727f9 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d in current term 7: Already voted for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in this term.
I20260809 15:47:20.078537 28779 raft_consensus.cc:2741] T a88421dba80d4d3388aa5eb62e23853b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
I20260809 15:47:20.081725 23274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "907bacfa79d04a55a68c8dbe60056462" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:20.001601 28784 raft_consensus.cc:504] T ae6d2c08a7a94f58ac8dd698e3fd456c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:20.087325 23517 raft_consensus.cc:2454] T 4bcac4b24e7a4448abdc934bba5a624e P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 4.
I20260809 15:47:20.088160 23517 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56766 (request call id 2196) took 1253 ms. Trace:
I20260809 15:47:20.088506 23517 rpcz_store.cc:276] 0809 15:47:18.834467 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:20.063014 (+1228547us) service_pool.cc:227] Handling call
0809 15:47:20.088108 (+ 25094us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:47:20.090579 23404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30c1740ef6874df0a332fbd6fd4b03ce" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:47:20.091642 28209 raft_consensus.cc:2726] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260809 15:47:20.092344 23517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07fcabb569414bddbd39df6cedd24e64" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:20.091181 23404 raft_consensus.cc:3037] T 30c1740ef6874df0a332fbd6fd4b03ce P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Advancing to term 9
I20260809 15:47:20.093667 23270 raft_consensus.cc:1226] T 07fcabb569414bddbd39df6cedd24e64 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 LEADER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 7. Current term is 8. Ops: []
I20260809 15:47:20.089188 23276 raft_consensus.cc:2454] T 0da4d5283c8a46878bd39b85e5c71f66 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 6.
I20260809 15:47:20.095041 23353 leader_election.cc:302] T 4bcac4b24e7a4448abdc934bba5a624e P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters: 61e69695258f48baa984c9d395e8ce3f
I20260809 15:47:20.096560 23518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df20671a7b874dbb98c0ef28371c1308" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:20.079373 28810 raft_consensus.cc:2726] T 294d4bcc1e7a477cb56455581277a60f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260809 15:47:20.098878 23274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73f6f44c259547f1b05a3a434616fa54" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:20.099653 23274 raft_consensus.cc:2362] T 73f6f44c259547f1b05a3a434616fa54 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 9 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 9. Re-sending same reply.
I20260809 15:47:20.082063 28416 raft_consensus.cc:504] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
W20260809 15:47:20.100250 23273 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55434 (request call id 315) took 2989 ms (client timeout 2054 ms). Trace:
W20260809 15:47:20.100660 23273 rpcz_store.cc:269] 0809 15:47:17.110441 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:17.110580 (+ 139us) service_pool.cc:227] Handling call
0809 15:47:20.099892 (+2989312us) spinlock_profiling.cc:243] Waited 93.9 ms on lock 0x7b54003aa7d4. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e92242ae 00007f34ee161c27 00007f34e7849c04 00007f34e7849b92 00007f34e7849ac1 00007f34e7849a3c 00007f34e7848cf2 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:47:20.100068 (+ 176us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":215910656}
I20260809 15:47:20.101959 28416 leader_election.cc:288] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:20.103000 24453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f9d6565aa4c45a3b3f01210d0eb71f7" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:47:20.101728 23228 leader_election.cc:302] T 05ad7d8fdca64e3da527280ca5be9280 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters: c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:20.101156 23274 raft_consensus.cc:1226] T 297c82ed906d4b48b184985eed3de601 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 9 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 8. Current term is 9. Ops: []
I20260809 15:47:20.108253 23274 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53554 (request call id 1846) took 2556 ms. Trace:
I20260809 15:47:20.108706 23274 rpcz_store.cc:276] 0809 15:47:17.551067 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:20.100558 (+2549491us) service_pool.cc:227] Handling call
0809 15:47:20.100913 (+ 355us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:47:20.107108 (+ 6195us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:47:20.107392 (+ 284us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:47:20.107443 23270 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53554 (request call id 1845) took 2642 ms. Trace:
I20260809 15:47:20.111791 23270 rpcz_store.cc:276] 0809 15:47:17.455528 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:20.070164 (+2614636us) service_pool.cc:227] Handling call
0809 15:47:20.093506 (+ 23342us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:47:20.097454 (+ 3948us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:47:20.097610 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:47:20.112691 23518 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56766 (request call id 2208) took 1104 ms. Trace:
I20260809 15:47:20.113037 23518 rpcz_store.cc:276] 0809 15:47:19.008116 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:20.070050 (+1061934us) service_pool.cc:227] Handling call
0809 15:47:20.112635 (+ 42585us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:47:20.110018 23274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "763e58d6ba644eea830a6c1664d39c9b" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: true dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:47:20.114077 23274 raft_consensus.cc:3032] T 763e58d6ba644eea830a6c1664d39c9b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 LEADER]: Stepping down as leader of term 8
I20260809 15:47:20.114775 23395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "953eea268e17491ca76f3a555860a512" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:47:20.114308 23274 raft_consensus.cc:729] T 763e58d6ba644eea830a6c1664d39c9b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Running, Role: LEADER
I20260809 15:47:20.115659 23280 raft_consensus.cc:2379] T ae6d2c08a7a94f58ac8dd698e3fd456c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d in current term 9: Already voted for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in this term.
I20260809 15:47:20.115899 23395 raft_consensus.cc:2379] T 953eea268e17491ca76f3a555860a512 P c0f38c1908414566aa9115bb2cbdd85d [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c in current term 9: Already voted for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in this term.
I20260809 15:47:20.116597 23276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e71a05ba1d4d42bd95c3f85f9313399e" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:20.117362 28416 raft_consensus.cc:2781] T 05ad7d8fdca64e3da527280ca5be9280 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260809 15:47:20.117794 28416 raft_consensus.cc:482] T 05ad7d8fdca64e3da527280ca5be9280 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Starting leader election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:47:20.117933 23518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7660dfa2ef694927b8eda4100a2bcd9b" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:20.118219 28416 raft_consensus.cc:3037] T 05ad7d8fdca64e3da527280ca5be9280 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Advancing to term 8
I20260809 15:47:20.119930 23274 consensus_queue.cc:260] T 763e58d6ba644eea830a6c1664d39c9b P 67b3aeadfa7a432abe8af4d292e04ab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:20.120395 23402 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55332 (request call id 1509) took 3312 ms. Trace:
I20260809 15:47:20.120718 23402 rpcz_store.cc:276] 0809 15:47:16.808401 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:16.811067 (+ 2666us) service_pool.cc:227] Handling call
0809 15:47:16.811406 (+ 339us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:47:17.774027 (+962621us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:47:20.120033 (+2346006us) spinlock_profiling.cc:243] Waited 886 ms on lock 0x7b54003741d4. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e9220f51 00007f34e921f99e 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772 00007f34e764a94c 00007f34e7649d86
0809 15:47:20.120247 (+ 214us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2038783360}
I20260809 15:47:20.120981 23274 raft_consensus.cc:3037] T 763e58d6ba644eea830a6c1664d39c9b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Advancing to term 9
I20260809 15:47:20.120626 23518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2db057fd10e149a9895dff6ef8c9f038" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:47:20.116202 23517 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56766 (request call id 2203) took 1276 ms. Trace:
I20260809 15:47:20.124629 23517 rpcz_store.cc:276] 0809 15:47:18.839534 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:20.092007 (+1252473us) service_pool.cc:227] Handling call
0809 15:47:20.116146 (+ 24139us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:47:20.107787 28447 raft_consensus.cc:2781] T 4bcac4b24e7a4448abdc934bba5a624e P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260809 15:47:20.127300 23392 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52856 (request call id 415) took 5078 ms. Trace:
I20260809 15:47:20.127674 23392 rpcz_store.cc:276] 0809 15:47:15.048542 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:15.049198 (+ 656us) service_pool.cc:227] Handling call
0809 15:47:15.069789 (+ 20591us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:47:15.182641 (+112852us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:47:15.182669 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 15:47:15.182706 (+ 37us) raft_consensus.cc:1631] Marking committed up to 6
0809 15:47:15.182735 (+ 29us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:47:20.126811 (+4944076us) spinlock_profiling.cc:243] Waited 113 ms on lock 0x7b54003750d4. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e9220f51 00007f34e921f99e 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772 00007f34e764a94c 00007f34e7649d86
0809 15:47:20.126893 (+ 82us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:47:20.127103 (+ 210us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":259129088}
I20260809 15:47:20.132865 28784 leader_election.cc:288] T ae6d2c08a7a94f58ac8dd698e3fd456c P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 9 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
W20260809 15:47:20.133596 28836 log.cc:924] Time spent T 763e58d6ba644eea830a6c1664d39c9b P 67b3aeadfa7a432abe8af4d292e04ab9: Append to log took a long time: real 0.071s user 0.001s sys 0.000s
W20260809 15:47:20.134079 24410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 69069.7us
I20260809 15:47:20.127223 23351 leader_election.cc:302] T 3296ef50be254fc29cfc77d22a8f7a23 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters: 61e69695258f48baa984c9d395e8ce3f
I20260809 15:47:20.138211 23517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7660dfa2ef694927b8eda4100a2bcd9b" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:47:20.139393 23518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4564dfc6d4094c1b9240b367ccecf3b1" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:47:20.139534 23353 leader_election.cc:302] T 7660dfa2ef694927b8eda4100a2bcd9b P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c
I20260809 15:47:20.140275 23392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "213543cdbe814c89a17a963469922c1b" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:47:20.140745 23277 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53562 (request call id 1996) took 2823 ms. Trace:
I20260809 15:47:20.141252 23518 raft_consensus.cc:3037] T 4564dfc6d4094c1b9240b367ccecf3b1 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Advancing to term 8
I20260809 15:47:20.141932 23392 raft_consensus.cc:2379] T 213543cdbe814c89a17a963469922c1b P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c in current term 5: Already voted for candidate c0f38c1908414566aa9115bb2cbdd85d in this term.
I20260809 15:47:20.146649 28474 leader_election.cc:288] T 297c82ed906d4b48b184985eed3de601 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:47:20.152704 23277 rpcz_store.cc:276] 0809 15:47:17.300088 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:17.301575 (+ 1487us) service_pool.cc:227] Handling call
0809 15:47:17.331958 (+ 30383us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:47:18.138598 (+806640us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:47:18.138627 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 15:47:18.138666 (+ 39us) raft_consensus.cc:1631] Marking committed up to 6
0809 15:47:18.138693 (+ 27us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:47:20.119124 (+1980431us) spinlock_profiling.cc:243] Waited 806 ms on lock 0x7b54000415d4. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e9220f51 00007f34e921f99e 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772 00007f34e764a94c 00007f34e7649d86
0809 15:47:20.119187 (+ 63us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:47:20.123203 (+ 4016us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1856349056}
I20260809 15:47:20.154034 23277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a51c0baaa6e432997c9d3552af4f254" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:47:20.154834 23277 raft_consensus.cc:2396] T 0a51c0baaa6e432997c9d3552af4f254 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d for term 8 because replica has last-logged OpId of term: 7 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 6.
I20260809 15:47:20.164009 28416 raft_consensus.cc:504] T 05ad7d8fdca64e3da527280ca5be9280 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:47:20.165004 23529 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56766 (request call id 2131) took 6679 ms (client timeout 3266 ms). Trace:
W20260809 15:47:20.165362 23529 rpcz_store.cc:269] 0809 15:47:13.485697 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:13.485801 (+ 104us) service_pool.cc:227] Handling call
0809 15:47:20.164624 (+6678823us) spinlock_profiling.cc:243] Waited 5.34 s on lock 0x7b5400108bd4. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e92242ae 00007f34ee161c27 00007f34e7849c04 00007f34e7849b92 00007f34e7849ac1 00007f34e7849a3c 00007f34e7848cf2 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:47:20.164818 (+ 194us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":12289942272}
I20260809 15:47:20.165869 28416 leader_election.cc:288] T 05ad7d8fdca64e3da527280ca5be9280 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 8 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:20.166430 23529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "953eea268e17491ca76f3a555860a512" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:47:20.167299 24410 leader_election.cc:302] T 3924b648461e463b9d7925e5671d68ae P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c; no voters: c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:20.165906 23518 raft_consensus.cc:2454] T 4564dfc6d4094c1b9240b367ccecf3b1 P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 8.
I20260809 15:47:20.167788 23277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df20671a7b874dbb98c0ef28371c1308" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:20.167178 23529 raft_consensus.cc:2379] T 953eea268e17491ca76f3a555860a512 P 65c36d164b0540f099d592f38307ee6c [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 in current term 9: Already voted for candidate 65c36d164b0540f099d592f38307ee6c in this term.
I20260809 15:47:20.169373 23270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "538a228ddfb54bc6a36528d5ff07597b" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:47:20.175733 23270 raft_consensus.cc:2396] T 538a228ddfb54bc6a36528d5ff07597b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d for term 8 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20260809 15:47:20.177281 28523 raft_consensus.cc:2726] T 7660dfa2ef694927b8eda4100a2bcd9b P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260809 15:47:20.178056 23270 raft_consensus.cc:1226] T a3622f6e3e3641cea1960e3a31498bb7 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 5. Current term is 6. Ops: []
I20260809 15:47:20.178700 23270 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53554 (request call id 1853) took 2190 ms. Trace:
I20260809 15:47:20.179024 24453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3aa6a9a52804df0ba9b0ca3c82f38c3" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:47:20.180414 23518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e17dcb5efe45b8b6a8c42da03b4819" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:20.179056 23270 rpcz_store.cc:276] 0809 15:47:17.988557 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:20.177407 (+2188850us) service_pool.cc:227] Handling call
0809 15:47:20.177810 (+ 403us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:47:20.178465 (+ 655us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:47:20.178631 (+ 166us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:47:20.128589 28447 raft_consensus.cc:482] T 4bcac4b24e7a4448abdc934bba5a624e P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Starting leader election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
I20260809 15:47:20.174362 23404 raft_consensus.cc:2454] T 30c1740ef6874df0a332fbd6fd4b03ce P c0f38c1908414566aa9115bb2cbdd85d [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 9.
I20260809 15:47:20.179713 28728 consensus_queue.cc:1035] T c5c2e425a74b4e64ae70b5def750434c P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260809 15:47:20.178068 28549 raft_consensus.cc:2781] T 3296ef50be254fc29cfc77d22a8f7a23 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260809 15:47:20.182852 23529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998c6cee72cf40228c6dba064c0db2da" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:20.183190 28714 leader_election.cc:288] T 563daf42d7784db195ea99c3cb21b370 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
W20260809 15:47:20.183586 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 56507.8us
I20260809 15:47:20.183887 28449 leader_election.cc:288] T 2db057fd10e149a9895dff6ef8c9f038 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:47:20.184646 28690 leader_election.cc:288] T 6ae14a74a26f434db08c652e097385c8 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:47:20.186808 24453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "563daf42d7784db195ea99c3cb21b370" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:47:20.185189 23472 leader_election.cc:302] T d1a134ba33ea4076bbaac0ae460c17ba P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:47:20.186002 23283 raft_consensus.cc:1226] T ba56725e42f74a1baa4c04d9660727f9 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 5. Current term is 7. Ops: []
I20260809 15:47:20.184808 28474 consensus_queue.cc:1046] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260809 15:47:20.181135 23518 raft_consensus.cc:2454] T 28e17dcb5efe45b8b6a8c42da03b4819 P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 8.
I20260809 15:47:20.191632 23273 raft_consensus.cc:1226] T ae6d2c08a7a94f58ac8dd698e3fd456c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 9 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 8. Current term is 9. Ops: []
I20260809 15:47:20.191783 23518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b30f28311964a0b9149a50495e83d8f" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:47:20.189836 23517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3622f6e3e3641cea1960e3a31498bb7" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:47:20.190780 23283 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53554 (request call id 1864) took 1386 ms. Trace:
I20260809 15:47:20.191931 23280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3622f6e3e3641cea1960e3a31498bb7" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:47:20.193079 23280 raft_consensus.cc:3037] T a3622f6e3e3641cea1960e3a31498bb7 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Advancing to term 7
I20260809 15:47:20.192518 23283 rpcz_store.cc:276] 0809 15:47:18.804325 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:20.185509 (+1381184us) service_pool.cc:227] Handling call
0809 15:47:20.185832 (+ 323us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:47:20.190560 (+ 4728us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:47:20.190730 (+ 170us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:47:20.194423 23472 leader_election.cc:302] T f9e7cadcd24d45fcbfa84209225e8f2d P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:47:20.196350 28676 raft_consensus.cc:482] T 16ca412a0dae4feebe85dd01bf1a9173 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
I20260809 15:47:20.198328 28627 raft_consensus.cc:2781] T d1a134ba33ea4076bbaac0ae460c17ba P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260809 15:47:20.198336 23283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f9d6565aa4c45a3b3f01210d0eb71f7" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:20.199420 23283 raft_consensus.cc:2454] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 4.
I20260809 15:47:20.200305 23283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "213543cdbe814c89a17a963469922c1b" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:47:20.200910 28784 raft_consensus.cc:482] T 4b30f28311964a0b9149a50495e83d8f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
W20260809 15:47:20.201241 23470 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 1.23763e+06us
I20260809 15:47:20.201531 28714 raft_consensus.cc:482] T c0741ab44d99439abc3aa3bdd27ee160 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:47:20.202185 28714 raft_consensus.cc:504] T c0741ab44d99439abc3aa3bdd27ee160 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:20.203281 28335 raft_consensus.cc:2781] T 3924b648461e463b9d7925e5671d68ae P 61e69695258f48baa984c9d395e8ce3f [term 10 FOLLOWER]: Leader election won for term 10
I20260809 15:47:20.198611 28676 raft_consensus.cc:504] T 16ca412a0dae4feebe85dd01bf1a9173 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:20.203943 28335 raft_consensus.cc:686] T 3924b648461e463b9d7925e5671d68ae P 61e69695258f48baa984c9d395e8ce3f [term 10 LEADER]: Becoming Leader. State: Replica: 61e69695258f48baa984c9d395e8ce3f, State: Running, Role: LEADER
I20260809 15:47:20.204730 28714 leader_election.cc:288] T c0741ab44d99439abc3aa3bdd27ee160 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:20.205495 28784 raft_consensus.cc:504] T 4b30f28311964a0b9149a50495e83d8f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } attrs { replace: true } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } }
I20260809 15:47:20.205889 28676 leader_election.cc:288] T 16ca412a0dae4feebe85dd01bf1a9173 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:20.207751 28784 leader_election.cc:288] T 4b30f28311964a0b9149a50495e83d8f P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:20.207849 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16ca412a0dae4feebe85dd01bf1a9173" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:47:20.210811 28597 leader_election.cc:288] T a3622f6e3e3641cea1960e3a31498bb7 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:47:20.210939 23529 raft_consensus.cc:2454] T 998c6cee72cf40228c6dba064c0db2da P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 8.
I20260809 15:47:20.211830 23529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472816994d814cb198b69e5248697440" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:20.209579 23280 raft_consensus.cc:2454] T a3622f6e3e3641cea1960e3a31498bb7 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 7.
I20260809 15:47:20.213506 23277 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53554 (request call id 1872) took 1296 ms. Trace:
I20260809 15:47:20.213873 23277 rpcz_store.cc:276] 0809 15:47:18.916876 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:20.194170 (+1277294us) service_pool.cc:227] Handling call
0809 15:47:20.212912 (+ 18742us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:47:20.213187 (+ 275us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:47:20.213211 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 15:47:20.213267 (+ 56us) raft_consensus.cc:1631] Marking committed up to 6
0809 15:47:20.213301 (+ 34us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:47:20.213351 (+ 50us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:47:20.213458 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:47:20.215677 28745 raft_consensus.cc:482] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 61e69695258f48baa984c9d395e8ce3f [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:47:20.216022 23404 raft_consensus.cc:1226] T 4b30f28311964a0b9149a50495e83d8f P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 5. Current term is 6. Ops: [5.5-5.5]
I20260809 15:47:20.216518 23351 leader_election.cc:302] T d1a134ba33ea4076bbaac0ae460c17ba P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:47:20.217491 28309 raft_consensus.cc:2726] T d1a134ba33ea4076bbaac0ae460c17ba P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260809 15:47:20.217963 28285 raft_consensus.cc:3032] T 4b30f28311964a0b9149a50495e83d8f P 65c36d164b0540f099d592f38307ee6c [term 5 LEADER]: Stepping down as leader of term 5
I20260809 15:47:20.205080 28335 consensus_queue.cc:237] T 3924b648461e463b9d7925e5671d68ae P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:20.218365 28285 raft_consensus.cc:729] T 4b30f28311964a0b9149a50495e83d8f P 65c36d164b0540f099d592f38307ee6c [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:47:20.218734 23518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5712f6b6fffc4cc5a8ddfbe63e4097df" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:20.216216 28745 raft_consensus.cc:504] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 61e69695258f48baa984c9d395e8ce3f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:20.188519 28447 raft_consensus.cc:3037] T 4bcac4b24e7a4448abdc934bba5a624e P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Advancing to term 5
I20260809 15:47:20.220036 23518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ae14a74a26f434db08c652e097385c8" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:20.218086 23351 leader_election.cc:302] T c5c2e425a74b4e64ae70b5def750434c P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:47:20.188419 28549 raft_consensus.cc:482] T 3296ef50be254fc29cfc77d22a8f7a23 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Starting leader election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
I20260809 15:47:20.200927 23283 raft_consensus.cc:3037] T 213543cdbe814c89a17a963469922c1b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 3 FOLLOWER]: Advancing to term 5
I20260809 15:47:20.212421 23529 raft_consensus.cc:2454] T 472816994d814cb198b69e5248697440 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 6.
I20260809 15:47:20.212334 24448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5712f6b6fffc4cc5a8ddfbe63e4097df" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:47:20.225605 24410 leader_election.cc:302] T 472816994d814cb198b69e5248697440 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c; no voters: c0f38c1908414566aa9115bb2cbdd85d
W20260809 15:47:20.201889 23282 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53554 (request call id 1861) took 1486 ms (client timeout 1917 ms). Trace:
I20260809 15:47:20.228003 23270 raft_consensus.cc:1226] T 0c29937a80724ddb9d07a27534a146f3 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 9 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 7. Current term is 9. Ops: []
W20260809 15:47:20.226526 23282 rpcz_store.cc:269] 0809 15:47:18.714933 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:18.830810 (+115877us) service_pool.cc:227] Handling call
0809 15:47:20.201526 (+1370716us) spinlock_profiling.cc:243] Waited 195 ms on lock 0x7b54003a8ed4. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e92242ae 00007f34ee161c27 00007f34e7849c04 00007f34e7849b92 00007f34e7849ac1 00007f34e7849a3c 00007f34e7848cf2 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:47:20.201730 (+ 204us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":448201984}
I20260809 15:47:20.231925 24453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0741ab44d99439abc3aa3bdd27ee160" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:47:20.233016 23276 raft_consensus.cc:2454] T e71a05ba1d4d42bd95c3f85f9313399e P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 7.
I20260809 15:47:20.233134 23270 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53554 (request call id 1862) took 1499 ms. Trace:
I20260809 15:47:20.230419 24448 raft_consensus.cc:2454] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 4.
I20260809 15:47:20.223307 28627 raft_consensus.cc:482] T d1a134ba33ea4076bbaac0ae460c17ba P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:47:20.233994 23392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "297c82ed906d4b48b184985eed3de601" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:20.234735 23518 raft_consensus.cc:2454] T 6ae14a74a26f434db08c652e097385c8 P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 8.
I20260809 15:47:20.236536 28532 raft_consensus.cc:2781] T 472816994d814cb198b69e5248697440 P 61e69695258f48baa984c9d395e8ce3f [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260809 15:47:20.239346 23518 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56766 (request call id 2202) took 1400 ms. Trace:
I20260809 15:47:20.239590 23518 rpcz_store.cc:276] 0809 15:47:18.839296 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:20.219650 (+1380354us) service_pool.cc:227] Handling call
0809 15:47:20.239299 (+ 19649us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:47:20.240128 23519 raft_consensus.cc:2454] T 763e58d6ba644eea830a6c1664d39c9b P 65c36d164b0540f099d592f38307ee6c [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 9.
I20260809 15:47:20.233927 23270 rpcz_store.cc:276] 0809 15:47:18.730304 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:20.194687 (+1464383us) service_pool.cc:227] Handling call
0809 15:47:20.225203 (+ 30516us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:47:20.228961 (+ 3758us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:47:20.229148 (+ 187us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:47:20.226409 28549 raft_consensus.cc:3037] T 3296ef50be254fc29cfc77d22a8f7a23 P c0f38c1908414566aa9115bb2cbdd85d [term 2 FOLLOWER]: Advancing to term 3
W20260809 15:47:20.243711 23519 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56766 (request call id 2207) took 1349 ms (client timeout 1529 ms). Trace:
I20260809 15:47:20.242920 24448 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38722 (request call id 569) took 1353 ms. Trace:
I20260809 15:47:20.243912 23517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30c1740ef6874df0a332fbd6fd4b03ce" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 9 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:20.219308 28285 consensus_queue.cc:260] T 4b30f28311964a0b9149a50495e83d8f P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } attrs { replace: true } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } }
W20260809 15:47:20.246598 28220 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 8 current term: 9 is higher.
I20260809 15:47:20.244344 24448 rpcz_store.cc:276] 0809 15:47:18.889636 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:18.889770 (+ 134us) service_pool.cc:227] Handling call
0809 15:47:20.212068 (+1322298us) spinlock_profiling.cc:243] Waited 1.23 ms on lock 0x7b5800005498. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e613cf11 00007f34e62c0851 00007f34e62c0702 00007f34ef1b0b09 00007f34ee0ec730 00007f34ee1b279b 00007f34ee1b2821 00007f34ee1621b0 00007f34ee161a94 00007f34e7849c04 00007f34e7849b92
0809 15:47:20.242873 (+ 30805us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2833152}
I20260809 15:47:20.244328 24453 raft_consensus.cc:2454] T c0741ab44d99439abc3aa3bdd27ee160 P 61e69695258f48baa984c9d395e8ce3f [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 8.
I20260809 15:47:20.244800 23404 raft_consensus.cc:1226] T 0c29937a80724ddb9d07a27534a146f3 P c0f38c1908414566aa9115bb2cbdd85d [term 9 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 7. Current term is 9. Ops: [7.3-7.3]
W20260809 15:47:20.245361 23519 rpcz_store.cc:269] 0809 15:47:18.891013 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:19.626134 (+735121us) service_pool.cc:227] Handling call
0809 15:47:20.240623 (+614489us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:47:20.248005 23529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0721a4403a440f383318a3cea9d1442" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:20.249722 23270 raft_consensus.cc:1261] T d2d4f749fcfd4fb3bfb8115af3471598 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260809 15:47:20.197176 28758 raft_consensus.cc:504] T e14d118b2eaa42aa8a7cd4025aba7128 P 61e69695258f48baa984c9d395e8ce3f [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
W20260809 15:47:20.251047 23472 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 56693.6us
W20260809 15:47:20.251920 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 56067.9us
I20260809 15:47:20.242348 28627 raft_consensus.cc:3037] T d1a134ba33ea4076bbaac0ae460c17ba P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Advancing to term 8
I20260809 15:47:20.246555 28309 raft_consensus.cc:2781] T c5c2e425a74b4e64ae70b5def750434c P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260809 15:47:20.257126 23472 leader_election.cc:302] T 30c1740ef6874df0a332fbd6fd4b03ce P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:47:20.252519 23518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "297c82ed906d4b48b184985eed3de601" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 10 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:20.250860 28285 raft_consensus.cc:3037] T 4b30f28311964a0b9149a50495e83d8f P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Advancing to term 6
I20260809 15:47:20.261297 24472 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38722 (request call id 566) took 2608 ms. Trace:
I20260809 15:47:20.261636 24472 rpcz_store.cc:276] 0809 15:47:17.653193 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:17.654992 (+ 1799us) service_pool.cc:227] Handling call
0809 15:47:17.767041 (+112049us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:47:17.771836 (+ 4795us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:47:17.771867 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 15:47:17.771908 (+ 41us) raft_consensus.cc:1631] Marking committed up to 7
0809 15:47:17.771938 (+ 30us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:47:20.260904 (+2488966us) spinlock_profiling.cc:243] Waited 4.6 ms on lock 0x7b54006bcbd4. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e9220f51 00007f34e921f99e 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772 00007f34e764a94c 00007f34e7649d86
0809 15:47:20.260957 (+ 53us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:47:20.261135 (+ 178us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":10576128}
I20260809 15:47:20.254357 23517 raft_consensus.cc:2379] T 30c1740ef6874df0a332fbd6fd4b03ce P 65c36d164b0540f099d592f38307ee6c [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d in current term 9: Already voted for candidate 65c36d164b0540f099d592f38307ee6c in this term.
I20260809 15:47:20.242116 28532 raft_consensus.cc:482] T 472816994d814cb198b69e5248697440 P 61e69695258f48baa984c9d395e8ce3f [term 6 FOLLOWER]: Starting leader election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:47:20.251294 24468 raft_consensus.cc:1261] T d3e94b7e50754a81b3d72a07476cb7b9 P 61e69695258f48baa984c9d395e8ce3f [term 9 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260809 15:47:20.257179 28462 raft_consensus.cc:3037] T 0c29937a80724ddb9d07a27534a146f3 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Advancing to term 8
I20260809 15:47:20.255542 23392 raft_consensus.cc:2454] T 297c82ed906d4b48b184985eed3de601 P c0f38c1908414566aa9115bb2cbdd85d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 9.
I20260809 15:47:20.269623 23273 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53554 (request call id 1852) took 2296 ms. Trace:
I20260809 15:47:20.260097 23529 raft_consensus.cc:2379] T d0721a4403a440f383318a3cea9d1442 P 65c36d164b0540f099d592f38307ee6c [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 in current term 9: Already voted for candidate 65c36d164b0540f099d592f38307ee6c in this term.
I20260809 15:47:20.270001 23273 rpcz_store.cc:276] 0809 15:47:17.971441 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:20.101448 (+2130007us) service_pool.cc:227] Handling call
0809 15:47:20.191449 (+ 90001us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:47:20.192046 (+ 597us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:47:20.264224 (+ 72178us) spinlock_profiling.cc:243] Waited 89.6 ms on lock 0x7b540030f050. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e921fa09 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:47:20.267528 (+ 3304us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":206170368}
I20260809 15:47:20.264545 23351 leader_election.cc:302] T 763e58d6ba644eea830a6c1664d39c9b P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:47:20.273921 23517 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56766 (request call id 2206) took 1402 ms. Trace:
I20260809 15:47:20.274231 23517 rpcz_store.cc:276] 0809 15:47:18.865584 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:20.223758 (+1358174us) service_pool.cc:227] Handling call
0809 15:47:20.268220 (+ 44462us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:47:20.270331 23353 leader_election.cc:302] T 6ae14a74a26f434db08c652e097385c8 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:47:20.275398 23519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c29937a80724ddb9d07a27534a146f3" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 9 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:20.258134 28309 raft_consensus.cc:482] T c5c2e425a74b4e64ae70b5def750434c P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:47:20.271337 23470 leader_election.cc:302] T 0acc47ce21c3422698d1ef7277c3ed63 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters: c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:20.242530 28450 raft_consensus.cc:504] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:20.281481 23529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad8a3b25d7b49e5a2142d5709edd0c7" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:20.282191 23518 raft_consensus.cc:2396] T 297c82ed906d4b48b184985eed3de601 P 65c36d164b0540f099d592f38307ee6c [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d for term 10 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20260809 15:47:20.270025 28532 raft_consensus.cc:3037] T 472816994d814cb198b69e5248697440 P 61e69695258f48baa984c9d395e8ce3f [term 6 FOLLOWER]: Advancing to term 7
I20260809 15:47:20.285089 28449 raft_consensus.cc:2781] T 30c1740ef6874df0a332fbd6fd4b03ce P 65c36d164b0540f099d592f38307ee6c [term 9 FOLLOWER]: Leader election won for term 9
I20260809 15:47:20.284780 23518 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56766 (request call id 2205) took 1443 ms. Trace:
I20260809 15:47:20.286221 23518 rpcz_store.cc:276] 0809 15:47:18.840008 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:20.244952 (+1404944us) service_pool.cc:227] Handling call
0809 15:47:20.283840 (+ 38888us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:47:20.287330 23529 raft_consensus.cc:2454] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 6.
I20260809 15:47:20.288508 24410 leader_election.cc:302] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c; no voters: c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:20.291635 23472 leader_election.cc:302] T 297c82ed906d4b48b184985eed3de601 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:47:20.284597 23517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba56725e42f74a1baa4c04d9660727f9" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:47:20.293869 23517 raft_consensus.cc:3037] T ba56725e42f74a1baa4c04d9660727f9 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Advancing to term 7
I20260809 15:47:20.295253 23270 raft_consensus.cc:1226] T 05ad7d8fdca64e3da527280ca5be9280 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Rejecting Update request from peer c0f38c1908414566aa9115bb2cbdd85d for earlier term 7. Current term is 8. Ops: []
I20260809 15:47:20.283298 28309 raft_consensus.cc:3037] T c5c2e425a74b4e64ae70b5def750434c P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Advancing to term 8
I20260809 15:47:20.285274 28756 raft_consensus.cc:3037] T 3d880de657e544468c82157769622a26 P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Advancing to term 5
W20260809 15:47:20.295009 23353 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 43907.7us
I20260809 15:47:20.297595 23518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5c2e425a74b4e64ae70b5def750434c" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:20.298434 23282 raft_consensus.cc:1226] T d0721a4403a440f383318a3cea9d1442 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 9 FOLLOWER]: Rejecting Update request from peer c0f38c1908414566aa9115bb2cbdd85d for earlier term 8. Current term is 9. Ops: []
I20260809 15:47:20.298543 23402 raft_consensus.cc:1226] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d [term 10 FOLLOWER]: Rejecting Update request from peer 61e69695258f48baa984c9d395e8ce3f for earlier term 9. Current term is 10. Ops: []
I20260809 15:47:20.301172 23396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3622f6e3e3641cea1960e3a31498bb7" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:20.301797 23396 raft_consensus.cc:2379] T a3622f6e3e3641cea1960e3a31498bb7 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c in current term 7: Already voted for candidate c0f38c1908414566aa9115bb2cbdd85d in this term.
I20260809 15:47:20.303339 23529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3aa6a9a52804df0ba9b0ca3c82f38c3" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
W20260809 15:47:20.304216 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 49388.7us
I20260809 15:47:20.294476 26966 raft_consensus.cc:3032] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 65c36d164b0540f099d592f38307ee6c [term 4 LEADER]: Stepping down as leader of term 4
I20260809 15:47:20.307147 23353 leader_election.cc:302] T 763e58d6ba644eea830a6c1664d39c9b P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:47:20.308754 28522 raft_consensus.cc:2781] T 6ae14a74a26f434db08c652e097385c8 P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260809 15:47:20.309235 28522 raft_consensus.cc:482] T 6ae14a74a26f434db08c652e097385c8 P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:47:20.309568 28522 raft_consensus.cc:3037] T 6ae14a74a26f434db08c652e097385c8 P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Advancing to term 9
I20260809 15:47:20.300822 28523 raft_consensus.cc:2726] T 763e58d6ba644eea830a6c1664d39c9b P c0f38c1908414566aa9115bb2cbdd85d [term 9 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260809 15:47:20.311036 28597 raft_consensus.cc:2781] T 297c82ed906d4b48b184985eed3de601 P 65c36d164b0540f099d592f38307ee6c [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260809 15:47:20.311432 28597 raft_consensus.cc:482] T 297c82ed906d4b48b184985eed3de601 P 65c36d164b0540f099d592f38307ee6c [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:47:20.311744 28597 raft_consensus.cc:3037] T 297c82ed906d4b48b184985eed3de601 P 65c36d164b0540f099d592f38307ee6c [term 9 FOLLOWER]: Advancing to term 10
I20260809 15:47:20.311841 23282 raft_consensus.cc:1261] T 538a228ddfb54bc6a36528d5ff07597b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 5. (term mismatch)
I20260809 15:47:20.312129 26966 raft_consensus.cc:729] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 65c36d164b0540f099d592f38307ee6c [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:47:20.313158 26966 consensus_queue.cc:260] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:20.308521 28525 raft_consensus.cc:2781] T f9e7cadcd24d45fcbfa84209225e8f2d P 65c36d164b0540f099d592f38307ee6c [term 9 FOLLOWER]: Leader election won for term 9
I20260809 15:47:20.315349 28302 raft_consensus.cc:2781] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 61e69695258f48baa984c9d395e8ce3f [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260809 15:47:20.315744 28302 raft_consensus.cc:482] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 61e69695258f48baa984c9d395e8ce3f [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:47:20.316071 28302 raft_consensus.cc:3037] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 61e69695258f48baa984c9d395e8ce3f [term 6 FOLLOWER]: Advancing to term 7
I20260809 15:47:20.303368 28449 raft_consensus.cc:686] T 30c1740ef6874df0a332fbd6fd4b03ce P 65c36d164b0540f099d592f38307ee6c [term 9 LEADER]: Becoming Leader. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:47:20.317350 28449 consensus_queue.cc:237] T 30c1740ef6874df0a332fbd6fd4b03ce P 65c36d164b0540f099d592f38307ee6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:20.321005 23282 pending_rounds.cc:76] T 538a228ddfb54bc6a36528d5ff07597b P 67b3aeadfa7a432abe8af4d292e04ab9: Aborting all ops after (but not including) 4
I20260809 15:47:20.322070 23392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2db057fd10e149a9895dff6ef8c9f038" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:20.322819 23392 raft_consensus.cc:2379] T 2db057fd10e149a9895dff6ef8c9f038 P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c in current term 8: Already voted for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in this term.
W20260809 15:47:20.325636 23470 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 67216us
I20260809 15:47:20.324688 23282 pending_rounds.cc:97] T 538a228ddfb54bc6a36528d5ff07597b P 67b3aeadfa7a432abe8af4d292e04ab9: Aborting uncommitted NO_OP operation due to leader change: 7.5
I20260809 15:47:20.328691 23270 pending_rounds.cc:76] T 7baa2492df0340c4a5ac241959ce1ac3 P 67b3aeadfa7a432abe8af4d292e04ab9: Aborting all ops after (but not including) 3
I20260809 15:47:20.328646 23282 raft_consensus.cc:2866] T 538a228ddfb54bc6a36528d5ff07597b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 15:47:20.329025 23270 pending_rounds.cc:97] T 7baa2492df0340c4a5ac241959ce1ac3 P 67b3aeadfa7a432abe8af4d292e04ab9: Aborting uncommitted NO_OP operation due to leader change: 5.4
I20260809 15:47:20.329603 23270 raft_consensus.cc:2866] T 7baa2492df0340c4a5ac241959ce1ac3 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 15:47:20.330336 23518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3622f6e3e3641cea1960e3a31498bb7" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:20.308701 28425 raft_consensus.cc:2781] T 0acc47ce21c3422698d1ef7277c3ed63 P 65c36d164b0540f099d592f38307ee6c [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260809 15:47:20.331763 23529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0da4d5283c8a46878bd39b85e5c71f66" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:20.347295 23392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3aa6a9a52804df0ba9b0ca3c82f38c3" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:20.345181 23470 leader_election.cc:302] T 00a1881267a14ba8b78c81040f17a014 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters: c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:20.346765 23529 raft_consensus.cc:2454] T 0da4d5283c8a46878bd39b85e5c71f66 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 6.
I20260809 15:47:20.349774 23353 leader_election.cc:302] T 30c1740ef6874df0a332fbd6fd4b03ce P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:47:20.324790 28523 raft_consensus.cc:2781] T 763e58d6ba644eea830a6c1664d39c9b P c0f38c1908414566aa9115bb2cbdd85d [term 9 FOLLOWER]: Leader election won for term 9
I20260809 15:47:20.351193 23351 leader_election.cc:302] T 8e4ffcf83c234ae7910ddef3ec0b4bda P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters: 61e69695258f48baa984c9d395e8ce3f
I20260809 15:47:20.350780 23274 raft_consensus.cc:2396] T 763e58d6ba644eea830a6c1664d39c9b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d for term 9 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 4.
W20260809 15:47:20.354542 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 55274.3us
I20260809 15:47:20.321452 28525 raft_consensus.cc:686] T f9e7cadcd24d45fcbfa84209225e8f2d P 65c36d164b0540f099d592f38307ee6c [term 9 LEADER]: Becoming Leader. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:47:20.358296 28546 raft_consensus.cc:2781] T 8e4ffcf83c234ae7910ddef3ec0b4bda P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260809 15:47:20.365733 23518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3622f6e3e3641cea1960e3a31498bb7" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:20.358783 23351 leader_election.cc:302] T 07fcabb569414bddbd39df6cedd24e64 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
W20260809 15:47:20.360512 23522 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56770 (request call id 1498) took 2562 ms (client timeout 2002 ms). Trace:
I20260809 15:47:20.361268 23400 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55320 (request call id 1648) took 3506 ms. Trace:
W20260809 15:47:20.366823 23522 rpcz_store.cc:269] 0809 15:47:17.798461 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:17.800289 (+ 1828us) service_pool.cc:227] Handling call
0809 15:47:20.351585 (+2551296us) spinlock_profiling.cc:243] Waited 8.67 ms on lock 0x7b540016ec54. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e92242ae 00007f34ee161c27 00007f34e7849c04 00007f34e7849b92 00007f34e7849ac1 00007f34e7849a3c 00007f34e7848cf2 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:47:20.360311 (+ 8726us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":19944832}
I20260809 15:47:20.372025 28550 raft_consensus.cc:2781] T 00a1881267a14ba8b78c81040f17a014 P 65c36d164b0540f099d592f38307ee6c [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260809 15:47:20.372453 28550 raft_consensus.cc:482] T 00a1881267a14ba8b78c81040f17a014 P 65c36d164b0540f099d592f38307ee6c [term 9 FOLLOWER]: Starting leader election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:47:20.349311 28425 raft_consensus.cc:482] T 0acc47ce21c3422698d1ef7277c3ed63 P 65c36d164b0540f099d592f38307ee6c [term 9 FOLLOWER]: Starting leader election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:47:20.367437 23400 rpcz_store.cc:276] 0809 15:47:16.854870 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:18.726402 (+1871532us) service_pool.cc:227] Handling call
0809 15:47:18.813234 (+ 86832us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:47:18.819389 (+ 6155us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:47:20.360868 (+1541479us) spinlock_profiling.cc:243] Waited 83.7 ms on lock 0x7b540031abd0. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e921fa09 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:47:20.361088 (+ 220us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":192574336}
I20260809 15:47:20.379761 23396 raft_consensus.cc:1226] T 7660dfa2ef694927b8eda4100a2bcd9b P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 6. Current term is 7. Ops: []
I20260809 15:47:20.372879 28550 raft_consensus.cc:3037] T 00a1881267a14ba8b78c81040f17a014 P 65c36d164b0540f099d592f38307ee6c [term 9 FOLLOWER]: Advancing to term 10
I20260809 15:47:20.370178 28546 raft_consensus.cc:482] T 8e4ffcf83c234ae7910ddef3ec0b4bda P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Starting leader election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
I20260809 15:47:20.366070 28523 raft_consensus.cc:686] T 763e58d6ba644eea830a6c1664d39c9b P c0f38c1908414566aa9115bb2cbdd85d [term 9 LEADER]: Becoming Leader. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:47:20.383630 23522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3aa6a9a52804df0ba9b0ca3c82f38c3" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:20.386116 23529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba56725e42f74a1baa4c04d9660727f9" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
W20260809 15:47:20.388809 23470 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 58758.7us
I20260809 15:47:20.374240 28745 leader_election.cc:288] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:20.390410 23519 raft_consensus.cc:2396] T 0c29937a80724ddb9d07a27534a146f3 P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d for term 9 because replica has last-logged OpId of term: 7 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 2.
I20260809 15:47:20.380947 28758 leader_election.cc:288] T e14d118b2eaa42aa8a7cd4025aba7128 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 9 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:20.389490 28457 raft_consensus.cc:2726] T 07fcabb569414bddbd39df6cedd24e64 P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260809 15:47:20.379192 28425 raft_consensus.cc:3037] T 0acc47ce21c3422698d1ef7277c3ed63 P 65c36d164b0540f099d592f38307ee6c [term 9 FOLLOWER]: Advancing to term 10
I20260809 15:47:20.397488 23518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0da4d5283c8a46878bd39b85e5c71f66" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:20.393533 23529 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56766 (request call id 2201) took 1554 ms. Trace:
I20260809 15:47:20.392947 23517 raft_consensus.cc:2454] T ba56725e42f74a1baa4c04d9660727f9 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 7.
I20260809 15:47:20.380633 24455 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38722 (request call id 565) took 2634 ms. Trace:
W20260809 15:47:20.399681 24406 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 157100us
I20260809 15:47:20.373935 28560 consensus_queue.cc:1035] T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.119s
I20260809 15:47:20.401553 23529 rpcz_store.cc:276] 0809 15:47:18.839041 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:20.356893 (+1517852us) service_pool.cc:227] Handling call
0809 15:47:20.393491 (+ 36598us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:47:20.390230 28546 raft_consensus.cc:3037] T 8e4ffcf83c234ae7910ddef3ec0b4bda P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Advancing to term 5
I20260809 15:47:20.401229 23518 raft_consensus.cc:2454] T 0da4d5283c8a46878bd39b85e5c71f66 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 6.
W20260809 15:47:20.406387 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 47676.9us
I20260809 15:47:20.406569 23518 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56766 (request call id 2197) took 1571 ms. Trace:
I20260809 15:47:20.406874 23518 rpcz_store.cc:276] 0809 15:47:18.834702 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:20.375174 (+1540472us) service_pool.cc:227] Handling call
0809 15:47:20.406510 (+ 31336us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:47:20.405478 23517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e14d118b2eaa42aa8a7cd4025aba7128" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:47:20.407666 23517 raft_consensus.cc:3037] T e14d118b2eaa42aa8a7cd4025aba7128 P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Advancing to term 9
I20260809 15:47:20.409113 24410 leader_election.cc:302] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:20.410953 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a823d35f4f348d1b52ba9e6d80b00d2" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f"
I20260809 15:47:20.411974 23522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c29937a80724ddb9d07a27534a146f3" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 9 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:47:20.413609 23351 leader_election.cc:302] T f7715d71dbac447b9351fc5eb36c5e80 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:47:20.408476 23470 leader_election.cc:302] T 7baa2492df0340c4a5ac241959ce1ac3 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d
W20260809 15:47:20.415357 23353 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 70458.7us
I20260809 15:47:20.419313 23529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "213543cdbe814c89a17a963469922c1b" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:47:20.407976 28440 raft_consensus.cc:482] T 3924b648461e463b9d7925e5671d68ae P 65c36d164b0540f099d592f38307ee6c [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:20.422432 23353 leader_election.cc:302] T 297c82ed906d4b48b184985eed3de601 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:47:20.423771 23529 raft_consensus.cc:2379] T 213543cdbe814c89a17a963469922c1b P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d in current term 5: Already voted for candidate 65c36d164b0540f099d592f38307ee6c in this term.
I20260809 15:47:20.424480 23228 leader_election.cc:302] T 213543cdbe814c89a17a963469922c1b P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters: c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:20.410643 28285 raft_consensus.cc:482] T 0c29937a80724ddb9d07a27534a146f3 P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:20.428633 23518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2d4f749fcfd4fb3bfb8115af3471598" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:20.404094 28450 leader_election.cc:288] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 10 election: Requested vote from peers 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:20.427493 28474 consensus_queue.cc:1046] T 7660dfa2ef694927b8eda4100a2bcd9b P 65c36d164b0540f099d592f38307ee6c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.004s
I20260809 15:47:20.427156 28462 consensus_queue.cc:1046] T 07fcabb569414bddbd39df6cedd24e64 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.005s
I20260809 15:47:20.427505 28285 raft_consensus.cc:504] T 0c29937a80724ddb9d07a27534a146f3 P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:20.434291 28521 raft_consensus.cc:2726] T 297c82ed906d4b48b184985eed3de601 P c0f38c1908414566aa9115bb2cbdd85d [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260809 15:47:20.435007 28845 log.cc:924] Time spent T ae6d2c08a7a94f58ac8dd698e3fd456c P c0f38c1908414566aa9115bb2cbdd85d: Append to log took a long time: real 0.154s user 0.001s sys 0.000s
I20260809 15:47:20.435436 23520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee02fff2c9c48a19f43eb358a89e7b0" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:47:20.436022 23520 raft_consensus.cc:2362] T cee02fff2c9c48a19f43eb358a89e7b0 P 65c36d164b0540f099d592f38307ee6c [term 9 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 9. Re-sending same reply.
W20260809 15:47:20.436507 23520 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44018 (request call id 350) took 2456 ms (client timeout 1575 ms). Trace:
I20260809 15:47:20.436936 28825 raft_consensus.cc:2726] T 7baa2492df0340c4a5ac241959ce1ac3 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260809 15:47:20.436791 28523 consensus_queue.cc:237] T 763e58d6ba644eea830a6c1664d39c9b P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:20.437625 28248 raft_consensus.cc:3037] T f7715d71dbac447b9351fc5eb36c5e80 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Advancing to term 7
W20260809 15:47:20.439523 28848 log.cc:924] Time spent T c0741ab44d99439abc3aa3bdd27ee160 P 65c36d164b0540f099d592f38307ee6c: Append to log took a long time: real 0.127s user 0.000s sys 0.001s
I20260809 15:47:20.415937 28745 raft_consensus.cc:2726] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 61e69695258f48baa984c9d395e8ce3f [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260809 15:47:20.436798 23520 rpcz_store.cc:269] 0809 15:47:17.980244 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:17.986671 (+ 6427us) service_pool.cc:227] Handling call
0809 15:47:20.435161 (+2448490us) spinlock_profiling.cc:243] Waited 12.7 ms on lock 0x7b5800004298. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e613cf11 00007f34e62c0851 00007f34e62c0702 00007f34ef1b0b09 00007f34ee0ec730 00007f34ee1b279b 00007f34ee1b2821 00007f34ee1621b0 00007f34ee161a94 00007f34e7849c04 00007f34e7849b92
0809 15:47:20.436413 (+ 1252us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":29300608}
I20260809 15:47:20.441248 23394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c29937a80724ddb9d07a27534a146f3" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 9 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:20.438740 28550 raft_consensus.cc:504] T 00a1881267a14ba8b78c81040f17a014 P 65c36d164b0540f099d592f38307ee6c [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:20.451253 23394 raft_consensus.cc:2379] T 0c29937a80724ddb9d07a27534a146f3 P c0f38c1908414566aa9115bb2cbdd85d [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c in current term 9: Already voted for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in this term.
I20260809 15:47:20.452898 23546 tablet_copy_source_session.cc:205] T ba56725e42f74a1baa4c04d9660727f9 P 65c36d164b0540f099d592f38307ee6c: Tablet Copy: opened 0 blocks and 1 log segments
I20260809 15:47:20.453832 28550 leader_election.cc:288] T 00a1881267a14ba8b78c81040f17a014 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 10 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:47:20.428256 28440 raft_consensus.cc:504] T 3924b648461e463b9d7925e5671d68ae P 65c36d164b0540f099d592f38307ee6c [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:20.456632 23470 leader_election.cc:302] T 73f6f44c259547f1b05a3a434616fa54 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d
W20260809 15:47:20.459398 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 45874.4us
I20260809 15:47:20.461118 23518 raft_consensus.cc:2454] T d2d4f749fcfd4fb3bfb8115af3471598 P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 8.
I20260809 15:47:20.438905 28406 raft_consensus.cc:3037] T cee02fff2c9c48a19f43eb358a89e7b0 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Advancing to term 9
I20260809 15:47:20.463691 23522 raft_consensus.cc:1226] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Rejecting Update request from peer c0f38c1908414566aa9115bb2cbdd85d for earlier term 5. Current term is 6. Ops: []
I20260809 15:47:20.464067 28440 leader_election.cc:288] T 3924b648461e463b9d7925e5671d68ae P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:20.464255 23522 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56766 (request call id 2183) took 2594 ms. Trace:
I20260809 15:47:20.464567 23522 rpcz_store.cc:276] 0809 15:47:17.869355 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:20.430886 (+2561531us) service_pool.cc:227] Handling call
0809 15:47:20.463542 (+ 32656us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:47:20.464034 (+ 492us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:47:20.464187 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:47:20.428390 28302 raft_consensus.cc:504] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 61e69695258f48baa984c9d395e8ce3f [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:20.469091 28302 leader_election.cc:288] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 7 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:20.470420 28695 raft_consensus.cc:2726] T 30c1740ef6874df0a332fbd6fd4b03ce P c0f38c1908414566aa9115bb2cbdd85d [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260809 15:47:20.470891 28319 leader_election.cc:288] T 0c48a3049897452383257aa757834fb5 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
W20260809 15:47:20.473721 23353 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 51429.6us
I20260809 15:47:20.473834 23396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3924b648461e463b9d7925e5671d68ae" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:20.478454 23396 raft_consensus.cc:2454] T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 10.
I20260809 15:47:20.480252 23394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad8a3b25d7b49e5a2142d5709edd0c7" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:47:20.481611 23223 leader_election.cc:302] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:47:20.482360 24472 raft_consensus.cc:1226] T e14d118b2eaa42aa8a7cd4025aba7128 P 61e69695258f48baa984c9d395e8ce3f [term 9 FOLLOWER]: Rejecting Update request from peer 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 8. Current term is 9. Ops: []
I20260809 15:47:20.482791 24451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3924b648461e463b9d7925e5671d68ae" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:47:20.483656 23353 leader_election.cc:302] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c
I20260809 15:47:20.434475 28285 leader_election.cc:288] T 0c29937a80724ddb9d07a27534a146f3 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:47:20.485378 23472 leader_election.cc:302] T 3924b648461e463b9d7925e5671d68ae P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
W20260809 15:47:20.482295 23223 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.68:34689, user_credentials={real_user=slave}} blocked reactor thread for 51966us
I20260809 15:47:20.489112 23517 raft_consensus.cc:2454] T e14d118b2eaa42aa8a7cd4025aba7128 P 65c36d164b0540f099d592f38307ee6c [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 9.
I20260809 15:47:20.490195 24410 leader_election.cc:302] T e14d118b2eaa42aa8a7cd4025aba7128 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c; no voters:
I20260809 15:47:20.490944 28560 raft_consensus.cc:2726] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260809 15:47:20.491678 28462 raft_consensus.cc:3032] T 07fcabb569414bddbd39df6cedd24e64 P 65c36d164b0540f099d592f38307ee6c [term 7 LEADER]: Stepping down as leader of term 7
I20260809 15:47:20.492075 28462 raft_consensus.cc:729] T 07fcabb569414bddbd39df6cedd24e64 P 65c36d164b0540f099d592f38307ee6c [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:47:20.492183 23353 leader_election.cc:302] T 0da4d5283c8a46878bd39b85e5c71f66 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:47:20.493009 28560 raft_consensus.cc:2781] T 0da4d5283c8a46878bd39b85e5c71f66 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260809 15:47:20.493438 28560 raft_consensus.cc:482] T 0da4d5283c8a46878bd39b85e5c71f66 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:47:20.493847 28560 raft_consensus.cc:3037] T 0da4d5283c8a46878bd39b85e5c71f66 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Advancing to term 7
I20260809 15:47:20.441412 28525 consensus_queue.cc:237] T f9e7cadcd24d45fcbfa84209225e8f2d P 65c36d164b0540f099d592f38307ee6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.6, Last appended by leader: 6, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:20.494300 28285 raft_consensus.cc:2781] T 3924b648461e463b9d7925e5671d68ae P 65c36d164b0540f099d592f38307ee6c [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260809 15:47:20.494731 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "907bacfa79d04a55a68c8dbe60056462" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
W20260809 15:47:20.495150 23470 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 45721.6us
I20260809 15:47:20.496364 23470 leader_election.cc:302] T 99bdb93503db4de4a1cfb0161ea0c07c P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters: c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:20.496799 23517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad8a3b25d7b49e5a2142d5709edd0c7" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:47:20.497426 23517 raft_consensus.cc:3037] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Advancing to term 7
I20260809 15:47:20.492959 28462 consensus_queue.cc:260] T 07fcabb569414bddbd39df6cedd24e64 P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:20.490935 28546 raft_consensus.cc:504] T 8e4ffcf83c234ae7910ddef3ec0b4bda P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:20.498999 23520 raft_consensus.cc:1226] T d0721a4403a440f383318a3cea9d1442 P 65c36d164b0540f099d592f38307ee6c [term 9 FOLLOWER]: Rejecting Update request from peer c0f38c1908414566aa9115bb2cbdd85d for earlier term 8. Current term is 9. Ops: []
I20260809 15:47:20.499516 23397 raft_consensus.cc:1226] T 8e4ffcf83c234ae7910ddef3ec0b4bda P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Rejecting Update request from peer 61e69695258f48baa984c9d395e8ce3f for earlier term 4. Current term is 5. Ops: []
I20260809 15:47:20.497920 28532 raft_consensus.cc:504] T 472816994d814cb198b69e5248697440 P 61e69695258f48baa984c9d395e8ce3f [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:20.499632 23520 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56766 (request call id 2184) took 2603 ms. Trace:
I20260809 15:47:20.500046 23520 rpcz_store.cc:276] 0809 15:47:17.896547 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:20.456500 (+2559953us) service_pool.cc:227] Handling call
0809 15:47:20.498844 (+ 42344us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:47:20.499418 (+ 574us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:47:20.499586 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:47:20.501024 28532 leader_election.cc:288] T 472816994d814cb198b69e5248697440 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 7 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:20.494300 28745 raft_consensus.cc:2781] T e14d118b2eaa42aa8a7cd4025aba7128 P 61e69695258f48baa984c9d395e8ce3f [term 9 FOLLOWER]: Leader election won for term 9
I20260809 15:47:20.493594 28521 consensus_queue.cc:1046] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P c0f38c1908414566aa9115bb2cbdd85d [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.005s
I20260809 15:47:20.500977 23520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a823d35f4f348d1b52ba9e6d80b00d2" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:47:20.502224 23520 raft_consensus.cc:3037] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 65c36d164b0540f099d592f38307ee6c [term 9 FOLLOWER]: Advancing to term 10
I20260809 15:47:20.485190 23519 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56766 (request call id 2199) took 1633 ms. Trace:
I20260809 15:47:20.503757 23519 rpcz_store.cc:276] 0809 15:47:18.838523 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:20.251138 (+1412615us) service_pool.cc:227] Handling call
0809 15:47:20.472009 (+220871us) spinlock_profiling.cc:243] Waited 114 ms on lock 0x7b54003ba1d4. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e92242ae 00007f34ee161c27 00007f34e7849c04 00007f34e7849b92 00007f34e7849ac1 00007f34e7849a3c 00007f34e7848cf2 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:47:20.472131 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":261945728}
I20260809 15:47:20.505236 23351 leader_election.cc:302] T ba56725e42f74a1baa4c04d9660727f9 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:47:20.508760 24453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c48a3049897452383257aa757834fb5" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:47:20.510293 23392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00a1881267a14ba8b78c81040f17a014" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:47:20.502892 24471 raft_consensus.cc:2454] T 907bacfa79d04a55a68c8dbe60056462 P 61e69695258f48baa984c9d395e8ce3f [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 9.
I20260809 15:47:20.489063 28577 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba56725e42f74a1baa4c04d9660727f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 15:47:20.507812 23223 leader_election.cc:302] T c0741ab44d99439abc3aa3bdd27ee160 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:47:20.501747 28745 raft_consensus.cc:686] T e14d118b2eaa42aa8a7cd4025aba7128 P 61e69695258f48baa984c9d395e8ce3f [term 9 LEADER]: Becoming Leader. State: Replica: 61e69695258f48baa984c9d395e8ce3f, State: Running, Role: LEADER
I20260809 15:47:20.517282 28825 raft_consensus.cc:2726] T 73f6f44c259547f1b05a3a434616fa54 P 65c36d164b0540f099d592f38307ee6c [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260809 15:47:20.512408 24471 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38724 (request call id 587) took 1872 ms (client timeout 2209 ms). Trace:
W20260809 15:47:20.519114 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 108964us
I20260809 15:47:20.501426 28319 raft_consensus.cc:2781] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260809 15:47:20.493788 23285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b30f28311964a0b9149a50495e83d8f" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:47:20.525261 23285 raft_consensus.cc:3037] T 4b30f28311964a0b9149a50495e83d8f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Advancing to term 6
I20260809 15:47:20.526826 23529 raft_consensus.cc:1226] T 3d880de657e544468c82157769622a26 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Rejecting Update request from peer 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 4. Current term is 5. Ops: []
I20260809 15:47:20.494750 28285 raft_consensus.cc:482] T 3924b648461e463b9d7925e5671d68ae P 65c36d164b0540f099d592f38307ee6c [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:47:20.529512 23393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2424fd9bd7f14e508fd106cbfda786a4" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 9 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:47:20.499732 28462 raft_consensus.cc:3037] T 07fcabb569414bddbd39df6cedd24e64 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Advancing to term 8
I20260809 15:47:20.526717 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472816994d814cb198b69e5248697440" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
W20260809 15:47:20.524097 24471 rpcz_store.cc:269] 0809 15:47:18.639719 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:18.639880 (+ 161us) service_pool.cc:227] Handling call
0809 15:47:20.494478 (+1854598us) spinlock_profiling.cc:243] Waited 5.42 ms on lock 0x7b5800005498. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e613cf11 00007f34e62c0851 00007f34e62c0702 00007f34ef1b0b09 00007f34ee0ec730 00007f34ee1b279b 00007f34ee1b2821 00007f34ee1621b0 00007f34ee161a94 00007f34e7849c04 00007f34e7849b92
0809 15:47:20.512359 (+ 17881us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":12461952}
I20260809 15:47:20.551476 28718 raft_consensus.cc:482] T cee02fff2c9c48a19f43eb358a89e7b0 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:20.537989 23470 leader_election.cc:302] T 0c48a3049897452383257aa757834fb5 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 61e69695258f48baa984c9d395e8ce3f, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:47:20.556816 28718 raft_consensus.cc:504] T cee02fff2c9c48a19f43eb358a89e7b0 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:20.558115 23286 raft_consensus.cc:2454] T cee02fff2c9c48a19f43eb358a89e7b0 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 9.
I20260809 15:47:20.529390 28521 raft_consensus.cc:2726] T ba56725e42f74a1baa4c04d9660727f9 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260809 15:47:20.537623 28621 raft_consensus.cc:2781] T c0741ab44d99439abc3aa3bdd27ee160 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260809 15:47:20.560271 28621 raft_consensus.cc:482] T c0741ab44d99439abc3aa3bdd27ee160 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:47:20.560391 28674 consensus_queue.cc:1046] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260809 15:47:20.560706 28621 raft_consensus.cc:3037] T c0741ab44d99439abc3aa3bdd27ee160 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Advancing to term 9
I20260809 15:47:20.560875 23470 leader_election.cc:302] T a88421dba80d4d3388aa5eb62e23853b P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:20.527578 28319 raft_consensus.cc:482] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:47:20.561694 28319 raft_consensus.cc:3037] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Advancing to term 5
I20260809 15:47:20.563397 23470 leader_election.cc:302] T 3d880de657e544468c82157769622a26 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 61e69695258f48baa984c9d395e8ce3f, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:47:20.564213 28440 raft_consensus.cc:2726] T a88421dba80d4d3388aa5eb62e23853b P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260809 15:47:20.564558 28661 raft_consensus.cc:2726] T 3d880de657e544468c82157769622a26 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260809 15:47:20.565821 23470 leader_election.cc:302] T d2d4f749fcfd4fb3bfb8115af3471598 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters: c0f38c1908414566aa9115bb2cbdd85d
W20260809 15:47:20.559333 23286 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55434 (request call id 326) took 3365 ms (client timeout 2093 ms). Trace:
W20260809 15:47:20.548434 23521 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56770 (request call id 1488) took 3201 ms (client timeout 2999 ms). Trace:
I20260809 15:47:20.560925 28456 raft_consensus.cc:2726] T 0c48a3049897452383257aa757834fb5 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260809 15:47:20.567919 23470 leader_election.cc:302] T 16ca412a0dae4feebe85dd01bf1a9173 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 61e69695258f48baa984c9d395e8ce3f, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:47:20.541322 23517 raft_consensus.cc:2454] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 7.
I20260809 15:47:20.569788 24453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e4ffcf83c234ae7910ddef3ec0b4bda" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f"
I20260809 15:47:20.538802 28246 raft_consensus.cc:2781] T 99bdb93503db4de4a1cfb0161ea0c07c P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260809 15:47:20.571964 24410 leader_election.cc:302] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c; no voters:
I20260809 15:47:20.572510 23393 raft_consensus.cc:3037] T 2424fd9bd7f14e508fd106cbfda786a4 P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Advancing to term 9
I20260809 15:47:20.544838 23526 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56766 (request call id 2175) took 2776 ms. Trace:
I20260809 15:47:20.574994 23529 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56770 (request call id 1502) took 2646 ms. Trace:
I20260809 15:47:20.575387 23529 rpcz_store.cc:276] 0809 15:47:17.884840 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:20.426825 (+2541985us) service_pool.cc:227] Handling call
0809 15:47:20.477421 (+ 50596us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:47:20.530803 (+ 53382us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:47:20.530973 (+ 170us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 15:47:20.549021 23528 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56766 (request call id 2185) took 2642 ms (client timeout 2466 ms). Trace:
I20260809 15:47:20.577314 23517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472816994d814cb198b69e5248697440" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:47:20.573984 23526 rpcz_store.cc:276] 0809 15:47:17.768644 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:18.635121 (+866477us) service_pool.cc:227] Handling call
0809 15:47:20.360982 (+1725861us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:47:20.361339 (+ 357us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:47:20.361362 (+ 23us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 15:47:20.361429 (+ 67us) raft_consensus.cc:1631] Marking committed up to 6
0809 15:47:20.361465 (+ 36us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:47:20.361530 (+ 65us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:47:20.544542 (+183012us) spinlock_profiling.cc:243] Waited 1.7 s on lock 0x7b540016ec50. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e921fa09 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:47:20.544661 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3913947520}
I20260809 15:47:20.577917 23517 raft_consensus.cc:3037] T 472816994d814cb198b69e5248697440 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Advancing to term 7
I20260809 15:47:20.581125 23520 raft_consensus.cc:2454] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 65c36d164b0540f099d592f38307ee6c [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 10.
W20260809 15:47:20.568349 23286 rpcz_store.cc:269] 0809 15:47:17.193514 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:17.266445 (+ 72931us) service_pool.cc:227] Handling call
0809 15:47:20.559150 (+3292705us) spinlock_profiling.cc:243] Waited 3.29 s on lock 0x7b54002f58d4. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e92242ae 00007f34ee161c27 00007f34e7849c04 00007f34e7849b92 00007f34e7849ac1 00007f34e7849a3c 00007f34e7848cf2 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:47:20.559257 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7568145664}
I20260809 15:47:20.567565 28586 raft_consensus.cc:2781] T d2d4f749fcfd4fb3bfb8115af3471598 P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260809 15:47:20.577122 23528 rpcz_store.cc:269] 0809 15:47:17.906607 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:17.938615 (+ 32008us) service_pool.cc:227] Handling call
0809 15:47:20.548714 (+2610099us) spinlock_profiling.cc:243] Waited 154 ms on lock 0x7b54001fb154. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e92242ae 00007f34ee161c27 00007f34e7849c04 00007f34e7849b92 00007f34e7849ac1 00007f34e7849a3c 00007f34e7848cf2 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:47:20.548859 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":353930624}
W20260809 15:47:20.568122 23521 rpcz_store.cc:269] 0809 15:47:17.347688 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:17.348003 (+ 315us) service_pool.cc:227] Handling call
0809 15:47:20.548232 (+3200229us) spinlock_profiling.cc:243] Waited 2.16 s on lock 0x7b54000f78d4. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e92242ae 00007f34ee161c27 00007f34e7849c04 00007f34e7849b92 00007f34e7849ac1 00007f34e7849a3c 00007f34e7848cf2 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:47:20.548369 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4958040448}
I20260809 15:47:20.593852 23519 raft_consensus.cc:1226] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 65c36d164b0540f099d592f38307ee6c [term 10 FOLLOWER]: Rejecting Update request from peer 61e69695258f48baa984c9d395e8ce3f for earlier term 9. Current term is 10. Ops: []
I20260809 15:47:20.550746 26966 raft_consensus.cc:3037] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Advancing to term 5
I20260809 15:47:20.548601 28546 leader_election.cc:288] T 8e4ffcf83c234ae7910ddef3ec0b4bda P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:20.555471 28285 raft_consensus.cc:3037] T 3924b648461e463b9d7925e5671d68ae P 65c36d164b0540f099d592f38307ee6c [term 10 FOLLOWER]: Advancing to term 11
I20260809 15:47:20.546627 28430 raft_consensus.cc:3032] T 7660dfa2ef694927b8eda4100a2bcd9b P 65c36d164b0540f099d592f38307ee6c [term 6 LEADER]: Stepping down as leader of term 6
I20260809 15:47:20.592849 23345 leader_election.cc:302] T 907bacfa79d04a55a68c8dbe60056462 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:47:20.599999 23519 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44018 (request call id 346) took 2656 ms. Trace:
I20260809 15:47:20.558845 28549 raft_consensus.cc:504] T 3296ef50be254fc29cfc77d22a8f7a23 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
W20260809 15:47:20.597067 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 91920.6us
I20260809 15:47:20.607311 23519 rpcz_store.cc:276] 0809 15:47:17.943393 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:20.504523 (+2561130us) service_pool.cc:227] Handling call
0809 15:47:20.585212 (+ 80689us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:47:20.599781 (+ 14569us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:47:20.599948 (+ 167us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:47:20.525235 28756 raft_consensus.cc:2726] T 3d880de657e544468c82157769622a26 P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260809 15:47:20.582557 28246 raft_consensus.cc:482] T 99bdb93503db4de4a1cfb0161ea0c07c P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Starting leader election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:47:20.547569 28532 consensus_queue.cc:1046] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: INVALID_TERM, Last received: 9.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.054s
I20260809 15:47:20.593147 28586 raft_consensus.cc:482] T d2d4f749fcfd4fb3bfb8115af3471598 P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:47:20.614897 28416 consensus_queue.cc:1046] T e14d118b2eaa42aa8a7cd4025aba7128 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260809 15:47:20.622505 23517 raft_consensus.cc:2454] T 472816994d814cb198b69e5248697440 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 7.
I20260809 15:47:20.600785 28302 raft_consensus.cc:2781] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 61e69695258f48baa984c9d395e8ce3f [term 7 FOLLOWER]: Leader election won for term 7
I20260809 15:47:20.618759 23283 raft_consensus.cc:2454] T 213543cdbe814c89a17a963469922c1b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 5.
I20260809 15:47:20.626432 23520 raft_consensus.cc:1226] T 472816994d814cb198b69e5248697440 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Rejecting Update request from peer c0f38c1908414566aa9115bb2cbdd85d for earlier term 6. Current term is 7. Ops: []
I20260809 15:47:20.549425 28560 raft_consensus.cc:504] T 0da4d5283c8a46878bd39b85e5c71f66 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:20.628255 23353 leader_election.cc:302] T 0c29937a80724ddb9d07a27534a146f3 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:47:20.632663 23228 leader_election.cc:302] T 4564dfc6d4094c1b9240b367ccecf3b1 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:47:20.632319 23400 raft_consensus.cc:1226] T 3296ef50be254fc29cfc77d22a8f7a23 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Rejecting Update request from peer 61e69695258f48baa984c9d395e8ce3f for earlier term 2. Current term is 3. Ops: []
I20260809 15:47:20.608170 28669 raft_consensus.cc:2726] T 16ca412a0dae4feebe85dd01bf1a9173 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260809 15:47:20.608709 28430 raft_consensus.cc:729] T 7660dfa2ef694927b8eda4100a2bcd9b P 65c36d164b0540f099d592f38307ee6c [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:47:20.637041 23518 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56766 (request call id 2186) took 2723 ms. Trace:
I20260809 15:47:20.638577 23518 rpcz_store.cc:276] 0809 15:47:17.910314 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:20.462710 (+2552396us) service_pool.cc:227] Handling call
0809 15:47:20.571814 (+109104us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:47:20.628806 (+ 56992us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:47:20.628833 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 15:47:20.633305 (+ 4472us) raft_consensus.cc:1631] Marking committed up to 6
0809 15:47:20.633345 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:47:20.633401 (+ 56us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:47:20.633501 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:47:20.623483 28586 raft_consensus.cc:3037] T d2d4f749fcfd4fb3bfb8115af3471598 P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Advancing to term 9
I20260809 15:47:20.615773 28246 raft_consensus.cc:3037] T 99bdb93503db4de4a1cfb0161ea0c07c P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Advancing to term 5
I20260809 15:47:20.589787 28597 raft_consensus.cc:504] T 297c82ed906d4b48b184985eed3de601 P 65c36d164b0540f099d592f38307ee6c [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:20.643728 23470 leader_election.cc:302] T 998c6cee72cf40228c6dba064c0db2da P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:20.648476 28758 raft_consensus.cc:3032] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 61e69695258f48baa984c9d395e8ce3f [term 4 LEADER]: Stepping down as leader of term 4
I20260809 15:47:20.648808 28309 raft_consensus.cc:504] T c5c2e425a74b4e64ae70b5def750434c P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:20.649380 28248 raft_consensus.cc:2726] T f7715d71dbac447b9351fc5eb36c5e80 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260809 15:47:20.649996 28745 consensus_queue.cc:237] T e14d118b2eaa42aa8a7cd4025aba7128 P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:20.628265 28612 raft_consensus.cc:2741] T 213543cdbe814c89a17a963469922c1b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
W20260809 15:47:20.631764 28850 log.cc:924] Time spent T 3d880de657e544468c82157769622a26 P 61e69695258f48baa984c9d395e8ce3f: Append to log took a long time: real 0.250s user 0.000s sys 0.001s
I20260809 15:47:20.655530 27957 raft_consensus.cc:2726] T 0c29937a80724ddb9d07a27534a146f3 P c0f38c1908414566aa9115bb2cbdd85d [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260809 15:47:20.656172 28319 raft_consensus.cc:504] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:20.656679 28532 consensus_queue.cc:1046] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: INVALID_TERM, Last received: 9.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260809 15:47:20.658083 28319 leader_election.cc:288] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 election: Requested vote from peers 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:20.636644 28546 raft_consensus.cc:2741] T 907bacfa79d04a55a68c8dbe60056462 P c0f38c1908414566aa9115bb2cbdd85d [term 9 FOLLOWER]: Leader pre-election decision vote started in defunct term 8: won
I20260809 15:47:20.658985 28430 consensus_queue.cc:260] T 7660dfa2ef694927b8eda4100a2bcd9b P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:20.660495 28549 leader_election.cc:288] T 3296ef50be254fc29cfc77d22a8f7a23 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:20.661260 28430 raft_consensus.cc:3037] T 7660dfa2ef694927b8eda4100a2bcd9b P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Advancing to term 7
I20260809 15:47:20.663779 23520 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56766 (request call id 2198) took 1828 ms. Trace:
I20260809 15:47:20.664484 28597 leader_election.cc:288] T 297c82ed906d4b48b184985eed3de601 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 10 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:47:20.666154 23228 leader_election.cc:302] T 28e17dcb5efe45b8b6a8c42da03b4819 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:47:20.667042 23228 leader_election.cc:302] T 998c6cee72cf40228c6dba064c0db2da P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:47:20.667369 23404 raft_consensus.cc:1226] T c5c2e425a74b4e64ae70b5def750434c P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20260809 15:47:20.668292 28309 leader_election.cc:288] T c5c2e425a74b4e64ae70b5def750434c P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 8 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:20.668571 23228 leader_election.cc:302] T 0da4d5283c8a46878bd39b85e5c71f66 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:47:20.668844 28645 raft_consensus.cc:2781] T 4564dfc6d4094c1b9240b367ccecf3b1 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Leader election won for term 8
I20260809 15:47:20.669123 28714 raft_consensus.cc:2781] T 28e17dcb5efe45b8b6a8c42da03b4819 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260809 15:47:20.669538 28714 raft_consensus.cc:482] T 28e17dcb5efe45b8b6a8c42da03b4819 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Starting leader election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:47:20.669950 28714 raft_consensus.cc:3037] T 28e17dcb5efe45b8b6a8c42da03b4819 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Advancing to term 9
I20260809 15:47:20.670572 28645 raft_consensus.cc:686] T 4564dfc6d4094c1b9240b367ccecf3b1 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 LEADER]: Becoming Leader. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Running, Role: LEADER
I20260809 15:47:20.671451 28645 consensus_queue.cc:237] T 4564dfc6d4094c1b9240b367ccecf3b1 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:20.672096 23353 leader_election.cc:302] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters: 61e69695258f48baa984c9d395e8ce3f
I20260809 15:47:20.672997 28309 raft_consensus.cc:2781] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d [term 10 FOLLOWER]: Leader election won for term 10
I20260809 15:47:20.674429 28309 raft_consensus.cc:686] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d [term 10 LEADER]: Becoming Leader. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:47:20.661590 28775 raft_consensus.cc:3032] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 61e69695258f48baa984c9d395e8ce3f [term 9 LEADER]: Stepping down as leader of term 9
I20260809 15:47:20.675164 28775 raft_consensus.cc:729] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 61e69695258f48baa984c9d395e8ce3f [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 61e69695258f48baa984c9d395e8ce3f, State: Running, Role: LEADER
I20260809 15:47:20.675359 23527 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56766 (request call id 2179) took 2806 ms. Trace:
I20260809 15:47:20.676038 28718 leader_election.cc:288] T cee02fff2c9c48a19f43eb358a89e7b0 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:20.648847 28758 raft_consensus.cc:729] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 61e69695258f48baa984c9d395e8ce3f [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 61e69695258f48baa984c9d395e8ce3f, State: Running, Role: LEADER
I20260809 15:47:20.675525 28309 consensus_queue.cc:237] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:20.677208 28758 consensus_queue.cc:260] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 61e69695258f48baa984c9d395e8ce3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:20.678409 28758 raft_consensus.cc:3037] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Advancing to term 5
I20260809 15:47:20.681180 23518 raft_consensus.cc:1261] T d3e94b7e50754a81b3d72a07476cb7b9 P 65c36d164b0540f099d592f38307ee6c [term 9 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260809 15:47:20.682250 23518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5c2e425a74b4e64ae70b5def750434c" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:47:20.624963 28674 raft_consensus.cc:482] T cee02fff2c9c48a19f43eb358a89e7b0 P 61e69695258f48baa984c9d395e8ce3f [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:20.683501 28674 raft_consensus.cc:504] T cee02fff2c9c48a19f43eb358a89e7b0 P 61e69695258f48baa984c9d395e8ce3f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:20.684183 28197 raft_consensus.cc:2726] T 998c6cee72cf40228c6dba064c0db2da P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260809 15:47:20.643476 28302 raft_consensus.cc:686] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 61e69695258f48baa984c9d395e8ce3f [term 7 LEADER]: Becoming Leader. State: Replica: 61e69695258f48baa984c9d395e8ce3f, State: Running, Role: LEADER
I20260809 15:47:20.685532 28674 leader_election.cc:288] T cee02fff2c9c48a19f43eb358a89e7b0 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:20.687732 28447 raft_consensus.cc:504] T 4bcac4b24e7a4448abdc934bba5a624e P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:20.688314 23528 raft_consensus.cc:1261] T a88421dba80d4d3388aa5eb62e23853b P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260809 15:47:20.693712 23528 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56766 (request call id 2204) took 1854 ms. Trace:
W20260809 15:47:20.683868 23470 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 49686.4us
I20260809 15:47:20.664119 23520 rpcz_store.cc:276] 0809 15:47:18.834926 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:20.597437 (+1762511us) service_pool.cc:227] Handling call
0809 15:47:20.623727 (+ 26290us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:47:20.626835 (+ 3108us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:47:20.663610 (+ 36775us) spinlock_profiling.cc:243] Waited 7.8 ms on lock 0x7b54001df6d0. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e921fa09 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:47:20.663728 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":17938816}
I20260809 15:47:20.694020 23528 rpcz_store.cc:276] 0809 15:47:18.839770 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:20.604543 (+1764773us) service_pool.cc:227] Handling call
0809 15:47:20.632834 (+ 28291us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:47:20.693498 (+ 60664us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:47:20.693664 (+ 166us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:47:20.656257 24410 leader_election.cc:302] T 472816994d814cb198b69e5248697440 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c; no voters: c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:20.689234 28627 raft_consensus.cc:504] T d1a134ba33ea4076bbaac0ae460c17ba P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:20.688477 23527 rpcz_store.cc:276] 0809 15:47:17.798774 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:18.679707 (+880933us) service_pool.cc:227] Handling call
0809 15:47:19.029299 (+349592us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:47:19.037152 (+ 7853us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:47:19.037182 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 15:47:19.037222 (+ 40us) raft_consensus.cc:1631] Marking committed up to 3
0809 15:47:19.037256 (+ 34us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:47:19.037312 (+ 56us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:47:20.595705 (+1558393us) spinlock_profiling.cc:243] Waited 349 ms on lock 0x7b540038de50. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e921fa09 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:47:20.605363 (+ 9658us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7534,"raft.queue_time_us":7954,"raft.run_cpu_time_us":120,"raft.run_wall_time_us":27780,"spinlock_wait_cycles":803368832}
I20260809 15:47:20.696611 23393 raft_consensus.cc:2454] T 2424fd9bd7f14e508fd106cbfda786a4 P c0f38c1908414566aa9115bb2cbdd85d [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 9.
I20260809 15:47:20.710199 28674 raft_consensus.cc:2781] T 472816994d814cb198b69e5248697440 P 61e69695258f48baa984c9d395e8ce3f [term 7 FOLLOWER]: Leader election won for term 7
I20260809 15:47:20.710731 28674 raft_consensus.cc:686] T 472816994d814cb198b69e5248697440 P 61e69695258f48baa984c9d395e8ce3f [term 7 LEADER]: Becoming Leader. State: Replica: 61e69695258f48baa984c9d395e8ce3f, State: Running, Role: LEADER
I20260809 15:47:20.710355 28775 consensus_queue.cc:260] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 61e69695258f48baa984c9d395e8ce3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:20.711781 28775 raft_consensus.cc:3037] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 61e69695258f48baa984c9d395e8ce3f [term 9 FOLLOWER]: Advancing to term 10
I20260809 15:47:20.711573 28674 consensus_queue.cc:237] T 472816994d814cb198b69e5248697440 P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:20.714043 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee02fff2c9c48a19f43eb358a89e7b0" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:20.714607 23527 raft_consensus.cc:2454] T cee02fff2c9c48a19f43eb358a89e7b0 P 65c36d164b0540f099d592f38307ee6c [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 9.
I20260809 15:47:20.709888 28621 raft_consensus.cc:504] T c0741ab44d99439abc3aa3bdd27ee160 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:20.718302 28621 leader_election.cc:288] T c0741ab44d99439abc3aa3bdd27ee160 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 9 election: Requested vote from peers 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:20.719065 24471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3296ef50be254fc29cfc77d22a8f7a23" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f"
I20260809 15:47:20.719733 23404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "297c82ed906d4b48b184985eed3de601" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:47:20.726325 28474 consensus_queue.cc:1046] T c5c2e425a74b4e64ae70b5def750434c P 65c36d164b0540f099d592f38307ee6c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20260809 15:47:20.727079 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 58561.8us
I20260809 15:47:20.727670 28621 raft_consensus.cc:2781] T 0da4d5283c8a46878bd39b85e5c71f66 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260809 15:47:20.671406 21289 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 15:47:20.764194 23472 leader_election.cc:302] T 2424fd9bd7f14e508fd106cbfda786a4 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:47:20.762183 28335 consensus_queue.cc:1046] T 3296ef50be254fc29cfc77d22a8f7a23 P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.003s
I20260809 15:47:20.765583 28621 raft_consensus.cc:482] T 0da4d5283c8a46878bd39b85e5c71f66 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:47:20.765985 28621 raft_consensus.cc:3037] T 0da4d5283c8a46878bd39b85e5c71f66 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Advancing to term 7
I20260809 15:47:20.766002 28335 raft_consensus.cc:3032] T 3296ef50be254fc29cfc77d22a8f7a23 P 61e69695258f48baa984c9d395e8ce3f [term 2 LEADER]: Stepping down as leader of term 2
I20260809 15:47:20.767097 28335 raft_consensus.cc:729] T 3296ef50be254fc29cfc77d22a8f7a23 P 61e69695258f48baa984c9d395e8ce3f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 61e69695258f48baa984c9d395e8ce3f, State: Running, Role: LEADER
I20260809 15:47:20.765808 23228 leader_election.cc:302] T 00a1881267a14ba8b78c81040f17a014 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters: c0f38c1908414566aa9115bb2cbdd85d
W20260809 15:47:20.776054 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 130687us
I20260809 15:47:20.669450 28538 raft_consensus.cc:2781] T 998c6cee72cf40228c6dba064c0db2da P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260809 15:47:20.624625 28521 consensus_queue.cc:1046] T d0721a4403a440f383318a3cea9d1442 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.074s
I20260809 15:47:20.766722 21289 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 15:47:20.754139 23519 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56766 (request call id 2209) took 1625 ms. Trace:
I20260809 15:47:20.755214 23404 raft_consensus.cc:3037] T 297c82ed906d4b48b184985eed3de601 P c0f38c1908414566aa9115bb2cbdd85d [term 9 FOLLOWER]: Advancing to term 10
I20260809 15:47:20.757114 24410 leader_election.cc:302] T cee02fff2c9c48a19f43eb358a89e7b0 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c; no voters:
I20260809 15:47:20.760330 23470 leader_election.cc:302] T cef3e7277d7d4760824a3436ed4a847d P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters: c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:20.776273 23285 raft_consensus.cc:2396] T 4b30f28311964a0b9149a50495e83d8f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20260809 15:47:20.728528 23395 raft_consensus.cc:1226] T 4bcac4b24e7a4448abdc934bba5a624e P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Rejecting Update request from peer 61e69695258f48baa984c9d395e8ce3f for earlier term 4. Current term is 5. Ops: []
I20260809 15:47:20.775859 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee02fff2c9c48a19f43eb358a89e7b0" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:47:20.780864 23519 rpcz_store.cc:276] 0809 15:47:19.128494 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:20.627782 (+1499288us) service_pool.cc:227] Handling call
0809 15:47:20.726154 (+ 98372us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:47:20.753868 (+ 27714us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:47:20.753892 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 15:47:20.753930 (+ 38us) raft_consensus.cc:1631] Marking committed up to 5
0809 15:47:20.753958 (+ 28us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:47:20.754006 (+ 48us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:47:20.754093 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:47:20.783249 23517 raft_consensus.cc:1261] T 538a228ddfb54bc6a36528d5ff07597b P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260809 15:47:20.782343 24472 raft_consensus.cc:2454] T cee02fff2c9c48a19f43eb358a89e7b0 P 61e69695258f48baa984c9d395e8ce3f [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 9.
I20260809 15:47:20.778201 28538 raft_consensus.cc:482] T 998c6cee72cf40228c6dba064c0db2da P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Starting leader election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:47:20.794533 28538 raft_consensus.cc:3037] T 998c6cee72cf40228c6dba064c0db2da P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Advancing to term 9
I20260809 15:47:20.659018 28522 raft_consensus.cc:504] T 6ae14a74a26f434db08c652e097385c8 P c0f38c1908414566aa9115bb2cbdd85d [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:20.718189 28714 raft_consensus.cc:504] T 28e17dcb5efe45b8b6a8c42da03b4819 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:20.800170 23400 raft_consensus.cc:1261] T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d [term 10 FOLLOWER]: Refusing update from remote peer 61e69695258f48baa984c9d395e8ce3f: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20260809 15:47:20.786998 26966 raft_consensus.cc:3032] T c5c2e425a74b4e64ae70b5def750434c P 65c36d164b0540f099d592f38307ee6c [term 7 LEADER]: Stepping down as leader of term 7
I20260809 15:47:20.783147 28319 raft_consensus.cc:2781] T 00a1881267a14ba8b78c81040f17a014 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260809 15:47:20.814985 23223 leader_election.cc:302] T cee02fff2c9c48a19f43eb358a89e7b0 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:47:20.792344 28725 raft_consensus.cc:2781] T 2424fd9bd7f14e508fd106cbfda786a4 P 65c36d164b0540f099d592f38307ee6c [term 9 FOLLOWER]: Leader election won for term 9
W20260809 15:47:20.829255 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 63528us
I20260809 15:47:20.829989 23228 leader_election.cc:302] T ba56725e42f74a1baa4c04d9660727f9 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:47:20.685632 28302 consensus_queue.cc:237] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:20.797695 24451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5712f6b6fffc4cc5a8ddfbe63e4097df" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f"
I20260809 15:47:20.775873 28425 raft_consensus.cc:504] T 0acc47ce21c3422698d1ef7277c3ed63 P 65c36d164b0540f099d592f38307ee6c [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:20.803764 28280 raft_consensus.cc:2781] T cee02fff2c9c48a19f43eb358a89e7b0 P 61e69695258f48baa984c9d395e8ce3f [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260809 15:47:20.840052 23400 raft_consensus.cc:1226] T c5c2e425a74b4e64ae70b5def750434c P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 7. Current term is 8. Ops: [7.5-7.5]
W20260809 15:47:20.843155 23470 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 82914us
I20260809 15:47:20.845422 28612 raft_consensus.cc:2781] T ba56725e42f74a1baa4c04d9660727f9 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Leader election won for term 7
I20260809 15:47:20.846037 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bcac4b24e7a4448abdc934bba5a624e" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f"
I20260809 15:47:20.846055 28612 raft_consensus.cc:686] T ba56725e42f74a1baa4c04d9660727f9 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 LEADER]: Becoming Leader. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Running, Role: LEADER
I20260809 15:47:20.810231 26966 raft_consensus.cc:729] T c5c2e425a74b4e64ae70b5def750434c P 65c36d164b0540f099d592f38307ee6c [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:47:20.828338 28319 raft_consensus.cc:482] T 00a1881267a14ba8b78c81040f17a014 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 9 FOLLOWER]: Starting leader election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:47:20.852411 23228 leader_election.cc:302] T d2d4f749fcfd4fb3bfb8115af3471598 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:47:20.828586 28406 raft_consensus.cc:3032] T e14d118b2eaa42aa8a7cd4025aba7128 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 LEADER]: Stepping down as leader of term 8
I20260809 15:47:20.813292 28549 consensus_queue.cc:1046] T 472816994d814cb198b69e5248697440 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.041s
I20260809 15:47:20.865201 23404 raft_consensus.cc:2454] T 297c82ed906d4b48b184985eed3de601 P c0f38c1908414566aa9115bb2cbdd85d [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 10.
I20260809 15:47:20.808243 28802 consensus_queue.cc:1046] T 4bcac4b24e7a4448abdc934bba5a624e P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: INVALID_TERM, Last received: 4.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.041s
I20260809 15:47:20.840545 28416 raft_consensus.cc:2781] T cee02fff2c9c48a19f43eb358a89e7b0 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260809 15:47:20.840705 28774 raft_consensus.cc:482] T f9e7cadcd24d45fcbfa84209225e8f2d P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:20.851508 28719 raft_consensus.cc:2781] T cef3e7277d7d4760824a3436ed4a847d P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260809 15:47:20.842653 28280 raft_consensus.cc:482] T cee02fff2c9c48a19f43eb358a89e7b0 P 61e69695258f48baa984c9d395e8ce3f [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:47:20.403435 24455 rpcz_store.cc:276] 0809 15:47:17.649485 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:17.761983 (+112498us) service_pool.cc:227] Handling call
0809 15:47:17.798051 (+ 36068us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:47:17.844648 (+ 46597us) raft_consensus.cc:1520] Early marking committed up to index 6
0809 15:47:17.899129 (+ 54481us) log.cc:842] Serialized 14 byte log entry
0809 15:47:20.109916 (+2210787us) spinlock_profiling.cc:243] Waited 7.06 ms on lock 0x7b18003ce020. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34ef1978b1 00007f34ef1a4661 00007f34e62da345 00007f34ef111a39 00007f34e62f78bb 00007f34e62f5cbf 00007f34e62f7e22 00007f34e8ea15bb 00007f34e8ea7063 00007f34e8ea757b 00007f34e922a88a
0809 15:47:20.251814 (+141898us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 15:47:20.257271 (+ 5457us) raft_consensus.cc:1631] Marking committed up to 6
0809 15:47:20.257354 (+ 83us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:47:20.257468 (+ 114us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:47:20.280911 (+ 23443us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":31535,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":44,"raft.queue_time_us":5848,"raft.run_cpu_time_us":102,"raft.run_wall_time_us":103,"spinlock_wait_cycles":16235904,"thread_start_us":114441,"threads_started":1,"wal-append.queue_time_us":2418319}
I20260809 15:47:20.848654 28447 leader_election.cc:288] T 4bcac4b24e7a4448abdc934bba5a624e P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 election: Requested vote from peers 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:20.857302 28406 raft_consensus.cc:729] T e14d118b2eaa42aa8a7cd4025aba7128 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Running, Role: LEADER
I20260809 15:47:20.871421 23472 leader_election.cc:302] T 297c82ed906d4b48b184985eed3de601 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:47:20.871305 28718 raft_consensus.cc:2781] T d2d4f749fcfd4fb3bfb8115af3471598 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260809 15:47:20.848438 28560 leader_election.cc:288] T 0da4d5283c8a46878bd39b85e5c71f66 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 7 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:20.847298 28612 consensus_queue.cc:237] T ba56725e42f74a1baa4c04d9660727f9 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } attrs { replace: true } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } }
W20260809 15:47:20.868106 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 91397.2us
I20260809 15:47:20.857048 28319 raft_consensus.cc:3037] T 00a1881267a14ba8b78c81040f17a014 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 9 FOLLOWER]: Advancing to term 10
I20260809 15:47:20.878221 28549 consensus_queue.cc:1035] T 538a228ddfb54bc6a36528d5ff07597b P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.001s
I20260809 15:47:20.879041 28353 raft_consensus.cc:3032] T 472816994d814cb198b69e5248697440 P c0f38c1908414566aa9115bb2cbdd85d [term 6 LEADER]: Stepping down as leader of term 6
I20260809 15:47:20.878237 28546 raft_consensus.cc:3032] T d0721a4403a440f383318a3cea9d1442 P c0f38c1908414566aa9115bb2cbdd85d [term 8 LEADER]: Stepping down as leader of term 8
I20260809 15:47:20.856511 27957 consensus_queue.cc:1035] T a88421dba80d4d3388aa5eb62e23853b P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.023s
I20260809 15:47:20.879576 24463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0741ab44d99439abc3aa3bdd27ee160" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f"
I20260809 15:47:20.879645 23522 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44018 (request call id 345) took 2943 ms. Trace:
I20260809 15:47:20.880407 23518 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.BulkChangeConfig from 127.0.0.1:44010 (request call id 8) took 2262 ms. Trace:
I20260809 15:47:20.880906 23517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bcac4b24e7a4448abdc934bba5a624e" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:47:20.856004 28725 raft_consensus.cc:686] T 2424fd9bd7f14e508fd106cbfda786a4 P 65c36d164b0540f099d592f38307ee6c [term 9 LEADER]: Becoming Leader. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:47:20.884639 23470 leader_election.cc:302] T e14d118b2eaa42aa8a7cd4025aba7128 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 61e69695258f48baa984c9d395e8ce3f, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:47:20.887451 23519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0da4d5283c8a46878bd39b85e5c71f66" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:47:20.847885 28688 consensus_queue.cc:1035] T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.103s
I20260809 15:47:20.871572 23526 raft_consensus.cc:1226] T 0acc47ce21c3422698d1ef7277c3ed63 P 65c36d164b0540f099d592f38307ee6c [term 10 FOLLOWER]: Rejecting Update request from peer c0f38c1908414566aa9115bb2cbdd85d for earlier term 9. Current term is 10. Ops: []
I20260809 15:47:20.855518 28532 consensus_queue.cc:1035] T 3924b648461e463b9d7925e5671d68ae P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.026s
I20260809 15:47:20.873262 28229 raft_consensus.cc:3037] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Advancing to term 6
I20260809 15:47:20.871881 28416 raft_consensus.cc:482] T cee02fff2c9c48a19f43eb358a89e7b0 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:47:20.850111 28285 raft_consensus.cc:504] T 3924b648461e463b9d7925e5671d68ae P 65c36d164b0540f099d592f38307ee6c [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:20.899397 23521 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44018 (request call id 356) took 1892 ms. Trace:
I20260809 15:47:20.885510 28280 raft_consensus.cc:3037] T cee02fff2c9c48a19f43eb358a89e7b0 P 61e69695258f48baa984c9d395e8ce3f [term 9 FOLLOWER]: Advancing to term 10
I20260809 15:47:20.882519 28719 raft_consensus.cc:482] T cef3e7277d7d4760824a3436ed4a847d P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Starting leader election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:47:20.891779 28669 raft_consensus.cc:2781] T 297c82ed906d4b48b184985eed3de601 P 65c36d164b0540f099d592f38307ee6c [term 10 FOLLOWER]: Leader election won for term 10
I20260809 15:47:20.945328 28669 raft_consensus.cc:686] T 297c82ed906d4b48b184985eed3de601 P 65c36d164b0540f099d592f38307ee6c [term 10 LEADER]: Becoming Leader. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:47:20.946230 28669 consensus_queue.cc:237] T 297c82ed906d4b48b184985eed3de601 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:20.902827 24463 raft_consensus.cc:3037] T c0741ab44d99439abc3aa3bdd27ee160 P 61e69695258f48baa984c9d395e8ce3f [term 8 FOLLOWER]: Advancing to term 9
I20260809 15:47:20.947353 28246 raft_consensus.cc:504] T 99bdb93503db4de4a1cfb0161ea0c07c P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:20.948724 28285 leader_election.cc:288] T 3924b648461e463b9d7925e5671d68ae P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 11 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:20.949460 28246 leader_election.cc:288] T 99bdb93503db4de4a1cfb0161ea0c07c P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 5 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:47:20.932008 23521 rpcz_store.cc:276] 0809 15:47:19.006826 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:20.602937 (+1596111us) service_pool.cc:227] Handling call
0809 15:47:20.608519 (+ 5582us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:47:20.615425 (+ 6906us) raft_consensus.cc:1520] Early marking committed up to index 7
0809 15:47:20.624512 (+ 9087us) log.cc:842] Serialized 14 byte log entry
0809 15:47:20.899104 (+274592us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 15:47:20.899148 (+ 44us) raft_consensus.cc:1631] Marking committed up to 7
0809 15:47:20.899183 (+ 35us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:47:20.899238 (+ 55us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:47:20.899348 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2072,"prepare.queue_time_us":3804,"prepare.run_cpu_time_us":62,"prepare.run_wall_time_us":6876,"raft.queue_time_us":2074,"raft.run_cpu_time_us":98,"raft.run_wall_time_us":20728,"thread_start_us":266956,"threads_started":1}
I20260809 15:47:20.868281 28525 consensus_queue.cc:1046] T c5c2e425a74b4e64ae70b5def750434c P 65c36d164b0540f099d592f38307ee6c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.030s
I20260809 15:47:20.953130 26966 consensus_queue.cc:260] T c5c2e425a74b4e64ae70b5def750434c P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:20.953279 28586 raft_consensus.cc:504] T d2d4f749fcfd4fb3bfb8115af3471598 P 65c36d164b0540f099d592f38307ee6c [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:20.954356 26966 raft_consensus.cc:3037] T c5c2e425a74b4e64ae70b5def750434c P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Advancing to term 8
I20260809 15:47:20.921886 23404 raft_consensus.cc:1226] T f7715d71dbac447b9351fc5eb36c5e80 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 6. Current term is 7. Ops: []
I20260809 15:47:20.903170 23518 rpcz_store.cc:276] 0809 15:47:18.610798 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:20.777373 (+2166575us) service_pool.cc:227] Handling call
0809 15:47:20.873499 (+ 96126us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 15:47:20.922521 23519 raft_consensus.cc:3037] T 0da4d5283c8a46878bd39b85e5c71f66 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Advancing to term 7
W20260809 15:47:20.960971 24410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 152404us
I20260809 15:47:20.902781 23522 rpcz_store.cc:276] 0809 15:47:17.936634 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:20.465200 (+2528566us) service_pool.cc:227] Handling call
0809 15:47:20.465545 (+ 345us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 15:47:20.465821 (+ 276us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:47:20.465843 (+ 22us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 15:47:20.466222 (+ 379us) log.cc:842] Serialized 26 byte log entry
0809 15:47:20.606144 (+139922us) raft_consensus.cc:1631] Marking committed up to 6
0809 15:47:20.606175 (+ 31us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:47:20.606231 (+ 56us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 15:47:20.879421 (+273190us) raft_consensus.cc:1671] finished
0809 15:47:20.879438 (+ 17us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:47:20.879575 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":139603,"threads_started":1,"wal-append.queue_time_us":347804}
I20260809 15:47:20.840986 28335 consensus_queue.cc:260] T 3296ef50be254fc29cfc77d22a8f7a23 P 61e69695258f48baa984c9d395e8ce3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:20.932427 23353 leader_election.cc:302] T 4bcac4b24e7a4448abdc934bba5a624e P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c
I20260809 15:47:20.900635 28627 leader_election.cc:288] T d1a134ba33ea4076bbaac0ae460c17ba P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 8 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:47:20.927590 23400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1a134ba33ea4076bbaac0ae460c17ba" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:47:20.967183 23400 raft_consensus.cc:3037] T d1a134ba33ea4076bbaac0ae460c17ba P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Advancing to term 8
I20260809 15:47:20.935155 23526 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56766 (request call id 2192) took 2952 ms. Trace:
I20260809 15:47:20.968083 23526 rpcz_store.cc:276] 0809 15:47:17.982979 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:20.589615 (+2606636us) service_pool.cc:227] Handling call
0809 15:47:20.628595 (+ 38980us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:47:20.928948 (+300353us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:47:20.933190 (+ 4242us) spinlock_profiling.cc:243] Waited 243 ms on lock 0x7b54000767d4. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e9220f51 00007f34e921f99e 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772 00007f34e764a94c 00007f34e7649d86
0809 15:47:20.935107 (+ 1917us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":558382336}
I20260809 15:47:20.935353 24453 raft_consensus.cc:1226] T 472816994d814cb198b69e5248697440 P 61e69695258f48baa984c9d395e8ce3f [term 7 LEADER]: Rejecting Update request from peer c0f38c1908414566aa9115bb2cbdd85d for earlier term 6. Current term is 7. Ops: []
I20260809 15:47:20.897719 28718 raft_consensus.cc:482] T d2d4f749fcfd4fb3bfb8115af3471598 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:47:20.900467 28353 raft_consensus.cc:729] T 472816994d814cb198b69e5248697440 P c0f38c1908414566aa9115bb2cbdd85d [term 6 LEADER]: Becoming Follower/Learner. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:47:20.931666 28430 raft_consensus.cc:2726] T e14d118b2eaa42aa8a7cd4025aba7128 P 65c36d164b0540f099d592f38307ee6c [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260809 15:47:20.942471 23395 raft_consensus.cc:1261] T 30c1740ef6874df0a332fbd6fd4b03ce P c0f38c1908414566aa9115bb2cbdd85d [term 9 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260809 15:47:20.900413 28784 consensus_queue.cc:1046] T 3d880de657e544468c82157769622a26 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.038s
W20260809 15:47:20.943655 23470 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 82618.5us
I20260809 15:47:20.902477 28546 raft_consensus.cc:729] T d0721a4403a440f383318a3cea9d1442 P c0f38c1908414566aa9115bb2cbdd85d [term 8 LEADER]: Becoming Follower/Learner. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:47:20.955648 23527 raft_consensus.cc:1226] T 3924b648461e463b9d7925e5671d68ae P 65c36d164b0540f099d592f38307ee6c [term 11 FOLLOWER]: Rejecting Update request from peer 61e69695258f48baa984c9d395e8ce3f for earlier term 10. Current term is 11. Ops: []
I20260809 15:47:20.931550 28416 raft_consensus.cc:3037] T cee02fff2c9c48a19f43eb358a89e7b0 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 9 FOLLOWER]: Advancing to term 10
I20260809 15:47:20.934587 28714 leader_election.cc:288] T 28e17dcb5efe45b8b6a8c42da03b4819 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 9 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
W20260809 15:47:20.903184 28853 log.cc:924] Time spent T 3924b648461e463b9d7925e5671d68ae P 61e69695258f48baa984c9d395e8ce3f: Append to log took a long time: real 0.211s user 0.001s sys 0.001s
I20260809 15:47:20.955892 23528 raft_consensus.cc:1226] T d2d4f749fcfd4fb3bfb8115af3471598 P 65c36d164b0540f099d592f38307ee6c [term 9 FOLLOWER]: Rejecting Update request from peer c0f38c1908414566aa9115bb2cbdd85d for earlier term 8. Current term is 9. Ops: [8.5-8.5]
I20260809 15:47:20.879402 23117 catalog_manager.cc:5495] T 953eea268e17491ca76f3a555860a512 P c0f38c1908414566aa9115bb2cbdd85d reported cstate change: term changed from 5 to 6, leader changed from 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65) to c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66). New cstate: current_term: 6 leader_uuid: "c0f38c1908414566aa9115bb2cbdd85d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } health_report { overall_health: UNKNOWN } } }
I20260809 15:47:20.887992 28774 raft_consensus.cc:504] T f9e7cadcd24d45fcbfa84209225e8f2d P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:20.964020 23351 leader_election.cc:302] T 0a51c0baaa6e432997c9d3552af4f254 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 61e69695258f48baa984c9d395e8ce3f, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:47:20.970566 23396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3924b648461e463b9d7925e5671d68ae" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:47:20.993919 23396 raft_consensus.cc:3037] T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d [term 10 FOLLOWER]: Advancing to term 11
I20260809 15:47:20.973698 24463 raft_consensus.cc:2454] T c0741ab44d99439abc3aa3bdd27ee160 P 61e69695258f48baa984c9d395e8ce3f [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 9.
I20260809 15:47:20.980671 23285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e71a05ba1d4d42bd95c3f85f9313399e" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:20.995035 23395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99bdb93503db4de4a1cfb0161ea0c07c" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:47:20.998167 23285 raft_consensus.cc:2454] T e71a05ba1d4d42bd95c3f85f9313399e P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 7.
I20260809 15:47:20.977654 28718 raft_consensus.cc:3037] T d2d4f749fcfd4fb3bfb8115af3471598 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Advancing to term 9
I20260809 15:47:20.985373 23529 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44018 (request call id 347) took 3034 ms. Trace:
I20260809 15:47:20.959381 28719 raft_consensus.cc:3037] T cef3e7277d7d4760824a3436ed4a847d P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Advancing to term 9
I20260809 15:47:20.955451 23520 raft_consensus.cc:1226] T 99bdb93503db4de4a1cfb0161ea0c07c P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Rejecting Update request from peer c0f38c1908414566aa9115bb2cbdd85d for earlier term 4. Current term is 5. Ops: []
I20260809 15:47:20.960599 28425 leader_election.cc:288] T 0acc47ce21c3422698d1ef7277c3ed63 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 10 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
W20260809 15:47:20.971108 28862 log.cc:924] Time spent T d3e94b7e50754a81b3d72a07476cb7b9 P 61e69695258f48baa984c9d395e8ce3f: Append to log took a long time: real 0.120s user 0.001s sys 0.000s
W20260809 15:47:21.011852 23116 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:56216 (request call id 253) took 22189 ms (client timeout 15000 ms). Trace:
I20260809 15:47:21.007076 23117 catalog_manager.cc:5495] T 05ad7d8fdca64e3da527280ca5be9280 P c0f38c1908414566aa9115bb2cbdd85d reported cstate change: term changed from 2 to 5, leader changed from c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66) to 65c36d164b0540f099d592f38307ee6c (127.20.202.67). New cstate: current_term: 5 leader_uuid: "65c36d164b0540f099d592f38307ee6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } }
W20260809 15:47:21.013967 28859 log.cc:924] Time spent T f9e7cadcd24d45fcbfa84209225e8f2d P 65c36d164b0540f099d592f38307ee6c: Append to log took a long time: real 0.053s user 0.001s sys 0.000s
I20260809 15:47:21.014161 23117 catalog_manager.cc:5495] T d2d4f749fcfd4fb3bfb8115af3471598 P c0f38c1908414566aa9115bb2cbdd85d reported cstate change: term changed from 4 to 5, leader changed from 65c36d164b0540f099d592f38307ee6c (127.20.202.67) to c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66). New cstate: current_term: 5 leader_uuid: "c0f38c1908414566aa9115bb2cbdd85d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } health_report { overall_health: UNKNOWN } } }
W20260809 15:47:21.015767 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 81432.1us
I20260809 15:47:20.959553 28406 consensus_queue.cc:260] T e14d118b2eaa42aa8a7cd4025aba7128 P 67b3aeadfa7a432abe8af4d292e04ab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:21.017418 28309 raft_consensus.cc:2726] T 0a51c0baaa6e432997c9d3552af4f254 P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260809 15:47:21.018689 23404 raft_consensus.cc:1261] T f9e7cadcd24d45fcbfa84209225e8f2d P c0f38c1908414566aa9115bb2cbdd85d [term 9 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20260809 15:47:20.955183 28586 leader_election.cc:288] T d2d4f749fcfd4fb3bfb8115af3471598 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 9 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:47:20.963343 28319 raft_consensus.cc:504] T 00a1881267a14ba8b78c81040f17a014 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:20.960587 24455 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3924b648461e463b9d7925e5671d68ae" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f"
I20260809 15:47:20.994441 28538 raft_consensus.cc:504] T 998c6cee72cf40228c6dba064c0db2da P 67b3aeadfa7a432abe8af4d292e04ab9 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:21.029330 28538 leader_election.cc:288] T 998c6cee72cf40228c6dba064c0db2da P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 9 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:21.029141 23117 catalog_manager.cc:5495] T 00a1881267a14ba8b78c81040f17a014 P c0f38c1908414566aa9115bb2cbdd85d reported cstate change: term changed from 3 to 7. New cstate: current_term: 7 leader_uuid: "c0f38c1908414566aa9115bb2cbdd85d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } health_report { overall_health: UNKNOWN } } }
I20260809 15:47:21.030004 23520 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56766 (request call id 2213) took 1091 ms. Trace:
I20260809 15:47:21.030580 28406 raft_consensus.cc:3037] T e14d118b2eaa42aa8a7cd4025aba7128 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Advancing to term 9
I20260809 15:47:21.031306 28335 raft_consensus.cc:3037] T 3296ef50be254fc29cfc77d22a8f7a23 P 61e69695258f48baa984c9d395e8ce3f [term 2 FOLLOWER]: Advancing to term 3
I20260809 15:47:21.032323 28784 raft_consensus.cc:3032] T 3d880de657e544468c82157769622a26 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 LEADER]: Stepping down as leader of term 4
I20260809 15:47:21.032698 28784 raft_consensus.cc:729] T 3d880de657e544468c82157769622a26 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Running, Role: LEADER
I20260809 15:47:21.033468 28784 consensus_queue.cc:260] T 3d880de657e544468c82157769622a26 P 67b3aeadfa7a432abe8af4d292e04ab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:21.034825 28784 raft_consensus.cc:3037] T 3d880de657e544468c82157769622a26 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Advancing to term 5
I20260809 15:47:20.978847 28523 raft_consensus.cc:2726] T 4bcac4b24e7a4448abdc934bba5a624e P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260809 15:47:20.990320 28353 consensus_queue.cc:260] T 472816994d814cb198b69e5248697440 P c0f38c1908414566aa9115bb2cbdd85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:21.007480 28774 leader_election.cc:288] T f9e7cadcd24d45fcbfa84209225e8f2d P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:21.000919 23529 rpcz_store.cc:276] 0809 15:47:17.950824 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:20.596173 (+2645349us) service_pool.cc:227] Handling call
0809 15:47:20.632072 (+ 35899us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:47:20.647315 (+ 15243us) raft_consensus.cc:1520] Early marking committed up to index 8
0809 15:47:20.895846 (+248531us) log.cc:842] Serialized 14 byte log entry
0809 15:47:20.985100 (+ 89254us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 15:47:20.985145 (+ 45us) raft_consensus.cc:1631] Marking committed up to 8
0809 15:47:20.985181 (+ 36us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:47:20.985236 (+ 55us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:47:20.985325 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":5568,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":43,"raft.queue_time_us":78,"raft.run_cpu_time_us":117,"raft.run_wall_time_us":4663,"thread_start_us":296426,"threads_started":2}
I20260809 15:47:20.956787 28522 leader_election.cc:288] T 6ae14a74a26f434db08c652e097385c8 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 9 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:21.023321 23351 leader_election.cc:302] T df20671a7b874dbb98c0ef28371c1308 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:47:21.022239 23519 raft_consensus.cc:2454] T 0da4d5283c8a46878bd39b85e5c71f66 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 7.
I20260809 15:47:21.035449 23470 leader_election.cc:302] T d0721a4403a440f383318a3cea9d1442 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters: c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:21.030361 23520 rpcz_store.cc:276] 0809 15:47:19.938517 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:20.694708 (+756191us) service_pool.cc:227] Handling call
0809 15:47:20.768760 (+ 74052us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:47:21.011328 (+242568us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:47:21.029813 (+ 18485us) spinlock_profiling.cc:243] Waited 186 ms on lock 0x7b54003ba6d4. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e9220f51 00007f34e921f99e 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772 00007f34e764a94c 00007f34e7649d86
0809 15:47:21.029954 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":428947200}
I20260809 15:47:21.047039 23404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2d4f749fcfd4fb3bfb8115af3471598" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:47:21.006886 23223 leader_election.cc:302] T c0741ab44d99439abc3aa3bdd27ee160 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:47:21.041188 28718 raft_consensus.cc:504] T d2d4f749fcfd4fb3bfb8115af3471598 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:21.045644 24448 raft_consensus.cc:1226] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 61e69695258f48baa984c9d395e8ce3f [term 7 LEADER]: Rejecting Update request from peer c0f38c1908414566aa9115bb2cbdd85d for earlier term 5. Current term is 7. Ops: []
I20260809 15:47:21.052956 23400 raft_consensus.cc:2454] T d1a134ba33ea4076bbaac0ae460c17ba P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 8.
I20260809 15:47:21.060708 23470 leader_election.cc:302] T df20671a7b874dbb98c0ef28371c1308 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:21.061292 28718 leader_election.cc:288] T d2d4f749fcfd4fb3bfb8115af3471598 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 9 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:21.062032 28430 raft_consensus.cc:2726] T df20671a7b874dbb98c0ef28371c1308 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260809 15:47:21.061795 28719 raft_consensus.cc:504] T cef3e7277d7d4760824a3436ed4a847d P 65c36d164b0540f099d592f38307ee6c [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:21.063908 28246 consensus_queue.cc:1046] T f7715d71dbac447b9351fc5eb36c5e80 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260809 15:47:21.040988 28586 raft_consensus.cc:482] T d3e94b7e50754a81b3d72a07476cb7b9 P 65c36d164b0540f099d592f38307ee6c [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:21.064971 28586 raft_consensus.cc:504] T d3e94b7e50754a81b3d72a07476cb7b9 P 65c36d164b0540f099d592f38307ee6c [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:21.066919 28586 leader_election.cc:288] T d3e94b7e50754a81b3d72a07476cb7b9 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:21.061371 28627 raft_consensus.cc:2781] T d0721a4403a440f383318a3cea9d1442 P 65c36d164b0540f099d592f38307ee6c [term 9 FOLLOWER]: Leader election won for term 9
I20260809 15:47:21.067749 28522 raft_consensus.cc:2726] T df20671a7b874dbb98c0ef28371c1308 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260809 15:47:21.068643 23404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0acc47ce21c3422698d1ef7277c3ed63" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 10 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
W20260809 15:47:21.012233 23116 rpcz_store.cc:269] 0809 15:46:58.822315 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:46:58.849500 (+ 27185us) service_pool.cc:227] Handling call
0809 15:47:21.011621 (+22162121us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0809 15:47:12.900107 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0809 15:47:12.926731 (+ 26624us) write_op.cc:432] Acquiring schema lock in shared mode
0809 15:47:12.926765 (+ 34us) write_op.cc:435] Acquired schema lock
0809 15:47:12.926781 (+ 16us) tablet.cc:660] Decoding operations
0809 15:47:12.928339 (+ 1558us) write_op.cc:620] Acquiring the partition lock for write op
0809 15:47:12.928399 (+ 60us) write_op.cc:641] Partition lock acquired for write op
0809 15:47:12.928425 (+ 26us) tablet.cc:683] Acquiring locks for 20 operations
0809 15:47:12.929694 (+ 1269us) tablet.cc:699] Row locks acquired
0809 15:47:12.929708 (+ 14us) write_op.cc:260] PREPARE: finished
0809 15:47:12.929843 (+ 135us) write_op.cc:270] Start()
0809 15:47:12.929956 (+ 113us) write_op.cc:276] Timestamp: P: 1786290432929797 usec, L: 0
0809 15:47:12.929978 (+ 22us) op_driver.cc:331] REPLICATION: starting
0809 15:47:12.930634 (+ 656us) log.cc:842] Serialized 7904 byte log entry
0809 15:47:13.663277 (+732643us) op_driver.cc:444] REPLICATION: finished
0809 15:47:14.036134 (+372857us) write_op.cc:301] APPLY: starting
0809 15:47:14.036206 (+ 72us) tablet.cc:1365] starting BulkCheckPresence
0809 15:47:14.036559 (+ 353us) tablet.cc:1368] finished BulkCheckPresence
0809 15:47:14.036679 (+ 120us) tablet.cc:1370] starting ApplyRowOperation cycle
0809 15:47:14.142098 (+105419us) tablet.cc:1381] finished ApplyRowOperation cycle
0809 15:47:14.142372 (+ 274us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=20
0809 15:47:14.142416 (+ 44us) write_op.cc:312] APPLY: finished
0809 15:47:14.142741 (+ 325us) log.cc:842] Serialized 139 byte log entry
0809 15:47:14.142890 (+ 149us) write_op.cc:489] Releasing partition, row and schema locks
0809 15:47:14.143062 (+ 172us) write_op.cc:454] Released schema lock
0809 15:47:14.143113 (+ 51us) write_op.cc:341] FINISH: Updating metrics
0809 15:47:14.143818 (+ 705us) write_op.cc:489] Releasing partition, row and schema locks
0809 15:47:14.143844 (+ 26us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":37442,"reported_tablets":47,"child_traces":[["op",{"apply.queue_time_us":351729,"apply.run_cpu_time_us":9029,"apply.run_wall_time_us":191323,"num_ops":20,"prepare.queue_time_us":656811,"prepare.run_cpu_time_us":34687,"prepare.run_wall_time_us":258560,"raft.queue_time_us":274437,"raft.run_cpu_time_us":17852,"raft.run_wall_time_us":204781,"replication_time_us":733192,"thread_start_us":754359,"threads_started":4,"wal-append.queue_time_us":242822,"wal-append.run_cpu_time_us":54992,"wal-append.run_wall_time_us":1969710}]]}
W20260809 15:47:21.067656 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 44414us
W20260809 15:47:21.053766 23223 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.68:34689, user_credentials={real_user=slave}} blocked reactor thread for 46945us
I20260809 15:47:21.072011 23400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3e94b7e50754a81b3d72a07476cb7b9" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:21.070137 23396 raft_consensus.cc:2454] T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 11.
I20260809 15:47:21.073521 23395 raft_consensus.cc:1226] T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d [term 11 FOLLOWER]: Rejecting Update request from peer 61e69695258f48baa984c9d395e8ce3f for earlier term 10. Current term is 11. Ops: [10.10-10.10]
I20260809 15:47:21.031456 28549 consensus_queue.cc:1046] T 0acc47ce21c3422698d1ef7277c3ed63 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: INVALID_TERM, Last received: 9.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.038s
I20260809 15:47:21.075829 28549 raft_consensus.cc:3032] T 0acc47ce21c3422698d1ef7277c3ed63 P c0f38c1908414566aa9115bb2cbdd85d [term 9 LEADER]: Stepping down as leader of term 9
I20260809 15:47:21.076098 28246 raft_consensus.cc:3032] T f7715d71dbac447b9351fc5eb36c5e80 P 65c36d164b0540f099d592f38307ee6c [term 6 LEADER]: Stepping down as leader of term 6
I20260809 15:47:21.075009 28451 consensus_queue.cc:1035] T 30c1740ef6874df0a332fbd6fd4b03ce P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.002s
I20260809 15:47:21.076534 28246 raft_consensus.cc:729] T f7715d71dbac447b9351fc5eb36c5e80 P 65c36d164b0540f099d592f38307ee6c [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:47:21.076152 28549 raft_consensus.cc:729] T 0acc47ce21c3422698d1ef7277c3ed63 P c0f38c1908414566aa9115bb2cbdd85d [term 9 LEADER]: Becoming Follower/Learner. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:47:21.078215 28474 consensus_queue.cc:1046] T ae6d2c08a7a94f58ac8dd698e3fd456c P 65c36d164b0540f099d592f38307ee6c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260809 15:47:21.078557 28549 consensus_queue.cc:260] T 0acc47ce21c3422698d1ef7277c3ed63 P c0f38c1908414566aa9115bb2cbdd85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 9.4, Last appended by leader: 4, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:21.079522 28627 raft_consensus.cc:686] T d0721a4403a440f383318a3cea9d1442 P 65c36d164b0540f099d592f38307ee6c [term 9 LEADER]: Becoming Leader. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:47:21.079890 28549 raft_consensus.cc:3037] T 0acc47ce21c3422698d1ef7277c3ed63 P c0f38c1908414566aa9115bb2cbdd85d [term 9 FOLLOWER]: Advancing to term 10
I20260809 15:47:21.080489 28627 consensus_queue.cc:237] T d0721a4403a440f383318a3cea9d1442 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:21.081120 23285 raft_consensus.cc:1226] T 763e58d6ba644eea830a6c1664d39c9b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 9 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 7. Current term is 9. Ops: []
I20260809 15:47:21.083832 28353 raft_consensus.cc:3037] T 472816994d814cb198b69e5248697440 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Advancing to term 7
I20260809 15:47:21.079049 28474 raft_consensus.cc:3032] T ae6d2c08a7a94f58ac8dd698e3fd456c P 65c36d164b0540f099d592f38307ee6c [term 8 LEADER]: Stepping down as leader of term 8
I20260809 15:47:21.085927 28474 raft_consensus.cc:729] T ae6d2c08a7a94f58ac8dd698e3fd456c P 65c36d164b0540f099d592f38307ee6c [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:47:21.063781 28719 leader_election.cc:288] T cef3e7277d7d4760824a3436ed4a847d P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 9 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:47:21.077339 28246 consensus_queue.cc:260] T f7715d71dbac447b9351fc5eb36c5e80 P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:21.073812 23472 leader_election.cc:302] T d1a134ba33ea4076bbaac0ae460c17ba P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:47:21.090939 23270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9e7cadcd24d45fcbfa84209225e8f2d" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:47:21.091784 23270 raft_consensus.cc:3037] T f9e7cadcd24d45fcbfa84209225e8f2d P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Advancing to term 9
I20260809 15:47:21.095382 23274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e17dcb5efe45b8b6a8c42da03b4819" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:21.096868 23280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1a134ba33ea4076bbaac0ae460c17ba" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:21.098588 23404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cef3e7277d7d4760824a3436ed4a847d" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:47:21.102388 23286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2424fd9bd7f14e508fd106cbfda786a4" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 9 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:47:21.118443 23273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "953eea268e17491ca76f3a555860a512" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:47:21.122563 23285 raft_consensus.cc:1226] T f7715d71dbac447b9351fc5eb36c5e80 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 6. Current term is 7. Ops: []
I20260809 15:47:21.124300 23351 leader_election.cc:302] T ae6d2c08a7a94f58ac8dd698e3fd456c P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:47:21.122551 23282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "294d4bcc1e7a477cb56455581277a60f" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
W20260809 15:47:21.129307 23472 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 55569us
I20260809 15:47:21.129730 28462 raft_consensus.cc:2781] T d1a134ba33ea4076bbaac0ae460c17ba P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Leader election won for term 8
I20260809 15:47:21.130249 23472 leader_election.cc:302] T 3924b648461e463b9d7925e5671d68ae P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters: 61e69695258f48baa984c9d395e8ce3f
I20260809 15:47:21.130265 28462 raft_consensus.cc:686] T d1a134ba33ea4076bbaac0ae460c17ba P 65c36d164b0540f099d592f38307ee6c [term 8 LEADER]: Becoming Leader. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:47:21.131981 28462 consensus_queue.cc:237] T d1a134ba33ea4076bbaac0ae460c17ba P 65c36d164b0540f099d592f38307ee6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:21.135245 28525 raft_consensus.cc:2781] T 3924b648461e463b9d7925e5671d68ae P 65c36d164b0540f099d592f38307ee6c [term 11 FOLLOWER]: Leader election won for term 11
I20260809 15:47:21.135867 28525 raft_consensus.cc:686] T 3924b648461e463b9d7925e5671d68ae P 65c36d164b0540f099d592f38307ee6c [term 11 LEADER]: Becoming Leader. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:47:21.136715 28525 consensus_queue.cc:237] T 3924b648461e463b9d7925e5671d68ae P 65c36d164b0540f099d592f38307ee6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
W20260809 15:47:21.147454 23353 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 47834.1us
I20260809 15:47:21.151739 23270 raft_consensus.cc:2454] T f9e7cadcd24d45fcbfa84209225e8f2d P 67b3aeadfa7a432abe8af4d292e04ab9 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 9.
I20260809 15:47:21.154049 23280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee02fff2c9c48a19f43eb358a89e7b0" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 10 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:21.023926 28725 consensus_queue.cc:237] T 2424fd9bd7f14e508fd106cbfda786a4 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.3, Last appended by leader: 3, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:21.157574 23353 leader_election.cc:302] T 0da4d5283c8a46878bd39b85e5c71f66 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:47:21.157948 23277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30c1740ef6874df0a332fbd6fd4b03ce" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:47:21.161229 23270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "297c82ed906d4b48b184985eed3de601" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:21.162596 23519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ae14a74a26f434db08c652e097385c8" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:47:21.162637 23282 raft_consensus.cc:2454] T 294d4bcc1e7a477cb56455581277a60f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 6.
I20260809 15:47:21.163203 23519 raft_consensus.cc:3037] T 6ae14a74a26f434db08c652e097385c8 P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Advancing to term 9
I20260809 15:47:21.167801 23277 raft_consensus.cc:3037] T 30c1740ef6874df0a332fbd6fd4b03ce P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Advancing to term 9
I20260809 15:47:21.170984 23273 raft_consensus.cc:2379] T 953eea268e17491ca76f3a555860a512 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c in current term 9: Already voted for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in this term.
I20260809 15:47:21.086772 28474 consensus_queue.cc:260] T ae6d2c08a7a94f58ac8dd698e3fd456c P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:21.074523 28302 consensus_queue.cc:1046] T 3924b648461e463b9d7925e5671d68ae P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.083s
I20260809 15:47:21.041227 28546 consensus_queue.cc:260] T d0721a4403a440f383318a3cea9d1442 P c0f38c1908414566aa9115bb2cbdd85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:47:21.173549 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 79744.2us
I20260809 15:47:21.174257 23351 leader_election.cc:302] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters: 61e69695258f48baa984c9d395e8ce3f
I20260809 15:47:21.143404 28874 raft_consensus.cc:482] T 538a228ddfb54bc6a36528d5ff07597b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:21.176738 23273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00a1881267a14ba8b78c81040f17a014" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:47:21.173210 23285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2db057fd10e149a9895dff6ef8c9f038" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:21.175145 23270 raft_consensus.cc:2454] T 297c82ed906d4b48b184985eed3de601 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 9.
I20260809 15:47:21.102759 28425 consensus_queue.cc:1035] T f9e7cadcd24d45fcbfa84209225e8f2d P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 2, Time since last communication: 0.069s
I20260809 15:47:21.173295 23274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "213543cdbe814c89a17a963469922c1b" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:47:21.179636 23274 raft_consensus.cc:2379] T 213543cdbe814c89a17a963469922c1b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c in current term 5: Already voted for candidate c0f38c1908414566aa9115bb2cbdd85d in this term.
I20260809 15:47:21.179943 23276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3622f6e3e3641cea1960e3a31498bb7" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:21.181970 23351 leader_election.cc:302] T a3622f6e3e3641cea1960e3a31498bb7 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c
I20260809 15:47:21.162631 28688 raft_consensus.cc:3037] T ae6d2c08a7a94f58ac8dd698e3fd456c P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Advancing to term 9
I20260809 15:47:21.189697 23286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "563daf42d7784db195ea99c3cb21b370" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:21.185781 23285 raft_consensus.cc:2379] T 2db057fd10e149a9895dff6ef8c9f038 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c in current term 8: Already voted for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in this term.
I20260809 15:47:21.163170 28246 raft_consensus.cc:3037] T f7715d71dbac447b9351fc5eb36c5e80 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Advancing to term 7
I20260809 15:47:21.186547 23270 raft_consensus.cc:1226] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 4. Current term is 5. Ops: []
I20260809 15:47:21.186754 23528 raft_consensus.cc:1261] T e14d118b2eaa42aa8a7cd4025aba7128 P 65c36d164b0540f099d592f38307ee6c [term 9 FOLLOWER]: Refusing update from remote peer 61e69695258f48baa984c9d395e8ce3f: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 9 index: 10. (index mismatch)
I20260809 15:47:21.193584 28887 raft_consensus.cc:2781] T c0741ab44d99439abc3aa3bdd27ee160 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 9 FOLLOWER]: Leader election won for term 9
I20260809 15:47:21.194228 28887 raft_consensus.cc:686] T c0741ab44d99439abc3aa3bdd27ee160 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 9 LEADER]: Becoming Leader. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Running, Role: LEADER
I20260809 15:47:21.162351 28592 consensus_queue.cc:1046] T d2d4f749fcfd4fb3bfb8115af3471598 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.033s
I20260809 15:47:21.195204 28887 consensus_queue.cc:237] T c0741ab44d99439abc3aa3bdd27ee160 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:21.168503 28695 raft_consensus.cc:2781] T 0da4d5283c8a46878bd39b85e5c71f66 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Leader election won for term 7
I20260809 15:47:21.167303 24455 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3e94b7e50754a81b3d72a07476cb7b9" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:47:21.187489 23282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c29937a80724ddb9d07a27534a146f3" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 9 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:21.213758 23282 raft_consensus.cc:2379] T 0c29937a80724ddb9d07a27534a146f3 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c in current term 9: Already voted for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in this term.
I20260809 15:47:21.197438 28522 raft_consensus.cc:2781] T a3622f6e3e3641cea1960e3a31498bb7 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Leader election won for term 7
I20260809 15:47:21.195622 23274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "297c82ed906d4b48b184985eed3de601" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:47:21.219575 23274 raft_consensus.cc:3037] T 297c82ed906d4b48b184985eed3de601 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 9 FOLLOWER]: Advancing to term 10
I20260809 15:47:21.214227 23394 raft_consensus.cc:3037] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Advancing to term 7
I20260809 15:47:21.188601 23276 raft_consensus.cc:2379] T a3622f6e3e3641cea1960e3a31498bb7 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c in current term 7: Already voted for candidate c0f38c1908414566aa9115bb2cbdd85d in this term.
I20260809 15:47:21.179149 28874 raft_consensus.cc:504] T 538a228ddfb54bc6a36528d5ff07597b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:21.220118 23285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e14d118b2eaa42aa8a7cd4025aba7128" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:21.165794 27957 consensus_queue.cc:1046] T 99bdb93503db4de4a1cfb0161ea0c07c P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.045s
I20260809 15:47:21.224983 23286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1a134ba33ea4076bbaac0ae460c17ba" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:47:21.190137 28560 raft_consensus.cc:2781] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260809 15:47:21.172123 28621 raft_consensus.cc:504] T 0da4d5283c8a46878bd39b85e5c71f66 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:21.228971 24448 raft_consensus.cc:1226] T c0741ab44d99439abc3aa3bdd27ee160 P 61e69695258f48baa984c9d395e8ce3f [term 9 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 8. Current term is 9. Ops: []
I20260809 15:47:21.235257 23519 raft_consensus.cc:2454] T 6ae14a74a26f434db08c652e097385c8 P 65c36d164b0540f099d592f38307ee6c [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 9.
I20260809 15:47:21.235770 28546 raft_consensus.cc:3037] T d0721a4403a440f383318a3cea9d1442 P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Advancing to term 9
W20260809 15:47:21.236096 28871 log.cc:924] Time spent T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d: Append to log took a long time: real 0.169s user 0.001s sys 0.001s
I20260809 15:47:21.236871 28416 raft_consensus.cc:504] T cee02fff2c9c48a19f43eb358a89e7b0 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:21.213801 28532 consensus_queue.cc:1035] T e14d118b2eaa42aa8a7cd4025aba7128 P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.034s
I20260809 15:47:21.249469 23285 raft_consensus.cc:2454] T e14d118b2eaa42aa8a7cd4025aba7128 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 9.
I20260809 15:47:21.240665 23117 catalog_manager.cc:5495] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P c0f38c1908414566aa9115bb2cbdd85d reported cstate change: term changed from 2 to 4, leader changed from 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) to c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66). New cstate: current_term: 4 leader_uuid: "c0f38c1908414566aa9115bb2cbdd85d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260809 15:47:21.253862 23276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99bdb93503db4de4a1cfb0161ea0c07c" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:47:21.255788 23286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0acc47ce21c3422698d1ef7277c3ed63" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 10 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:47:21.256119 23527 raft_consensus.cc:1261] T 472816994d814cb198b69e5248697440 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Refusing update from remote peer 61e69695258f48baa984c9d395e8ce3f: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260809 15:47:21.251852 23282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2d4f749fcfd4fb3bfb8115af3471598" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:47:21.259990 23282 raft_consensus.cc:2379] T d2d4f749fcfd4fb3bfb8115af3471598 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c in current term 9: Already voted for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in this term.
I20260809 15:47:21.260987 23274 raft_consensus.cc:2454] T 297c82ed906d4b48b184985eed3de601 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 10.
I20260809 15:47:21.261970 28695 raft_consensus.cc:686] T 0da4d5283c8a46878bd39b85e5c71f66 P c0f38c1908414566aa9115bb2cbdd85d [term 7 LEADER]: Becoming Leader. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:47:21.262984 23274 raft_consensus.cc:1261] T f9e7cadcd24d45fcbfa84209225e8f2d P 67b3aeadfa7a432abe8af4d292e04ab9 [term 9 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20260809 15:47:21.262879 28695 consensus_queue.cc:237] T 0da4d5283c8a46878bd39b85e5c71f66 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:47:21.239703 23193 auto_rebalancer.cc:254] failed to send replica move request: Illegal state: Replica 65c36d164b0540f099d592f38307ee6c is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 5 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } }
I20260809 15:47:21.267050 28447 raft_consensus.cc:3032] T d2d4f749fcfd4fb3bfb8115af3471598 P c0f38c1908414566aa9115bb2cbdd85d [term 8 LEADER]: Stepping down as leader of term 8
I20260809 15:47:21.267503 28674 raft_consensus.cc:3032] T 4bcac4b24e7a4448abdc934bba5a624e P 61e69695258f48baa984c9d395e8ce3f [term 4 LEADER]: Stepping down as leader of term 4
I20260809 15:47:21.270113 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7715d71dbac447b9351fc5eb36c5e80" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:47:21.285734 23283 raft_consensus.cc:1226] T 0da4d5283c8a46878bd39b85e5c71f66 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 5. Current term is 7. Ops: []
I20260809 15:47:21.216615 28756 consensus_queue.cc:1046] T 3924b648461e463b9d7925e5671d68ae P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: INVALID_TERM, Last received: 8.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.041s
I20260809 15:47:21.272172 28874 leader_election.cc:288] T 538a228ddfb54bc6a36528d5ff07597b P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:21.272783 23285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e14d118b2eaa42aa8a7cd4025aba7128" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:21.301728 23285 raft_consensus.cc:2454] T e14d118b2eaa42aa8a7cd4025aba7128 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 9.
I20260809 15:47:21.273738 23470 leader_election.cc:302] T 28e17dcb5efe45b8b6a8c42da03b4819 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:21.276535 23394 raft_consensus.cc:2454] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 7.
I20260809 15:47:21.283735 23519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ced718c1a5c4da79cb8c1ebd1435b87" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:21.286835 23280 raft_consensus.cc:2379] T cee02fff2c9c48a19f43eb358a89e7b0 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c in current term 10: Already voted for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in this term.
I20260809 15:47:21.270308 28474 raft_consensus.cc:3037] T ae6d2c08a7a94f58ac8dd698e3fd456c P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Advancing to term 9
I20260809 15:47:21.288347 23274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e71a05ba1d4d42bd95c3f85f9313399e" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:21.289230 23397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3be1e356b254d22b995be32f1c12f6f" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:21.239241 28319 leader_election.cc:288] T 00a1881267a14ba8b78c81040f17a014 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 10 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:21.232214 28560 raft_consensus.cc:482] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Starting leader election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
I20260809 15:47:21.303440 23519 raft_consensus.cc:2454] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 5.
I20260809 15:47:21.311336 23286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cef3e7277d7d4760824a3436ed4a847d" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:47:21.311734 23517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae6d2c08a7a94f58ac8dd698e3fd456c" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:47:21.311825 23395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7715d71dbac447b9351fc5eb36c5e80" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:47:21.307611 23393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ced718c1a5c4da79cb8c1ebd1435b87" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:21.316083 23393 raft_consensus.cc:2454] T 6ced718c1a5c4da79cb8c1ebd1435b87 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 5.
I20260809 15:47:21.306027 23396 raft_consensus.cc:1226] T e71a05ba1d4d42bd95c3f85f9313399e P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Rejecting Update request from peer 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 6. Current term is 7. Ops: []
I20260809 15:47:21.302179 23276 pending_rounds.cc:76] T 16ca412a0dae4feebe85dd01bf1a9173 P 67b3aeadfa7a432abe8af4d292e04ab9: Aborting all ops after (but not including) 6
I20260809 15:47:21.317250 23276 pending_rounds.cc:97] T 16ca412a0dae4feebe85dd01bf1a9173 P 67b3aeadfa7a432abe8af4d292e04ab9: Aborting uncommitted NO_OP operation due to leader change: 6.7
I20260809 15:47:21.317505 23276 raft_consensus.cc:2866] T 16ca412a0dae4feebe85dd01bf1a9173 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 15:47:21.306835 23274 raft_consensus.cc:2454] T e71a05ba1d4d42bd95c3f85f9313399e P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 7.
I20260809 15:47:21.333057 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae6d2c08a7a94f58ac8dd698e3fd456c" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:47:21.333199 23470 leader_election.cc:302] T 294d4bcc1e7a477cb56455581277a60f P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 61e69695258f48baa984c9d395e8ce3f
I20260809 15:47:21.336565 23285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "563daf42d7784db195ea99c3cb21b370" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:21.334856 23470 leader_election.cc:302] T 953eea268e17491ca76f3a555860a512 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:21.284946 28447 raft_consensus.cc:729] T d2d4f749fcfd4fb3bfb8115af3471598 P c0f38c1908414566aa9115bb2cbdd85d [term 8 LEADER]: Becoming Follower/Learner. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
W20260809 15:47:21.347036 23353 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 68732.2us
I20260809 15:47:21.353148 23522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0741ab44d99439abc3aa3bdd27ee160" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:21.354036 23399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05ad7d8fdca64e3da527280ca5be9280" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:47:21.347797 23283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e14d118b2eaa42aa8a7cd4025aba7128" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:47:21.348404 23392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f9d6565aa4c45a3b3f01210d0eb71f7" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:21.287578 28674 raft_consensus.cc:729] T 4bcac4b24e7a4448abdc934bba5a624e P 61e69695258f48baa984c9d395e8ce3f [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 61e69695258f48baa984c9d395e8ce3f, State: Running, Role: LEADER
I20260809 15:47:21.354877 23277 raft_consensus.cc:2454] T 30c1740ef6874df0a332fbd6fd4b03ce P 67b3aeadfa7a432abe8af4d292e04ab9 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 9.
I20260809 15:47:21.365603 23353 leader_election.cc:302] T 6ae14a74a26f434db08c652e097385c8 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:47:21.366421 23280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee02fff2c9c48a19f43eb358a89e7b0" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:21.367161 23280 raft_consensus.cc:2379] T cee02fff2c9c48a19f43eb358a89e7b0 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 61e69695258f48baa984c9d395e8ce3f in current term 10: Already voted for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in this term.
I20260809 15:47:21.312958 28197 raft_consensus.cc:2726] T 28e17dcb5efe45b8b6a8c42da03b4819 P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260809 15:47:21.375790 23395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e17dcb5efe45b8b6a8c42da03b4819" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:47:21.369031 23273 raft_consensus.cc:1261] T 30c1740ef6874df0a332fbd6fd4b03ce P 67b3aeadfa7a432abe8af4d292e04ab9 [term 9 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260809 15:47:21.376282 23519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05ad7d8fdca64e3da527280ca5be9280" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:47:21.376832 23393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b30f28311964a0b9149a50495e83d8f" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:21.377650 23521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b30f28311964a0b9149a50495e83d8f" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:21.378212 23521 raft_consensus.cc:2454] T 4b30f28311964a0b9149a50495e83d8f P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 6.
I20260809 15:47:21.378999 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16ca412a0dae4feebe85dd01bf1a9173" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:21.284335 28302 consensus_queue.cc:1035] T 472816994d814cb198b69e5248697440 P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.068s
I20260809 15:47:21.383945 23270 raft_consensus.cc:1226] T c0741ab44d99439abc3aa3bdd27ee160 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 9 LEADER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 8. Current term is 9. Ops: []
I20260809 15:47:21.369176 23397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9e7cadcd24d45fcbfa84209225e8f2d" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:21.384614 23393 raft_consensus.cc:2379] T 4b30f28311964a0b9149a50495e83d8f P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 in current term 6: Already voted for candidate c0f38c1908414566aa9115bb2cbdd85d in this term.
I20260809 15:47:21.373354 23470 leader_election.cc:302] T 213543cdbe814c89a17a963469922c1b P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:21.381260 23399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998c6cee72cf40228c6dba064c0db2da" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:47:21.381826 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00a1881267a14ba8b78c81040f17a014" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:47:21.262794 28280 raft_consensus.cc:504] T cee02fff2c9c48a19f43eb358a89e7b0 P 61e69695258f48baa984c9d395e8ce3f [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:21.329057 28560 raft_consensus.cc:3037] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Advancing to term 5
I20260809 15:47:21.311244 28719 raft_consensus.cc:482] T 538a228ddfb54bc6a36528d5ff07597b P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:47:21.284732 28285 consensus_queue.cc:1046] T c0741ab44d99439abc3aa3bdd27ee160 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.069s
W20260809 15:47:21.375069 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 104233us
I20260809 15:47:21.354394 28319 raft_consensus.cc:482] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:21.365702 28586 raft_consensus.cc:2781] T 294d4bcc1e7a477cb56455581277a60f P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260809 15:47:21.400914 23521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c48a3049897452383257aa757834fb5" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:21.391304 23396 raft_consensus.cc:1261] T 07fcabb569414bddbd39df6cedd24e64 P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Refusing update from remote peer 67b3aeadfa7a432abe8af4d292e04ab9: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260809 15:47:21.398826 23470 leader_election.cc:302] T 2db057fd10e149a9895dff6ef8c9f038 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:21.403493 23283 raft_consensus.cc:2454] T e14d118b2eaa42aa8a7cd4025aba7128 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 9.
I20260809 15:47:21.403632 23228 leader_election.cc:302] T d0721a4403a440f383318a3cea9d1442 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:21.311347 28229 consensus_queue.cc:1035] T d2d4f749fcfd4fb3bfb8115af3471598 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.070s
I20260809 15:47:21.404457 23392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2d4f749fcfd4fb3bfb8115af3471598" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:47:21.380913 28430 raft_consensus.cc:2726] T 953eea268e17491ca76f3a555860a512 P 65c36d164b0540f099d592f38307ee6c [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260809 15:47:21.413228 23228 leader_election.cc:302] T a3622f6e3e3641cea1960e3a31498bb7 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c
I20260809 15:47:21.389158 28248 raft_consensus.cc:2781] T 6ae14a74a26f434db08c652e097385c8 P c0f38c1908414566aa9115bb2cbdd85d [term 9 FOLLOWER]: Leader election won for term 9
I20260809 15:47:21.415704 23522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5712f6b6fffc4cc5a8ddfbe63e4097df" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:47:21.423753 23519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e17dcb5efe45b8b6a8c42da03b4819" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:47:21.425709 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee02fff2c9c48a19f43eb358a89e7b0" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:21.426287 23526 raft_consensus.cc:2454] T cee02fff2c9c48a19f43eb358a89e7b0 P 65c36d164b0540f099d592f38307ee6c [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 9.
I20260809 15:47:21.405802 28688 raft_consensus.cc:2726] T ae6d2c08a7a94f58ac8dd698e3fd456c P c0f38c1908414566aa9115bb2cbdd85d [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260809 15:47:21.400573 28586 raft_consensus.cc:482] T 294d4bcc1e7a477cb56455581277a60f P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Starting leader election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
I20260809 15:47:21.428602 23228 leader_election.cc:302] T 563daf42d7784db195ea99c3cb21b370 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:21.411628 28197 raft_consensus.cc:2726] T 213543cdbe814c89a17a963469922c1b P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260809 15:47:21.404696 28522 raft_consensus.cc:686] T a3622f6e3e3641cea1960e3a31498bb7 P c0f38c1908414566aa9115bb2cbdd85d [term 7 LEADER]: Becoming Leader. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:47:21.435974 23468 leader_election.cc:302] T d3e94b7e50754a81b3d72a07476cb7b9 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d
W20260809 15:47:21.437518 23468 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.68:34689, user_credentials={real_user=slave}} blocked reactor thread for 160900us
I20260809 15:47:21.433336 23470 leader_election.cc:302] T 00a1881267a14ba8b78c81040f17a014 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:21.435323 23522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9e7cadcd24d45fcbfa84209225e8f2d" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:21.438885 23470 leader_election.cc:302] T 0c29937a80724ddb9d07a27534a146f3 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:21.439865 23470 leader_election.cc:302] T a3622f6e3e3641cea1960e3a31498bb7 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:21.440249 28248 raft_consensus.cc:686] T 6ae14a74a26f434db08c652e097385c8 P c0f38c1908414566aa9115bb2cbdd85d [term 9 LEADER]: Becoming Leader. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:47:21.441231 28903 raft_consensus.cc:3037] T a3622f6e3e3641cea1960e3a31498bb7 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Advancing to term 7
I20260809 15:47:21.397385 28719 raft_consensus.cc:504] T 538a228ddfb54bc6a36528d5ff07597b P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:21.425850 28887 raft_consensus.cc:2726] T d0721a4403a440f383318a3cea9d1442 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260809 15:47:21.446506 23521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998c6cee72cf40228c6dba064c0db2da" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:47:21.425422 28718 raft_consensus.cc:2741] T a3622f6e3e3641cea1960e3a31498bb7 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
I20260809 15:47:21.449054 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0741ab44d99439abc3aa3bdd27ee160" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:47:21.451000 23470 leader_election.cc:302] T 99bdb93503db4de4a1cfb0161ea0c07c P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:21.428237 28525 raft_consensus.cc:3037] T 2db057fd10e149a9895dff6ef8c9f038 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Advancing to term 8
I20260809 15:47:21.453109 23519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2d4f749fcfd4fb3bfb8115af3471598" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:47:21.441174 28725 raft_consensus.cc:3037] T 0c29937a80724ddb9d07a27534a146f3 P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Advancing to term 9
I20260809 15:47:21.455255 23228 leader_election.cc:302] T 3296ef50be254fc29cfc77d22a8f7a23 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:21.411510 28319 raft_consensus.cc:504] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:21.436503 28586 raft_consensus.cc:3037] T 294d4bcc1e7a477cb56455581277a60f P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Advancing to term 7
I20260809 15:47:21.460741 23522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00a1881267a14ba8b78c81040f17a014" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:47:21.461676 23526 raft_consensus.cc:1226] T e14d118b2eaa42aa8a7cd4025aba7128 P 65c36d164b0540f099d592f38307ee6c [term 9 FOLLOWER]: Rejecting Update request from peer 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 8. Current term is 9. Ops: []
I20260809 15:47:21.448583 28612 raft_consensus.cc:2726] T 563daf42d7784db195ea99c3cb21b370 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260809 15:47:21.465833 28893 log.cc:924] Time spent T d3e94b7e50754a81b3d72a07476cb7b9 P c0f38c1908414566aa9115bb2cbdd85d: Append to log took a long time: real 0.096s user 0.000s sys 0.001s
I20260809 15:47:21.466826 23521 raft_consensus.cc:1226] T 2424fd9bd7f14e508fd106cbfda786a4 P 65c36d164b0540f099d592f38307ee6c [term 9 LEADER]: Rejecting Update request from peer 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 8. Current term is 9. Ops: []
I20260809 15:47:21.467360 28522 consensus_queue.cc:237] T a3622f6e3e3641cea1960e3a31498bb7 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:47:21.458681 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 48308.9us
I20260809 15:47:21.455734 23470 leader_election.cc:302] T d2d4f749fcfd4fb3bfb8115af3471598 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:21.459081 23519 raft_consensus.cc:2379] T d2d4f749fcfd4fb3bfb8115af3471598 P 65c36d164b0540f099d592f38307ee6c [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 in current term 9: Already voted for candidate 65c36d164b0540f099d592f38307ee6c in this term.
I20260809 15:47:21.441836 28285 raft_consensus.cc:2726] T 00a1881267a14ba8b78c81040f17a014 P 65c36d164b0540f099d592f38307ee6c [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260809 15:47:21.452857 28621 leader_election.cc:288] T 0da4d5283c8a46878bd39b85e5c71f66 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 7 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:21.456969 28456 raft_consensus.cc:2726] T 99bdb93503db4de4a1cfb0161ea0c07c P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260809 15:47:21.457005 28416 leader_election.cc:288] T cee02fff2c9c48a19f43eb358a89e7b0 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 10 election: Requested vote from peers 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:21.508301 23274 raft_consensus.cc:1226] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 9 FOLLOWER]: Rejecting Update request from peer 61e69695258f48baa984c9d395e8ce3f for earlier term 8. Current term is 9. Ops: [8.8-8.8]
I20260809 15:47:21.464509 28887 raft_consensus.cc:2726] T 3296ef50be254fc29cfc77d22a8f7a23 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 15:47:21.447852 28688 consensus_queue.cc:1046] T 05ad7d8fdca64e3da527280ca5be9280 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: INVALID_TERM, Last received: 7.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.016s
W20260809 15:47:21.512358 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 57156.6us
I20260809 15:47:21.518316 23526 raft_consensus.cc:1261] T 07fcabb569414bddbd39df6cedd24e64 P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Refusing update from remote peer 67b3aeadfa7a432abe8af4d292e04ab9: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260809 15:47:21.524590 23351 leader_election.cc:302] T e71a05ba1d4d42bd95c3f85f9313399e P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters: 61e69695258f48baa984c9d395e8ce3f
I20260809 15:47:21.441303 28248 consensus_queue.cc:237] T 6ae14a74a26f434db08c652e097385c8 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:21.508806 28451 raft_consensus.cc:2726] T d2d4f749fcfd4fb3bfb8115af3471598 P 65c36d164b0540f099d592f38307ee6c [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260809 15:47:21.513019 24455 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee02fff2c9c48a19f43eb358a89e7b0" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f"
W20260809 15:47:21.534622 23345 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.68:34689, user_credentials={real_user=slave}} blocked reactor thread for 109350us
I20260809 15:47:21.556175 23351 leader_election.cc:302] T 213543cdbe814c89a17a963469922c1b P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c
W20260809 15:47:21.554790 23193 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 6ced718c1a5c4da79cb8c1ebd1435b87, TS 65c36d164b0540f099d592f38307ee6c -> TS 61e69695258f48baa984c9d395e8ce3f move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:47:21.563589 28913 log.cc:924] Time spent T 294d4bcc1e7a477cb56455581277a60f P 67b3aeadfa7a432abe8af4d292e04ab9: Append to log took a long time: real 0.056s user 0.000s sys 0.001s
W20260809 15:47:21.562222 23193 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 6ced718c1a5c4da79cb8c1ebd1435b87, TS 65c36d164b0540f099d592f38307ee6c -> TS 61e69695258f48baa984c9d395e8ce3f move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:21.562759 24455 raft_consensus.cc:2379] T cee02fff2c9c48a19f43eb358a89e7b0 P 61e69695258f48baa984c9d395e8ce3f [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 in current term 10: Already voted for candidate 61e69695258f48baa984c9d395e8ce3f in this term.
I20260809 15:47:21.575730 23470 leader_election.cc:302] T 0acc47ce21c3422698d1ef7277c3ed63 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:21.551457 28577 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.0.0.1:38734 (request call id 421) took 7045 ms. Trace:
W20260809 15:47:21.562192 28895 log.cc:924] Time spent T a88421dba80d4d3388aa5eb62e23853b P 65c36d164b0540f099d592f38307ee6c: Append to log took a long time: real 0.180s user 0.001s sys 0.000s
I20260809 15:47:21.581409 28921 raft_consensus.cc:2781] T 213543cdbe814c89a17a963469922c1b P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Leader election won for term 5
I20260809 15:47:21.579352 28456 consensus_queue.cc:1035] T f9e7cadcd24d45fcbfa84209225e8f2d P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.012s
I20260809 15:47:21.581540 28577 rpcz_store.cc:276] 0809 15:47:14.512290 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:14.581442 (+ 69152us) service_pool.cc:227] Handling call
0809 15:47:14.964456 (+383014us) ts_tablet_manager.cc:928] T ba56725e42f74a1baa4c04d9660727f9 P 61e69695258f48baa984c9d395e8ce3f: Initiating tablet copy from peer 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
0809 15:47:18.739902 (+3775446us) spinlock_profiling.cc:243] Waited 4.08 ms on lock 0x7b44000232cc. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e613cf11 00007f34e620aa95 00007f34e620b4b2 00007f34e8eb3511 00007f34e8eaee1a 00007f34e62da5fe 00007f34ef111a39 00007f34e62f78bb 00007f34e62f462a 00007f34e62f4552 00007f34ee111691
0809 15:47:20.487491 (+1747589us) tablet_copy_client.cc:361] Tablet copy session begun
0809 15:47:20.944676 (+457185us) tablet_metadata.cc:641] Metadata flushed
0809 15:47:20.944967 (+ 291us) tablet_copy_client.cc:490] Wrote new tablet metadata
0809 15:47:21.404472 (+459505us) tablet_copy_client.cc:411] Wrote new consensus metadata
0809 15:47:21.435840 (+ 31368us) tablet_replica.cc:205] Creating consensus instance
0809 15:47:21.510587 (+ 74747us) ts_tablet_manager.cc:983] Replica registered.
0809 15:47:21.536875 (+ 26288us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":13307,"spinlock_wait_cycles":9394432,"tablet-copy.queue_time_us":313645,"thread_start_us":4092956,"threads_started":2}
I20260809 15:47:21.594187 28577 tablet_copy_client.cc:791] T ba56725e42f74a1baa4c04d9660727f9 P 61e69695258f48baa984c9d395e8ce3f: tablet copy: Starting download of 0 data blocks...
I20260809 15:47:21.594810 28577 tablet_copy_client.cc:655] T ba56725e42f74a1baa4c04d9660727f9 P 61e69695258f48baa984c9d395e8ce3f: tablet copy: Starting download of 1 WAL segments...
I20260809 15:47:21.597461 28416 raft_consensus.cc:482] T 6ae14a74a26f434db08c652e097385c8 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:21.598064 23351 leader_election.cc:302] T 4b30f28311964a0b9149a50495e83d8f P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:47:21.597923 28416 raft_consensus.cc:504] T 6ae14a74a26f434db08c652e097385c8 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:21.598502 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "907bacfa79d04a55a68c8dbe60056462" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:47:21.599341 24468 raft_consensus.cc:2396] T 907bacfa79d04a55a68c8dbe60056462 P 61e69695258f48baa984c9d395e8ce3f [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 for term 10 because replica has last-logged OpId of term: 8 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
I20260809 15:47:21.599889 28936 raft_consensus.cc:2726] T 4b30f28311964a0b9149a50495e83d8f P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260809 15:47:21.599889 28416 leader_election.cc:288] T 6ae14a74a26f434db08c652e097385c8 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:21.600589 23276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e4ffcf83c234ae7910ddef3ec0b4bda" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:47:21.601094 28921 raft_consensus.cc:686] T 213543cdbe814c89a17a963469922c1b P c0f38c1908414566aa9115bb2cbdd85d [term 5 LEADER]: Becoming Leader. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:47:21.602037 28921 consensus_queue.cc:237] T 213543cdbe814c89a17a963469922c1b P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:21.602597 23276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3296ef50be254fc29cfc77d22a8f7a23" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:47:21.603515 23273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5c2e425a74b4e64ae70b5def750434c" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:47:21.604069 23273 raft_consensus.cc:3037] T c5c2e425a74b4e64ae70b5def750434c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Advancing to term 8
I20260809 15:47:21.604630 23517 raft_consensus.cc:2396] T ae6d2c08a7a94f58ac8dd698e3fd456c P 65c36d164b0540f099d592f38307ee6c [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 for term 9 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 4.
I20260809 15:47:21.569854 28932 raft_consensus.cc:3032] T 05ad7d8fdca64e3da527280ca5be9280 P c0f38c1908414566aa9115bb2cbdd85d [term 7 LEADER]: Stepping down as leader of term 7
I20260809 15:47:21.605038 28932 raft_consensus.cc:729] T 05ad7d8fdca64e3da527280ca5be9280 P c0f38c1908414566aa9115bb2cbdd85d [term 7 LEADER]: Becoming Follower/Learner. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:47:21.605795 28932 consensus_queue.cc:260] T 05ad7d8fdca64e3da527280ca5be9280 P c0f38c1908414566aa9115bb2cbdd85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:21.606942 28932 raft_consensus.cc:3037] T 05ad7d8fdca64e3da527280ca5be9280 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Advancing to term 8
I20260809 15:47:21.602639 23282 raft_consensus.cc:1261] T 73f6f44c259547f1b05a3a434616fa54 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 9 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260809 15:47:21.611806 28586 raft_consensus.cc:504] T 294d4bcc1e7a477cb56455581277a60f P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:21.552881 28280 leader_election.cc:288] T cee02fff2c9c48a19f43eb358a89e7b0 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 10 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:21.615502 28586 leader_election.cc:288] T 294d4bcc1e7a477cb56455581277a60f P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 7 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
W20260809 15:47:21.572265 28894 log.cc:924] Time spent T d0721a4403a440f383318a3cea9d1442 P 65c36d164b0540f099d592f38307ee6c: Append to log took a long time: real 0.168s user 0.001s sys 0.000s
I20260809 15:47:21.613233 23276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0da4d5283c8a46878bd39b85e5c71f66" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:47:21.581048 28669 raft_consensus.cc:2726] T 0acc47ce21c3422698d1ef7277c3ed63 P 65c36d164b0540f099d592f38307ee6c [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
W20260809 15:47:21.583241 28916 log.cc:924] Time spent T 2424fd9bd7f14e508fd106cbfda786a4 P 65c36d164b0540f099d592f38307ee6c: Append to log took a long time: real 0.069s user 0.000s sys 0.000s
W20260809 15:47:21.609356 23193 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 538a228ddfb54bc6a36528d5ff07597b, TS 67b3aeadfa7a432abe8af4d292e04ab9 -> TS 61e69695258f48baa984c9d395e8ce3f move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:47:21.617233 23193 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 538a228ddfb54bc6a36528d5ff07597b, TS 67b3aeadfa7a432abe8af4d292e04ab9 -> TS 61e69695258f48baa984c9d395e8ce3f move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:21.587798 28319 leader_election.cc:288] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:21.588164 23228 leader_election.cc:302] T 953eea268e17491ca76f3a555860a512 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c
I20260809 15:47:21.618567 23470 leader_election.cc:302] T cee02fff2c9c48a19f43eb358a89e7b0 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 61e69695258f48baa984c9d395e8ce3f, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:47:21.619288 23228 leader_election.cc:302] T 2db057fd10e149a9895dff6ef8c9f038 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c
I20260809 15:47:21.619690 23470 leader_election.cc:302] T cef3e7277d7d4760824a3436ed4a847d P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:21.620319 23228 leader_election.cc:302] T 0c29937a80724ddb9d07a27534a146f3 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c
W20260809 15:47:21.588727 28896 log.cc:924] Time spent T ba56725e42f74a1baa4c04d9660727f9 P 67b3aeadfa7a432abe8af4d292e04ab9: Append to log took a long time: real 0.163s user 0.000s sys 0.001s
I20260809 15:47:21.569196 28688 consensus_queue.cc:1035] T 538a228ddfb54bc6a36528d5ff07597b P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.020s
I20260809 15:47:21.562109 28353 raft_consensus.cc:2781] T e71a05ba1d4d42bd95c3f85f9313399e P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260809 15:47:21.623001 28353 raft_consensus.cc:482] T e71a05ba1d4d42bd95c3f85f9313399e P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:47:21.623377 28353 raft_consensus.cc:3037] T e71a05ba1d4d42bd95c3f85f9313399e P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Advancing to term 8
I20260809 15:47:21.624292 23396 raft_consensus.cc:1261] T d0721a4403a440f383318a3cea9d1442 P c0f38c1908414566aa9115bb2cbdd85d [term 9 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260809 15:47:21.618885 24455 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "294d4bcc1e7a477cb56455581277a60f" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f"
I20260809 15:47:21.627601 28775 raft_consensus.cc:3032] T 3924b648461e463b9d7925e5671d68ae P 61e69695258f48baa984c9d395e8ce3f [term 10 LEADER]: Stepping down as leader of term 10
I20260809 15:47:21.627959 28775 raft_consensus.cc:729] T 3924b648461e463b9d7925e5671d68ae P 61e69695258f48baa984c9d395e8ce3f [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 61e69695258f48baa984c9d395e8ce3f, State: Running, Role: LEADER
I20260809 15:47:21.623004 23397 raft_consensus.cc:1261] T 297c82ed906d4b48b184985eed3de601 P c0f38c1908414566aa9115bb2cbdd85d [term 10 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260809 15:47:21.628738 28775 consensus_queue.cc:260] T 3924b648461e463b9d7925e5671d68ae P 61e69695258f48baa984c9d395e8ce3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.10, Last appended by leader: 10, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:21.622155 28474 raft_consensus.cc:2726] T cef3e7277d7d4760824a3436ed4a847d P 65c36d164b0540f099d592f38307ee6c [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260809 15:47:21.616757 23276 raft_consensus.cc:2379] T 0da4d5283c8a46878bd39b85e5c71f66 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d in current term 7: Already voted for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in this term.
I20260809 15:47:21.621304 23404 raft_consensus.cc:1261] T d1a134ba33ea4076bbaac0ae460c17ba P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260809 15:47:21.622146 28319 raft_consensus.cc:2781] T 2db057fd10e149a9895dff6ef8c9f038 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Leader election won for term 8
I20260809 15:47:21.622385 23283 raft_consensus.cc:1261] T 297c82ed906d4b48b184985eed3de601 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 10 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260809 15:47:21.621032 28621 raft_consensus.cc:2781] T 953eea268e17491ca76f3a555860a512 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 9 FOLLOWER]: Leader election won for term 9
I20260809 15:47:21.620774 28586 raft_consensus.cc:3037] T cee02fff2c9c48a19f43eb358a89e7b0 P 65c36d164b0540f099d592f38307ee6c [term 9 FOLLOWER]: Advancing to term 10
I20260809 15:47:21.611908 23273 raft_consensus.cc:2454] T c5c2e425a74b4e64ae70b5def750434c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 8.
I20260809 15:47:21.630373 23527 raft_consensus.cc:2454] T f7715d71dbac447b9351fc5eb36c5e80 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 7.
I20260809 15:47:21.642764 23400 raft_consensus.cc:1261] T 3924b648461e463b9d7925e5671d68ae P c0f38c1908414566aa9115bb2cbdd85d [term 11 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260809 15:47:21.644788 23228 leader_election.cc:302] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:47:21.647171 23282 raft_consensus.cc:3032] T d1a134ba33ea4076bbaac0ae460c17ba P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 LEADER]: Stepping down as leader of term 7
I20260809 15:47:21.647205 23286 raft_consensus.cc:1226] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 4. Current term is 5. Ops: []
I20260809 15:47:21.650449 23282 raft_consensus.cc:729] T d1a134ba33ea4076bbaac0ae460c17ba P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Running, Role: LEADER
I20260809 15:47:21.654752 23273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "294d4bcc1e7a477cb56455581277a60f" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:47:21.656348 23280 raft_consensus.cc:1226] T d2d4f749fcfd4fb3bfb8115af3471598 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 9 FOLLOWER]: Rejecting Update request from peer c0f38c1908414566aa9115bb2cbdd85d for earlier term 8. Current term is 9. Ops: [8.5-8.5]
I20260809 15:47:21.657100 23285 raft_consensus.cc:1226] T 00a1881267a14ba8b78c81040f17a014 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 10 FOLLOWER]: Rejecting Update request from peer c0f38c1908414566aa9115bb2cbdd85d for earlier term 9. Current term is 10. Ops: []
I20260809 15:47:21.656383 23228 leader_election.cc:302] T d3be1e356b254d22b995be32f1c12f6f P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d
W20260809 15:47:21.662101 23193 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 0da4d5283c8a46878bd39b85e5c71f66, TS 65c36d164b0540f099d592f38307ee6c -> TS 61e69695258f48baa984c9d395e8ce3f move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:47:21.662405 23193 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 0da4d5283c8a46878bd39b85e5c71f66, TS 65c36d164b0540f099d592f38307ee6c -> TS 61e69695258f48baa984c9d395e8ce3f move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:21.662626 28627 consensus_queue.cc:1035] T 3924b648461e463b9d7925e5671d68ae P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.003s
I20260809 15:47:21.656940 28612 raft_consensus.cc:2781] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260809 15:47:21.666917 28612 raft_consensus.cc:482] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:47:21.667258 28612 raft_consensus.cc:3037] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Advancing to term 6
I20260809 15:47:21.668898 28903 raft_consensus.cc:2726] T a3622f6e3e3641cea1960e3a31498bb7 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260809 15:47:21.648403 28456 consensus_queue.cc:1035] T d0721a4403a440f383318a3cea9d1442 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.023s
I20260809 15:47:21.672513 28406 raft_consensus.cc:2726] T d3be1e356b254d22b995be32f1c12f6f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260809 15:47:21.673403 23282 consensus_queue.cc:260] T d1a134ba33ea4076bbaac0ae460c17ba P 67b3aeadfa7a432abe8af4d292e04ab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:21.654790 28425 consensus_queue.cc:1035] T 297c82ed906d4b48b184985eed3de601 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.020s
I20260809 15:47:21.585565 28560 raft_consensus.cc:504] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:21.675529 28621 raft_consensus.cc:686] T 953eea268e17491ca76f3a555860a512 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 9 LEADER]: Becoming Leader. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Running, Role: LEADER
I20260809 15:47:21.676815 28560 leader_election.cc:288] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:21.658237 28451 consensus_queue.cc:1035] T d1a134ba33ea4076bbaac0ae460c17ba P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.023s
I20260809 15:47:21.680181 24455 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f9d6565aa4c45a3b3f01210d0eb71f7" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f"
I20260809 15:47:21.644306 28416 raft_consensus.cc:2781] T 0c29937a80724ddb9d07a27534a146f3 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 9 FOLLOWER]: Leader election won for term 9
I20260809 15:47:21.661550 23276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ae14a74a26f434db08c652e097385c8" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:47:21.688211 23519 raft_consensus.cc:1226] T cef3e7277d7d4760824a3436ed4a847d P 65c36d164b0540f099d592f38307ee6c [term 9 FOLLOWER]: Rejecting Update request from peer c0f38c1908414566aa9115bb2cbdd85d for earlier term 8. Current term is 9. Ops: []
I20260809 15:47:21.660508 28319 raft_consensus.cc:686] T 2db057fd10e149a9895dff6ef8c9f038 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 LEADER]: Becoming Leader. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Running, Role: LEADER
I20260809 15:47:21.676445 28621 consensus_queue.cc:237] T 953eea268e17491ca76f3a555860a512 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:21.686451 23526 raft_consensus.cc:1226] T 00a1881267a14ba8b78c81040f17a014 P 65c36d164b0540f099d592f38307ee6c [term 10 FOLLOWER]: Rejecting Update request from peer c0f38c1908414566aa9115bb2cbdd85d for earlier term 9. Current term is 10. Ops: []
I20260809 15:47:21.687043 23228 leader_election.cc:302] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:21.688213 23276 raft_consensus.cc:3037] T 6ae14a74a26f434db08c652e097385c8 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Advancing to term 9
I20260809 15:47:21.662969 28430 raft_consensus.cc:2726] T d3e94b7e50754a81b3d72a07476cb7b9 P 65c36d164b0540f099d592f38307ee6c [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260809 15:47:21.694491 23529 raft_consensus.cc:1261] T 73f6f44c259547f1b05a3a434616fa54 P 65c36d164b0540f099d592f38307ee6c [term 9 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260809 15:47:21.697834 23528 raft_consensus.cc:1226] T d2d4f749fcfd4fb3bfb8115af3471598 P 65c36d164b0540f099d592f38307ee6c [term 9 FOLLOWER]: Rejecting Update request from peer c0f38c1908414566aa9115bb2cbdd85d for earlier term 8. Current term is 9. Ops: [8.5-8.5]
I20260809 15:47:21.703593 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee02fff2c9c48a19f43eb358a89e7b0" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:47:21.705175 23285 raft_consensus.cc:1226] T 998c6cee72cf40228c6dba064c0db2da P 67b3aeadfa7a432abe8af4d292e04ab9 [term 9 FOLLOWER]: Rejecting Update request from peer c0f38c1908414566aa9115bb2cbdd85d for earlier term 8. Current term is 9. Ops: []
I20260809 15:47:21.710091 23399 raft_consensus.cc:1226] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Rejecting Update request from peer 61e69695258f48baa984c9d395e8ce3f for earlier term 4. Current term is 5. Ops: []
I20260809 15:47:21.712983 23396 raft_consensus.cc:1226] T 4bcac4b24e7a4448abdc934bba5a624e P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Rejecting Update request from peer 61e69695258f48baa984c9d395e8ce3f for earlier term 4. Current term is 5. Ops: []
I20260809 15:47:21.713616 23402 raft_consensus.cc:1261] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Refusing update from remote peer 61e69695258f48baa984c9d395e8ce3f: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260809 15:47:21.699295 28406 raft_consensus.cc:2726] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260809 15:47:21.715307 28725 raft_consensus.cc:2726] T 0c29937a80724ddb9d07a27534a146f3 P 65c36d164b0540f099d592f38307ee6c [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260809 15:47:21.716068 28784 consensus_queue.cc:1035] T 07fcabb569414bddbd39df6cedd24e64 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260809 15:47:21.716802 23280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f9d6565aa4c45a3b3f01210d0eb71f7" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:47:21.720695 23528 raft_consensus.cc:1226] T 294d4bcc1e7a477cb56455581277a60f P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Rejecting Update request from peer 61e69695258f48baa984c9d395e8ce3f for earlier term 6. Current term is 7. Ops: []
I20260809 15:47:21.699635 28416 raft_consensus.cc:686] T 0c29937a80724ddb9d07a27534a146f3 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 9 LEADER]: Becoming Leader. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Running, Role: LEADER
I20260809 15:47:21.726594 28525 raft_consensus.cc:2726] T 2db057fd10e149a9895dff6ef8c9f038 P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260809 15:47:21.716712 28230 raft_consensus.cc:3032] T 99bdb93503db4de4a1cfb0161ea0c07c P c0f38c1908414566aa9115bb2cbdd85d [term 4 LEADER]: Stepping down as leader of term 4
W20260809 15:47:21.726704 28899 log.cc:924] Time spent T d1a134ba33ea4076bbaac0ae460c17ba P 65c36d164b0540f099d592f38307ee6c: Append to log took a long time: real 0.279s user 0.006s sys 0.003s
I20260809 15:47:21.723699 28406 raft_consensus.cc:482] T 0acc47ce21c3422698d1ef7277c3ed63 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:47:21.743446 28230 raft_consensus.cc:729] T 99bdb93503db4de4a1cfb0161ea0c07c P c0f38c1908414566aa9115bb2cbdd85d [term 4 LEADER]: Becoming Follower/Learner. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:47:21.783691 28532 consensus_queue.cc:1046] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260809 15:47:21.784782 28971 raft_consensus.cc:482] T 0a51c0baaa6e432997c9d3552af4f254 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:21.784999 28353 raft_consensus.cc:504] T e71a05ba1d4d42bd95c3f85f9313399e P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:21.785342 28971 raft_consensus.cc:504] T 0a51c0baaa6e432997c9d3552af4f254 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:21.787467 28416 consensus_queue.cc:237] T 0c29937a80724ddb9d07a27534a146f3 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.3, Last appended by leader: 3, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:21.788254 28230 consensus_queue.cc:260] T 99bdb93503db4de4a1cfb0161ea0c07c P c0f38c1908414566aa9115bb2cbdd85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:21.789305 23280 raft_consensus.cc:1261] T d0721a4403a440f383318a3cea9d1442 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 9 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260809 15:47:21.790483 23283 raft_consensus.cc:3032] T 2424fd9bd7f14e508fd106cbfda786a4 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 LEADER]: Stepping down as leader of term 8
I20260809 15:47:21.790840 23283 raft_consensus.cc:729] T 2424fd9bd7f14e508fd106cbfda786a4 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Running, Role: LEADER
I20260809 15:47:21.756037 28319 consensus_queue.cc:237] T 2db057fd10e149a9895dff6ef8c9f038 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:21.791592 23283 consensus_queue.cc:260] T 2424fd9bd7f14e508fd106cbfda786a4 P 67b3aeadfa7a432abe8af4d292e04ab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 8.4, Last appended by leader: 4, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:21.792510 23283 raft_consensus.cc:3037] T 2424fd9bd7f14e508fd106cbfda786a4 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Advancing to term 9
I20260809 15:47:21.792296 28756 consensus_queue.cc:1046] T 294d4bcc1e7a477cb56455581277a60f P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260809 15:47:21.799095 23280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e71a05ba1d4d42bd95c3f85f9313399e" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:47:21.799571 24453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a51c0baaa6e432997c9d3552af4f254" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:47:21.799729 23280 raft_consensus.cc:3037] T e71a05ba1d4d42bd95c3f85f9313399e P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Advancing to term 8
I20260809 15:47:21.800047 24453 raft_consensus.cc:2454] T 0a51c0baaa6e432997c9d3552af4f254 P 61e69695258f48baa984c9d395e8ce3f [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 8.
I20260809 15:47:21.801630 23528 raft_consensus.cc:1261] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Refusing update from remote peer 61e69695258f48baa984c9d395e8ce3f: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260809 15:47:21.803468 28612 raft_consensus.cc:504] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:21.805226 23283 raft_consensus.cc:1261] T 2424fd9bd7f14e508fd106cbfda786a4 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 9 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 8 index: 4. Preceding OpId from leader: term: 9 index: 4. (term mismatch)
I20260809 15:47:21.787094 28353 leader_election.cc:288] T e71a05ba1d4d42bd95c3f85f9313399e P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 8 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:21.805928 23283 pending_rounds.cc:76] T 2424fd9bd7f14e508fd106cbfda786a4 P 67b3aeadfa7a432abe8af4d292e04ab9: Aborting all ops after (but not including) 3
I20260809 15:47:21.806205 23283 pending_rounds.cc:97] T 2424fd9bd7f14e508fd106cbfda786a4 P 67b3aeadfa7a432abe8af4d292e04ab9: Aborting uncommitted NO_OP operation due to leader change: 8.4
I20260809 15:47:21.795603 28406 raft_consensus.cc:504] T 0acc47ce21c3422698d1ef7277c3ed63 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:21.806407 23283 raft_consensus.cc:2866] T 2424fd9bd7f14e508fd106cbfda786a4 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 15:47:21.807753 28406 leader_election.cc:288] T 0acc47ce21c3422698d1ef7277c3ed63 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
W20260809 15:47:21.812247 23353 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 119196us
I20260809 15:47:21.813660 23276 raft_consensus.cc:2454] T 6ae14a74a26f434db08c652e097385c8 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 9.
I20260809 15:47:21.789428 28230 raft_consensus.cc:3037] T 99bdb93503db4de4a1cfb0161ea0c07c P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Advancing to term 5
I20260809 15:47:21.803350 28577 tablet_copy_client.cc:531] T ba56725e42f74a1baa4c04d9660727f9 P 61e69695258f48baa984c9d395e8ce3f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260809 15:47:21.821341 24448 raft_consensus.cc:1226] T c0741ab44d99439abc3aa3bdd27ee160 P 61e69695258f48baa984c9d395e8ce3f [term 9 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 8. Current term is 9. Ops: []
I20260809 15:47:21.822613 24463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e71a05ba1d4d42bd95c3f85f9313399e" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f"
I20260809 15:47:21.823635 24468 raft_consensus.cc:1261] T c0741ab44d99439abc3aa3bdd27ee160 P 61e69695258f48baa984c9d395e8ce3f [term 9 FOLLOWER]: Refusing update from remote peer 67b3aeadfa7a432abe8af4d292e04ab9: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260809 15:47:21.797708 28971 leader_election.cc:288] T 0a51c0baaa6e432997c9d3552af4f254 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:21.822010 28522 consensus_queue.cc:1046] T cef3e7277d7d4760824a3436ed4a847d P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: INVALID_TERM, Last received: 8.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.012s
W20260809 15:47:21.834817 28958 log.cc:924] Time spent T d3e94b7e50754a81b3d72a07476cb7b9 P 65c36d164b0540f099d592f38307ee6c: Append to log took a long time: real 0.083s user 0.000s sys 0.001s
I20260809 15:47:21.822731 28921 consensus_queue.cc:1046] T 00a1881267a14ba8b78c81040f17a014 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: INVALID_TERM, Last received: 9.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.013s
I20260809 15:47:21.834748 24463 raft_consensus.cc:3037] T e71a05ba1d4d42bd95c3f85f9313399e P 61e69695258f48baa984c9d395e8ce3f [term 6 FOLLOWER]: Advancing to term 8
I20260809 15:47:21.805465 28612 leader_election.cc:288] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 6 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:21.836253 28525 consensus_queue.cc:1046] T c0741ab44d99439abc3aa3bdd27ee160 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.004s
W20260809 15:47:21.840194 23223 consensus_peers.cc:487] T ba56725e42f74a1baa4c04d9660727f9 P 67b3aeadfa7a432abe8af4d292e04ab9 -> Peer 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689): Couldn't send request to peer 61e69695258f48baa984c9d395e8ce3f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20260809 15:47:21.840631 23223 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.68:34689, user_credentials={real_user=slave}} blocked reactor thread for 44103.8us
I20260809 15:47:21.840790 23399 raft_consensus.cc:1261] T 2424fd9bd7f14e508fd106cbfda786a4 P c0f38c1908414566aa9115bb2cbdd85d [term 9 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 7 index: 3. Preceding OpId from leader: term: 9 index: 4. (index mismatch)
I20260809 15:47:21.848176 23276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "538a228ddfb54bc6a36528d5ff07597b" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:21.847587 23399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "538a228ddfb54bc6a36528d5ff07597b" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:21.798504 28758 consensus_queue.cc:1046] T 4bcac4b24e7a4448abdc934bba5a624e P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: INVALID_TERM, Last received: 4.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.039s
I20260809 15:47:21.824728 28586 raft_consensus.cc:2726] T cee02fff2c9c48a19f43eb358a89e7b0 P 65c36d164b0540f099d592f38307ee6c [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260809 15:47:21.841355 23223 leader_election.cc:302] T 0a51c0baaa6e432997c9d3552af4f254 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
W20260809 15:47:21.856040 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 157633us
I20260809 15:47:21.863873 23280 raft_consensus.cc:2454] T e71a05ba1d4d42bd95c3f85f9313399e P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 8.
I20260809 15:47:21.865345 24406 leader_election.cc:302] T e71a05ba1d4d42bd95c3f85f9313399e P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 67b3aeadfa7a432abe8af4d292e04ab9; no voters: c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:21.857328 24463 raft_consensus.cc:2454] T e71a05ba1d4d42bd95c3f85f9313399e P 61e69695258f48baa984c9d395e8ce3f [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0f38c1908414566aa9115bb2cbdd85d in term 8.
I20260809 15:47:21.868372 23345 leader_election.cc:302] T e71a05ba1d4d42bd95c3f85f9313399e P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:47:21.870664 23519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0da4d5283c8a46878bd39b85e5c71f66" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:47:21.871438 23519 raft_consensus.cc:2379] T 0da4d5283c8a46878bd39b85e5c71f66 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 in current term 7: Already voted for candidate c0f38c1908414566aa9115bb2cbdd85d in this term.
I20260809 15:47:21.841583 28695 raft_consensus.cc:482] T 4564dfc6d4094c1b9240b367ccecf3b1 P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:21.879326 23519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ced718c1a5c4da79cb8c1ebd1435b87" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:47:21.879900 23519 raft_consensus.cc:3037] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Advancing to term 6
I20260809 15:47:21.850184 28522 raft_consensus.cc:3032] T cef3e7277d7d4760824a3436ed4a847d P c0f38c1908414566aa9115bb2cbdd85d [term 8 LEADER]: Stepping down as leader of term 8
I20260809 15:47:21.840216 28756 raft_consensus.cc:3032] T 294d4bcc1e7a477cb56455581277a60f P 61e69695258f48baa984c9d395e8ce3f [term 6 LEADER]: Stepping down as leader of term 6
I20260809 15:47:21.881321 23517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee02fff2c9c48a19f43eb358a89e7b0" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:47:21.888547 23528 raft_consensus.cc:1261] T 4564dfc6d4094c1b9240b367ccecf3b1 P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Refusing update from remote peer 67b3aeadfa7a432abe8af4d292e04ab9: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260809 15:47:21.889008 23522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ae14a74a26f434db08c652e097385c8" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:21.895130 23522 raft_consensus.cc:2379] T 6ae14a74a26f434db08c652e097385c8 P 65c36d164b0540f099d592f38307ee6c [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 in current term 9: Already voted for candidate c0f38c1908414566aa9115bb2cbdd85d in this term.
I20260809 15:47:21.844787 28719 leader_election.cc:288] T 538a228ddfb54bc6a36528d5ff07597b P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:47:21.891608 23529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "538a228ddfb54bc6a36528d5ff07597b" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:21.893204 23520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0acc47ce21c3422698d1ef7277c3ed63" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 10 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:21.897974 23399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "538a228ddfb54bc6a36528d5ff07597b" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:21.836499 28745 consensus_queue.cc:1035] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.040s
I20260809 15:47:21.902191 23520 raft_consensus.cc:2379] T 0acc47ce21c3422698d1ef7277c3ed63 P 65c36d164b0540f099d592f38307ee6c [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 in current term 10: Already voted for candidate 65c36d164b0540f099d592f38307ee6c in this term.
I20260809 15:47:21.902782 23394 raft_consensus.cc:1226] T 2424fd9bd7f14e508fd106cbfda786a4 P c0f38c1908414566aa9115bb2cbdd85d [term 9 FOLLOWER]: Rejecting Update request from peer 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 8. Current term is 9. Ops: []
I20260809 15:47:21.904930 23280 raft_consensus.cc:1261] T e14d118b2eaa42aa8a7cd4025aba7128 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 9 FOLLOWER]: Refusing update from remote peer 61e69695258f48baa984c9d395e8ce3f: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 9 index: 10. (index mismatch)
I20260809 15:47:21.905439 23394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "907bacfa79d04a55a68c8dbe60056462" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:21.906220 23394 raft_consensus.cc:2396] T 907bacfa79d04a55a68c8dbe60056462 P c0f38c1908414566aa9115bb2cbdd85d [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 for term 10 because replica has last-logged OpId of term: 8 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
I20260809 15:47:21.848385 28627 consensus_queue.cc:1035] T 2424fd9bd7f14e508fd106cbfda786a4 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.045s
I20260809 15:47:21.906463 23521 raft_consensus.cc:1226] T 3924b648461e463b9d7925e5671d68ae P 65c36d164b0540f099d592f38307ee6c [term 11 LEADER]: Rejecting Update request from peer 61e69695258f48baa984c9d395e8ce3f for earlier term 10. Current term is 11. Ops: [10.10-10.10]
I20260809 15:47:21.912002 23519 raft_consensus.cc:2454] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 6.
I20260809 15:47:21.906909 24406 leader_election.cc:302] T 563daf42d7784db195ea99c3cb21b370 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f; no voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:21.907469 23517 raft_consensus.cc:1226] T d1a134ba33ea4076bbaac0ae460c17ba P 65c36d164b0540f099d592f38307ee6c [term 8 LEADER]: Rejecting Update request from peer 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 7. Current term is 8. Ops: []
I20260809 15:47:21.908720 23277 raft_consensus.cc:1226] T 28e17dcb5efe45b8b6a8c42da03b4819 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 9 FOLLOWER]: Rejecting Update request from peer c0f38c1908414566aa9115bb2cbdd85d for earlier term 8. Current term is 9. Ops: []
I20260809 15:47:21.880700 28612 raft_consensus.cc:2781] T 0a51c0baaa6e432997c9d3552af4f254 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260809 15:47:21.907054 23228 leader_election.cc:302] T 4b30f28311964a0b9149a50495e83d8f P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters: c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:21.889518 28522 raft_consensus.cc:729] T cef3e7277d7d4760824a3436ed4a847d P c0f38c1908414566aa9115bb2cbdd85d [term 8 LEADER]: Becoming Follower/Learner. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:47:21.894275 28756 raft_consensus.cc:729] T 294d4bcc1e7a477cb56455581277a60f P 61e69695258f48baa984c9d395e8ce3f [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 61e69695258f48baa984c9d395e8ce3f, State: Running, Role: LEADER
I20260809 15:47:21.916085 23393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0da4d5283c8a46878bd39b85e5c71f66" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:47:21.916424 23280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee02fff2c9c48a19f43eb358a89e7b0" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:47:21.848912 28935 consensus_queue.cc:1035] T 73f6f44c259547f1b05a3a434616fa54 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.050s
I20260809 15:47:21.916857 23394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ae14a74a26f434db08c652e097385c8" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:21.919935 23399 raft_consensus.cc:1261] T ba56725e42f74a1baa4c04d9660727f9 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Refusing update from remote peer 67b3aeadfa7a432abe8af4d292e04ab9: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260809 15:47:21.921228 23228 leader_election.cc:302] T 05ad7d8fdca64e3da527280ca5be9280 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:21.921737 23280 raft_consensus.cc:2379] T cee02fff2c9c48a19f43eb358a89e7b0 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 61e69695258f48baa984c9d395e8ce3f in current term 10: Already voted for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in this term.
I20260809 15:47:21.899463 28758 raft_consensus.cc:2741] T e71a05ba1d4d42bd95c3f85f9313399e P 61e69695258f48baa984c9d395e8ce3f [term 8 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
I20260809 15:47:21.923763 23228 leader_election.cc:302] T 16ca412a0dae4feebe85dd01bf1a9173 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c
I20260809 15:47:21.907052 28918 raft_consensus.cc:2781] T e71a05ba1d4d42bd95c3f85f9313399e P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Leader election won for term 8
I20260809 15:47:21.925626 23228 leader_election.cc:302] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c
I20260809 15:47:21.925884 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ced718c1a5c4da79cb8c1ebd1435b87" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:47:21.926414 23402 raft_consensus.cc:3037] T 6ced718c1a5c4da79cb8c1ebd1435b87 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Advancing to term 6
I20260809 15:47:21.926700 23228 leader_election.cc:302] T 0c48a3049897452383257aa757834fb5 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c
I20260809 15:47:21.845703 28447 consensus_queue.cc:260] T d2d4f749fcfd4fb3bfb8115af3471598 P c0f38c1908414566aa9115bb2cbdd85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:21.927502 23395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a51c0baaa6e432997c9d3552af4f254" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:21.928084 23395 raft_consensus.cc:2454] T 0a51c0baaa6e432997c9d3552af4f254 P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 8.
I20260809 15:47:21.928270 23228 leader_election.cc:302] T 28e17dcb5efe45b8b6a8c42da03b4819 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:21.929414 23228 leader_election.cc:302] T f9e7cadcd24d45fcbfa84209225e8f2d P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:21.914559 28612 raft_consensus.cc:482] T 0a51c0baaa6e432997c9d3552af4f254 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:47:21.887579 28695 raft_consensus.cc:504] T 4564dfc6d4094c1b9240b367ccecf3b1 P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:21.947790 28756 consensus_queue.cc:260] T 294d4bcc1e7a477cb56455581277a60f P 61e69695258f48baa984c9d395e8ce3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:21.949071 28756 raft_consensus.cc:3037] T 294d4bcc1e7a477cb56455581277a60f P 61e69695258f48baa984c9d395e8ce3f [term 6 FOLLOWER]: Advancing to term 7
I20260809 15:47:21.951710 28522 consensus_queue.cc:260] T cef3e7277d7d4760824a3436ed4a847d P c0f38c1908414566aa9115bb2cbdd85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.4, Last appended by leader: 4, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:21.952907 28522 raft_consensus.cc:3037] T cef3e7277d7d4760824a3436ed4a847d P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Advancing to term 9
I20260809 15:47:21.955379 28447 raft_consensus.cc:3037] T d2d4f749fcfd4fb3bfb8115af3471598 P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Advancing to term 9
I20260809 15:47:21.962430 23276 raft_consensus.cc:1261] T 0da4d5283c8a46878bd39b85e5c71f66 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260809 15:47:21.967646 28873 raft_consensus.cc:482] T 3296ef50be254fc29cfc77d22a8f7a23 P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:21.971385 28335 raft_consensus.cc:2726] T 563daf42d7784db195ea99c3cb21b370 P 61e69695258f48baa984c9d395e8ce3f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260809 15:47:21.975445 28546 raft_consensus.cc:3032] T 00a1881267a14ba8b78c81040f17a014 P c0f38c1908414566aa9115bb2cbdd85d [term 9 LEADER]: Stepping down as leader of term 9
I20260809 15:47:21.968170 28873 raft_consensus.cc:504] T 3296ef50be254fc29cfc77d22a8f7a23 P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:21.975850 28546 raft_consensus.cc:729] T 00a1881267a14ba8b78c81040f17a014 P c0f38c1908414566aa9115bb2cbdd85d [term 9 LEADER]: Becoming Follower/Learner. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:47:21.976698 28546 consensus_queue.cc:260] T 00a1881267a14ba8b78c81040f17a014 P c0f38c1908414566aa9115bb2cbdd85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:21.977798 28546 raft_consensus.cc:3037] T 00a1881267a14ba8b78c81040f17a014 P c0f38c1908414566aa9115bb2cbdd85d [term 9 FOLLOWER]: Advancing to term 10
I20260809 15:47:21.978224 28873 leader_election.cc:288] T 3296ef50be254fc29cfc77d22a8f7a23 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:47:21.979562 23392 raft_consensus.cc:2396] T d2d4f749fcfd4fb3bfb8115af3471598 P c0f38c1908414566aa9115bb2cbdd85d [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 for term 9 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20260809 15:47:21.942140 28406 raft_consensus.cc:2726] T 0c48a3049897452383257aa757834fb5 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260809 15:47:21.930059 28887 raft_consensus.cc:2726] T 28e17dcb5efe45b8b6a8c42da03b4819 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
W20260809 15:47:21.984647 28959 log.cc:924] Time spent T 0da4d5283c8a46878bd39b85e5c71f66 P c0f38c1908414566aa9115bb2cbdd85d: Append to log took a long time: real 0.163s user 0.001s sys 0.000s
W20260809 15:47:21.985180 28353 consensus_queue.cc:1159] T d2d4f749fcfd4fb3bfb8115af3471598 P c0f38c1908414566aa9115bb2cbdd85d [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "65c36d164b0540f099d592f38307ee6c" responder_term: 9 status { last_received { term: 6 index: 4 } last_committed_idx: 4 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer c0f38c1908414566aa9115bb2cbdd85d for earlier term 8. Current term is 9. Ops: [8.5-8.5]" } } last_received_current_leader { term: 0 index: 0 } }
I20260809 15:47:21.986200 23393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0acc47ce21c3422698d1ef7277c3ed63" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 10 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:21.986812 23393 raft_consensus.cc:2454] T 0acc47ce21c3422698d1ef7277c3ed63 P c0f38c1908414566aa9115bb2cbdd85d [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 10.
I20260809 15:47:21.930498 28319 raft_consensus.cc:2726] T 16ca412a0dae4feebe85dd01bf1a9173 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260809 15:47:21.927364 28874 raft_consensus.cc:2726] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260809 15:47:21.986122 28988 raft_consensus.cc:2726] T 05ad7d8fdca64e3da527280ca5be9280 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260809 15:47:21.987600 28525 consensus_queue.cc:1035] T 297c82ed906d4b48b184985eed3de601 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260809 15:47:21.988638 28586 consensus_queue.cc:1046] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 65c36d164b0540f099d592f38307ee6c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260809 15:47:21.986397 28926 log.cc:924] Time spent T 30c1740ef6874df0a332fbd6fd4b03ce P c0f38c1908414566aa9115bb2cbdd85d: Append to log took a long time: real 0.064s user 0.000s sys 0.000s
I20260809 15:47:21.987186 23470 leader_election.cc:302] T 294d4bcc1e7a477cb56455581277a60f P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 61e69695258f48baa984c9d395e8ce3f, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:47:21.887912 28973 consensus_queue.cc:1035] T c0741ab44d99439abc3aa3bdd27ee160 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.042s
I20260809 15:47:21.924450 28971 raft_consensus.cc:2741] T 4b30f28311964a0b9149a50495e83d8f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20260809 15:47:21.996253 23394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3296ef50be254fc29cfc77d22a8f7a23" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:21.996181 23470 leader_election.cc:302] T 538a228ddfb54bc6a36528d5ff07597b P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:21.997735 28335 raft_consensus.cc:482] T 7660dfa2ef694927b8eda4100a2bcd9b P 61e69695258f48baa984c9d395e8ce3f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
W20260809 15:47:21.997747 24406 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 96715.4us
I20260809 15:47:22.001055 23394 raft_consensus.cc:2454] T 3296ef50be254fc29cfc77d22a8f7a23 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 3.
I20260809 15:47:21.992466 28970 raft_consensus.cc:2726] T f9e7cadcd24d45fcbfa84209225e8f2d P 67b3aeadfa7a432abe8af4d292e04ab9 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260809 15:47:22.011111 23351 leader_election.cc:302] T 8e4ffcf83c234ae7910ddef3ec0b4bda P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 61e69695258f48baa984c9d395e8ce3f, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:47:22.012028 28425 consensus_queue.cc:1046] T c0741ab44d99439abc3aa3bdd27ee160 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 6, Time since last communication: 0.000s
I20260809 15:47:22.012528 23392 raft_consensus.cc:1261] T 472816994d814cb198b69e5248697440 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Refusing update from remote peer 61e69695258f48baa984c9d395e8ce3f: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260809 15:47:21.992002 28612 raft_consensus.cc:3037] T 0a51c0baaa6e432997c9d3552af4f254 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Advancing to term 9
I20260809 15:47:21.989841 28918 raft_consensus.cc:686] T e71a05ba1d4d42bd95c3f85f9313399e P c0f38c1908414566aa9115bb2cbdd85d [term 8 LEADER]: Becoming Leader. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:47:22.021095 24410 leader_election.cc:302] T 3296ef50be254fc29cfc77d22a8f7a23 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:47:22.001122 28586 raft_consensus.cc:2726] T 294d4bcc1e7a477cb56455581277a60f P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260809 15:47:22.009443 28669 raft_consensus.cc:2726] T 538a228ddfb54bc6a36528d5ff07597b P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260809 15:47:22.058972 23395 raft_consensus.cc:1226] T d1a134ba33ea4076bbaac0ae460c17ba P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Rejecting Update request from peer 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 7. Current term is 8. Ops: []
I20260809 15:47:22.012719 28425 consensus_queue.cc:1225] T c0741ab44d99439abc3aa3bdd27ee160 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Peer 67b3aeadfa7a432abe8af4d292e04ab9 log is divergent from this leader: its last log entry 9.9 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260809 15:47:22.008169 28451 raft_consensus.cc:3032] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 65c36d164b0540f099d592f38307ee6c [term 4 LEADER]: Stepping down as leader of term 4
I20260809 15:47:21.990015 28758 consensus_queue.cc:1035] T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.030s
W20260809 15:47:22.097620 24410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 71112us
I20260809 15:47:21.986805 28627 consensus_queue.cc:1035] T 30c1740ef6874df0a332fbd6fd4b03ce P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.033s
I20260809 15:47:22.009792 28725 consensus_queue.cc:1035] T d0721a4403a440f383318a3cea9d1442 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.090s
W20260809 15:47:22.110126 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 99092.8us
I20260809 15:47:22.111495 23351 leader_election.cc:302] T 3296ef50be254fc29cfc77d22a8f7a23 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 61e69695258f48baa984c9d395e8ce3f, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:47:22.014503 28719 consensus_queue.cc:1035] T 2424fd9bd7f14e508fd106cbfda786a4 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.092s
W20260809 15:47:22.102499 28994 log.cc:924] Time spent T 538a228ddfb54bc6a36528d5ff07597b P 67b3aeadfa7a432abe8af4d292e04ab9: Append to log took a long time: real 0.084s user 0.000s sys 0.000s
I20260809 15:47:22.086535 28532 raft_consensus.cc:2781] T 3296ef50be254fc29cfc77d22a8f7a23 P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260809 15:47:22.100919 28873 raft_consensus.cc:482] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:22.098065 28932 raft_consensus.cc:2726] T 8e4ffcf83c234ae7910ddef3ec0b4bda P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260809 15:47:22.113193 23276 raft_consensus.cc:1226] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 4. Current term is 5. Ops: []
I20260809 15:47:22.014746 28335 raft_consensus.cc:504] T 7660dfa2ef694927b8eda4100a2bcd9b P 61e69695258f48baa984c9d395e8ce3f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
W20260809 15:47:22.107213 28859 log.cc:924] Time spent T f9e7cadcd24d45fcbfa84209225e8f2d P 65c36d164b0540f099d592f38307ee6c: Append to log took a long time: real 0.084s user 0.000s sys 0.000s
I20260809 15:47:22.100847 28874 raft_consensus.cc:482] T 73f6f44c259547f1b05a3a434616fa54 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:22.099942 28451 raft_consensus.cc:729] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 65c36d164b0540f099d592f38307ee6c [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
W20260809 15:47:22.124842 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 195478us
I20260809 15:47:22.126724 23351 leader_election.cc:302] T c5c2e425a74b4e64ae70b5def750434c P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c
I20260809 15:47:22.136430 23228 leader_election.cc:302] T 998c6cee72cf40228c6dba064c0db2da P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:22.116760 28532 raft_consensus.cc:482] T 3296ef50be254fc29cfc77d22a8f7a23 P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:47:22.144868 28970 raft_consensus.cc:2726] T 998c6cee72cf40228c6dba064c0db2da P 67b3aeadfa7a432abe8af4d292e04ab9 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260809 15:47:22.145063 23228 leader_election.cc:302] T 00a1881267a14ba8b78c81040f17a014 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:22.145776 28970 raft_consensus.cc:2726] T 00a1881267a14ba8b78c81040f17a014 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260809 15:47:22.147241 23228 leader_election.cc:302] T ae6d2c08a7a94f58ac8dd698e3fd456c P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:22.147113 28988 consensus_queue.cc:1035] T 07fcabb569414bddbd39df6cedd24e64 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260809 15:47:22.148448 28988 raft_consensus.cc:2726] T ae6d2c08a7a94f58ac8dd698e3fd456c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260809 15:47:22.148573 23228 leader_election.cc:302] T cee02fff2c9c48a19f43eb358a89e7b0 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c
I20260809 15:47:22.153311 23228 leader_election.cc:302] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:47:22.134845 28936 raft_consensus.cc:2726] T 3296ef50be254fc29cfc77d22a8f7a23 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260809 15:47:22.141059 29008 raft_consensus.cc:482] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 61e69695258f48baa984c9d395e8ce3f [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:22.149251 28532 raft_consensus.cc:3037] T 3296ef50be254fc29cfc77d22a8f7a23 P 61e69695258f48baa984c9d395e8ce3f [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:47:22.170289 23402 raft_consensus.cc:2454] T 6ced718c1a5c4da79cb8c1ebd1435b87 P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 6.
I20260809 15:47:22.170778 29008 raft_consensus.cc:504] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 61e69695258f48baa984c9d395e8ce3f [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:22.133898 28425 consensus_queue.cc:1046] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 65c36d164b0540f099d592f38307ee6c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.025s
I20260809 15:47:22.162159 28973 raft_consensus.cc:2781] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Leader election won for term 6
I20260809 15:47:22.118891 29019 consensus_queue.cc:1035] T 472816994d814cb198b69e5248697440 P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.062s
I20260809 15:47:22.181886 23396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7660dfa2ef694927b8eda4100a2bcd9b" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:22.193360 23228 leader_election.cc:302] T 538a228ddfb54bc6a36528d5ff07597b P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:22.193905 23396 raft_consensus.cc:2379] T 7660dfa2ef694927b8eda4100a2bcd9b P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 61e69695258f48baa984c9d395e8ce3f in current term 7: Already voted for candidate c0f38c1908414566aa9115bb2cbdd85d in this term.
I20260809 15:47:22.122901 28873 raft_consensus.cc:504] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:22.179204 28335 leader_election.cc:288] T 7660dfa2ef694927b8eda4100a2bcd9b P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:22.141348 28447 consensus_queue.cc:1035] T 73f6f44c259547f1b05a3a434616fa54 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.038s
I20260809 15:47:22.139662 28874 raft_consensus.cc:504] T 73f6f44c259547f1b05a3a434616fa54 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:22.190434 23529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7660dfa2ef694927b8eda4100a2bcd9b" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:22.156236 28918 consensus_queue.cc:237] T e71a05ba1d4d42bd95c3f85f9313399e P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:22.155419 28970 raft_consensus.cc:2726] T cee02fff2c9c48a19f43eb358a89e7b0 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260809 15:47:22.155306 28988 consensus_queue.cc:1035] T 4564dfc6d4094c1b9240b367ccecf3b1 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.037s
I20260809 15:47:22.194708 23228 leader_election.cc:302] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:22.198462 23400 raft_consensus.cc:1261] T 4564dfc6d4094c1b9240b367ccecf3b1 P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Refusing update from remote peer 67b3aeadfa7a432abe8af4d292e04ab9: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260809 15:47:22.200515 23396 raft_consensus.cc:1226] T 4bcac4b24e7a4448abdc934bba5a624e P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Rejecting Update request from peer 61e69695258f48baa984c9d395e8ce3f for earlier term 4. Current term is 5. Ops: []
I20260809 15:47:22.209400 23351 leader_election.cc:302] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0f38c1908414566aa9115bb2cbdd85d; no voters: 61e69695258f48baa984c9d395e8ce3f, 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:47:22.158620 28695 leader_election.cc:288] T 4564dfc6d4094c1b9240b367ccecf3b1 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:22.205755 23517 raft_consensus.cc:1261] T ba56725e42f74a1baa4c04d9660727f9 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Refusing update from remote peer 67b3aeadfa7a432abe8af4d292e04ab9: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260809 15:47:22.157047 27957 raft_consensus.cc:2781] T c5c2e425a74b4e64ae70b5def750434c P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Leader election won for term 8
I20260809 15:47:22.217836 23517 raft_consensus.cc:1261] T 213543cdbe814c89a17a963469922c1b P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260809 15:47:22.218601 23521 raft_consensus.cc:3032] T c0741ab44d99439abc3aa3bdd27ee160 P 65c36d164b0540f099d592f38307ee6c [term 8 LEADER]: Stepping down as leader of term 8
I20260809 15:47:22.218922 23521 raft_consensus.cc:729] T c0741ab44d99439abc3aa3bdd27ee160 P 65c36d164b0540f099d592f38307ee6c [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:47:22.205768 23529 raft_consensus.cc:2454] T 7660dfa2ef694927b8eda4100a2bcd9b P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 7.
I20260809 15:47:22.212301 23228 leader_election.cc:302] T 0da4d5283c8a46878bd39b85e5c71f66 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:22.213866 23399 raft_consensus.cc:1261] T 2db057fd10e149a9895dff6ef8c9f038 P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Refusing update from remote peer 67b3aeadfa7a432abe8af4d292e04ab9: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260809 15:47:22.224256 24468 raft_consensus.cc:1261] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 61e69695258f48baa984c9d395e8ce3f [term 10 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260809 15:47:22.199491 28971 raft_consensus.cc:2726] T 538a228ddfb54bc6a36528d5ff07597b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260809 15:47:22.208343 28973 raft_consensus.cc:686] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 LEADER]: Becoming Leader. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Running, Role: LEADER
I20260809 15:47:22.258926 23529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4564dfc6d4094c1b9240b367ccecf3b1" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:22.259418 23522 raft_consensus.cc:1261] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 65c36d164b0540f099d592f38307ee6c [term 10 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260809 15:47:22.269826 23529 raft_consensus.cc:2396] T 4564dfc6d4094c1b9240b367ccecf3b1 P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c0f38c1908414566aa9115bb2cbdd85d for term 9 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20260809 15:47:22.260044 23228 leader_election.cc:302] T 6ae14a74a26f434db08c652e097385c8 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:22.272120 23228 leader_election.cc:302] T d2d4f749fcfd4fb3bfb8115af3471598 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
W20260809 15:47:22.256654 28592 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.669s user 0.011s sys 0.015s
I20260809 15:47:22.266551 23528 raft_consensus.cc:1261] T 0da4d5283c8a46878bd39b85e5c71f66 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260809 15:47:22.223061 28319 raft_consensus.cc:2726] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260809 15:47:22.265796 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 56476.4us
I20260809 15:47:22.205716 28935 consensus_queue.cc:1046] T 998c6cee72cf40228c6dba064c0db2da P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: INVALID_TERM, Last received: 8.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.058s
I20260809 15:47:22.282528 23526 raft_consensus.cc:2454] T cee02fff2c9c48a19f43eb358a89e7b0 P 65c36d164b0540f099d592f38307ee6c [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 10.
I20260809 15:47:22.285065 23521 consensus_queue.cc:260] T c0741ab44d99439abc3aa3bdd27ee160 P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
W20260809 15:47:22.258865 24410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 48303.3us
I20260809 15:47:22.266785 27957 raft_consensus.cc:686] T c5c2e425a74b4e64ae70b5def750434c P c0f38c1908414566aa9115bb2cbdd85d [term 8 LEADER]: Becoming Leader. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:47:22.297459 23521 raft_consensus.cc:3037] T c0741ab44d99439abc3aa3bdd27ee160 P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Advancing to term 9
W20260809 15:47:22.280966 28592 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.731s user 0.011s sys 0.016s
I20260809 15:47:22.259158 28695 raft_consensus.cc:2726] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260809 15:47:22.301942 23228 leader_election.cc:302] T 0acc47ce21c3422698d1ef7277c3ed63 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters: 65c36d164b0540f099d592f38307ee6c
I20260809 15:47:22.259487 29024 raft_consensus.cc:2726] T 0da4d5283c8a46878bd39b85e5c71f66 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260809 15:47:22.279615 28718 raft_consensus.cc:2726] T 6ae14a74a26f434db08c652e097385c8 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260809 15:47:22.294203 23392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a823d35f4f348d1b52ba9e6d80b00d2" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:22.222834 28996 consensus_queue.cc:1035] T e14d118b2eaa42aa8a7cd4025aba7128 P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.070s
I20260809 15:47:22.287793 28971 raft_consensus.cc:2726] T d2d4f749fcfd4fb3bfb8115af3471598 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260809 15:47:22.272086 28887 consensus_queue.cc:1035] T ba56725e42f74a1baa4c04d9660727f9 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.031s
I20260809 15:47:22.341198 23528 raft_consensus.cc:1261] T 2db057fd10e149a9895dff6ef8c9f038 P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Refusing update from remote peer 67b3aeadfa7a432abe8af4d292e04ab9: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260809 15:47:22.343075 23285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3296ef50be254fc29cfc77d22a8f7a23" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
W20260809 15:47:22.354049 23472 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 48559.3us
I20260809 15:47:22.271083 29028 consensus_queue.cc:1046] T 4bcac4b24e7a4448abdc934bba5a624e P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: INVALID_TERM, Last received: 4.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.040s
I20260809 15:47:22.340318 23522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a823d35f4f348d1b52ba9e6d80b00d2" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:22.357905 23522 raft_consensus.cc:2454] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 65c36d164b0540f099d592f38307ee6c [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 10.
I20260809 15:47:22.292989 28230 consensus_queue.cc:1035] T 213543cdbe814c89a17a963469922c1b P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.046s
I20260809 15:47:22.306572 28988 raft_consensus.cc:2781] T 0acc47ce21c3422698d1ef7277c3ed63 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260809 15:47:22.310140 29008 leader_election.cc:288] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
W20260809 15:47:22.367085 29000 log.cc:924] Time spent T d1a134ba33ea4076bbaac0ae460c17ba P c0f38c1908414566aa9115bb2cbdd85d: Append to log took a long time: real 0.058s user 0.000s sys 0.000s
W20260809 15:47:22.368571 29030 log.cc:924] Time spent T 297c82ed906d4b48b184985eed3de601 P 67b3aeadfa7a432abe8af4d292e04ab9: Append to log took a long time: real 0.088s user 0.000s sys 0.000s
I20260809 15:47:22.375420 23400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73f6f44c259547f1b05a3a434616fa54" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:22.310498 28319 consensus_queue.cc:1035] T 4564dfc6d4094c1b9240b367ccecf3b1 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.060s
I20260809 15:47:22.397557 27957 consensus_queue.cc:237] T c5c2e425a74b4e64ae70b5def750434c P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:22.310716 29039 consensus_queue.cc:1035] T 2a823d35f4f348d1b52ba9e6d80b00d2 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.094s
I20260809 15:47:22.398676 29024 raft_consensus.cc:482] T 99bdb93503db4de4a1cfb0161ea0c07c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:47:22.336226 28932 consensus_queue.cc:1035] T 0da4d5283c8a46878bd39b85e5c71f66 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.064s
I20260809 15:47:22.400110 28874 leader_election.cc:288] T 73f6f44c259547f1b05a3a434616fa54 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:22.401050 23520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73f6f44c259547f1b05a3a434616fa54" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:22.361395 28973 consensus_queue.cc:237] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:22.404309 28756 raft_consensus.cc:482] T 3d880de657e544468c82157769622a26 P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:22.404814 24448 raft_consensus.cc:1226] T c0741ab44d99439abc3aa3bdd27ee160 P 61e69695258f48baa984c9d395e8ce3f [term 9 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 8. Current term is 9. Ops: []
W20260809 15:47:22.366163 28894 log.cc:924] Time spent T d0721a4403a440f383318a3cea9d1442 P 65c36d164b0540f099d592f38307ee6c: Append to log took a long time: real 0.054s user 0.000s sys 0.000s
I20260809 15:47:22.430209 23521 raft_consensus.cc:1261] T c0741ab44d99439abc3aa3bdd27ee160 P 65c36d164b0540f099d592f38307ee6c [term 9 FOLLOWER]: Refusing update from remote peer 67b3aeadfa7a432abe8af4d292e04ab9: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260809 15:47:22.445641 23527 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56770 (request call id 1515) took 1176 ms. Trace:
I20260809 15:47:22.311408 28406 consensus_queue.cc:1035] T 2db057fd10e149a9895dff6ef8c9f038 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.065s
I20260809 15:47:22.365880 28988 raft_consensus.cc:482] T 0acc47ce21c3422698d1ef7277c3ed63 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 9 FOLLOWER]: Starting leader election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:47:22.355414 28546 consensus_queue.cc:1046] T 28e17dcb5efe45b8b6a8c42da03b4819 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.091s
W20260809 15:47:22.454105 23353 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 101434us
W20260809 15:47:22.504921 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 168100us
W20260809 15:47:22.519691 23472 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 141801us
I20260809 15:47:22.510975 23527 rpcz_store.cc:276] 0809 15:47:21.269584 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:21.269900 (+ 316us) service_pool.cc:227] Handling call
0809 15:47:22.445479 (+1175579us) spinlock_profiling.cc:243] Waited 337 ms on lock 0x7b540016d854. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e92242ae 00007f34ee161c27 00007f34e7849c04 00007f34e7849b92 00007f34e7849ac1 00007f34e7849a3c 00007f34e7848cf2 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:47:22.445584 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":776070016}
I20260809 15:47:22.531868 28430 raft_consensus.cc:482] T ba56725e42f74a1baa4c04d9660727f9 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:22.532351 28430 raft_consensus.cc:504] T ba56725e42f74a1baa4c04d9660727f9 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } attrs { replace: true } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } }
I20260809 15:47:22.514750 28592 consensus_queue.cc:1035] T 0da4d5283c8a46878bd39b85e5c71f66 P c0f38c1908414566aa9115bb2cbdd85d [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.019s
I20260809 15:47:22.497073 28887 raft_consensus.cc:482] T 3d880de657e544468c82157769622a26 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:22.497548 28718 raft_consensus.cc:482] T 213543cdbe814c89a17a963469922c1b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:22.456038 29031 raft_consensus.cc:482] T 07fcabb569414bddbd39df6cedd24e64 P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
I20260809 15:47:22.456183 28919 raft_consensus.cc:482] T 907bacfa79d04a55a68c8dbe60056462 P 61e69695258f48baa984c9d395e8ce3f [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 15:47:22.514680 28462 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 9 is higher.
I20260809 15:47:22.512892 28988 raft_consensus.cc:3037] T 0acc47ce21c3422698d1ef7277c3ed63 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 9 FOLLOWER]: Advancing to term 10
I20260809 15:47:22.546635 28919 raft_consensus.cc:504] T 907bacfa79d04a55a68c8dbe60056462 P 61e69695258f48baa984c9d395e8ce3f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:22.460549 28775 raft_consensus.cc:3037] T 3924b648461e463b9d7925e5671d68ae P 61e69695258f48baa984c9d395e8ce3f [term 10 FOLLOWER]: Advancing to term 11
I20260809 15:47:22.571920 23400 raft_consensus.cc:1226] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Rejecting Update request from peer 61e69695258f48baa984c9d395e8ce3f for earlier term 4. Current term is 5. Ops: []
I20260809 15:47:22.514662 28756 raft_consensus.cc:504] T 3d880de657e544468c82157769622a26 P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:22.506289 28612 raft_consensus.cc:504] T 0a51c0baaa6e432997c9d3552af4f254 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:22.582077 23285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba56725e42f74a1baa4c04d9660727f9" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:22.515576 29024 raft_consensus.cc:504] T 99bdb93503db4de4a1cfb0161ea0c07c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:22.574617 28319 raft_consensus.cc:482] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
I20260809 15:47:22.591363 23527 raft_consensus.cc:1261] T 0c29937a80724ddb9d07a27534a146f3 P 65c36d164b0540f099d592f38307ee6c [term 9 FOLLOWER]: Refusing update from remote peer 67b3aeadfa7a432abe8af4d292e04ab9: Log matching property violated. Preceding OpId in replica: term: 7 index: 3. Preceding OpId from leader: term: 9 index: 4. (index mismatch)
I20260809 15:47:22.558738 28718 raft_consensus.cc:504] T 213543cdbe814c89a17a963469922c1b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:22.591854 23400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "907bacfa79d04a55a68c8dbe60056462" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:22.599959 23400 raft_consensus.cc:2454] T 907bacfa79d04a55a68c8dbe60056462 P c0f38c1908414566aa9115bb2cbdd85d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 9.
I20260809 15:47:22.543071 28887 raft_consensus.cc:504] T 3d880de657e544468c82157769622a26 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:22.600775 23396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba56725e42f74a1baa4c04d9660727f9" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:22.602813 24410 leader_election.cc:302] T 907bacfa79d04a55a68c8dbe60056462 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:47:22.585107 28802 consensus_queue.cc:1046] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.015s
I20260809 15:47:22.592595 28919 leader_election.cc:288] T 907bacfa79d04a55a68c8dbe60056462 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:47:22.556219 29031 raft_consensus.cc:504] T 07fcabb569414bddbd39df6cedd24e64 P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:22.621393 23400 raft_consensus.cc:1261] T 0c29937a80724ddb9d07a27534a146f3 P c0f38c1908414566aa9115bb2cbdd85d [term 9 FOLLOWER]: Refusing update from remote peer 67b3aeadfa7a432abe8af4d292e04ab9: Log matching property violated. Preceding OpId in replica: term: 5 index: 2. Preceding OpId from leader: term: 9 index: 4. (index mismatch)
I20260809 15:47:22.601135 28430 leader_election.cc:288] T ba56725e42f74a1baa4c04d9660727f9 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
W20260809 15:47:22.624073 29036 log.cc:924] Time spent T 5ad8a3b25d7b49e5a2142d5709edd0c7 P 65c36d164b0540f099d592f38307ee6c: Append to log took a long time: real 0.083s user 0.001s sys 0.000s
I20260809 15:47:22.612525 28577 tablet_bootstrap.cc:492] T ba56725e42f74a1baa4c04d9660727f9 P 61e69695258f48baa984c9d395e8ce3f: Bootstrap starting.
I20260809 15:47:22.595729 28319 raft_consensus.cc:504] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:22.621078 28756 leader_election.cc:288] T 3d880de657e544468c82157769622a26 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:22.634290 29028 raft_consensus.cc:2781] T 907bacfa79d04a55a68c8dbe60056462 P 61e69695258f48baa984c9d395e8ce3f [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260809 15:47:22.660318 23396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99bdb93503db4de4a1cfb0161ea0c07c" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:22.636015 28802 raft_consensus.cc:482] T d3e94b7e50754a81b3d72a07476cb7b9 P 61e69695258f48baa984c9d395e8ce3f [term 9 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:47:22.668187 23518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99bdb93503db4de4a1cfb0161ea0c07c" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:22.662315 23396 raft_consensus.cc:2454] T 99bdb93503db4de4a1cfb0161ea0c07c P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 5.
I20260809 15:47:22.666644 23400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a51c0baaa6e432997c9d3552af4f254" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:47:22.669262 23400 raft_consensus.cc:3037] T 0a51c0baaa6e432997c9d3552af4f254 P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Advancing to term 9
I20260809 15:47:22.670145 23228 leader_election.cc:302] T 99bdb93503db4de4a1cfb0161ea0c07c P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d; no voters:
I20260809 15:47:22.671155 23518 raft_consensus.cc:2379] T 99bdb93503db4de4a1cfb0161ea0c07c P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 in current term 5: Already voted for candidate 65c36d164b0540f099d592f38307ee6c in this term.
I20260809 15:47:22.633411 28406 consensus_queue.cc:1035] T 0c29937a80724ddb9d07a27534a146f3 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.033s
I20260809 15:47:22.653411 29028 raft_consensus.cc:482] T 907bacfa79d04a55a68c8dbe60056462 P 61e69695258f48baa984c9d395e8ce3f [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:47:22.675676 28456 raft_consensus.cc:482] T 0da4d5283c8a46878bd39b85e5c71f66 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 15:47:22.675989 29073 log.cc:924] Time spent T 07fcabb569414bddbd39df6cedd24e64 P 65c36d164b0540f099d592f38307ee6c: Append to log took a long time: real 0.072s user 0.001s sys 0.000s
I20260809 15:47:22.665575 29087 raft_consensus.cc:482] T 16ca412a0dae4feebe85dd01bf1a9173 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
I20260809 15:47:22.668727 29024 leader_election.cc:288] T 99bdb93503db4de4a1cfb0161ea0c07c P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:22.678275 28971 raft_consensus.cc:2781] T 99bdb93503db4de4a1cfb0161ea0c07c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260809 15:47:22.729130 28971 raft_consensus.cc:482] T 99bdb93503db4de4a1cfb0161ea0c07c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting leader election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:47:22.729583 28971 raft_consensus.cc:3037] T 99bdb93503db4de4a1cfb0161ea0c07c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Advancing to term 5
I20260809 15:47:22.729912 28718 leader_election.cc:288] T 213543cdbe814c89a17a963469922c1b P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:22.732765 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "213543cdbe814c89a17a963469922c1b" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:22.680788 28456 raft_consensus.cc:504] T 0da4d5283c8a46878bd39b85e5c71f66 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:22.678469 29028 raft_consensus.cc:3037] T 907bacfa79d04a55a68c8dbe60056462 P 61e69695258f48baa984c9d395e8ce3f [term 9 FOLLOWER]: Advancing to term 10
W20260809 15:47:22.728247 29059 log.cc:924] Time spent T a88421dba80d4d3388aa5eb62e23853b P c0f38c1908414566aa9115bb2cbdd85d: Append to log took a long time: real 0.139s user 0.000s sys 0.000s
I20260809 15:47:22.733530 28988 raft_consensus.cc:504] T 0acc47ce21c3422698d1ef7277c3ed63 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:22.739943 23400 raft_consensus.cc:2454] T 0a51c0baaa6e432997c9d3552af4f254 P c0f38c1908414566aa9115bb2cbdd85d [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 9.
I20260809 15:47:22.672322 28802 raft_consensus.cc:504] T d3e94b7e50754a81b3d72a07476cb7b9 P 61e69695258f48baa984c9d395e8ce3f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:22.740994 23518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d880de657e544468c82157769622a26" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
W20260809 15:47:22.729853 29049 log.cc:924] Time spent T c0741ab44d99439abc3aa3bdd27ee160 P 61e69695258f48baa984c9d395e8ce3f: Append to log took a long time: real 0.052s user 0.000s sys 0.000s
W20260809 15:47:22.741756 24410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 139016us
I20260809 15:47:22.747319 23518 raft_consensus.cc:2454] T 3d880de657e544468c82157769622a26 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 5.
I20260809 15:47:22.687552 28612 leader_election.cc:288] T 0a51c0baaa6e432997c9d3552af4f254 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 9 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:22.754911 24453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a51c0baaa6e432997c9d3552af4f254" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f"
I20260809 15:47:22.755465 24453 raft_consensus.cc:3037] T 0a51c0baaa6e432997c9d3552af4f254 P 61e69695258f48baa984c9d395e8ce3f [term 8 FOLLOWER]: Advancing to term 9
I20260809 15:47:22.755429 23518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d880de657e544468c82157769622a26" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:22.756119 23518 raft_consensus.cc:2454] T 3d880de657e544468c82157769622a26 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 5.
W20260809 15:47:22.746833 29089 log.cc:924] Time spent T 213543cdbe814c89a17a963469922c1b P 65c36d164b0540f099d592f38307ee6c: Append to log took a long time: real 0.079s user 0.000s sys 0.001s
I20260809 15:47:22.758049 29068 raft_consensus.cc:482] T ae6d2c08a7a94f58ac8dd698e3fd456c P 65c36d164b0540f099d592f38307ee6c [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:22.727705 29087 raft_consensus.cc:504] T 16ca412a0dae4feebe85dd01bf1a9173 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:22.764784 29068 raft_consensus.cc:504] T ae6d2c08a7a94f58ac8dd698e3fd456c P 65c36d164b0540f099d592f38307ee6c [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:22.768034 24451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d880de657e544468c82157769622a26" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:47:22.769874 24451 raft_consensus.cc:2454] T 3d880de657e544468c82157769622a26 P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 5.
I20260809 15:47:22.765203 28887 leader_election.cc:288] T 3d880de657e544468c82157769622a26 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:22.773526 28456 leader_election.cc:288] T 0da4d5283c8a46878bd39b85e5c71f66 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:47:22.784461 28319 leader_election.cc:288] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:22.772382 29107 raft_consensus.cc:482] T 294d4bcc1e7a477cb56455581277a60f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
W20260809 15:47:22.806821 28532 consensus_meta.cc:287] T 3296ef50be254fc29cfc77d22a8f7a23 P 61e69695258f48baa984c9d395e8ce3f: Time spent flushing consensus metadata: real 0.602s user 0.010s sys 0.000s
I20260809 15:47:22.808848 29086 raft_consensus.cc:482] T c5c2e425a74b4e64ae70b5def750434c P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:22.815829 29086 raft_consensus.cc:504] T c5c2e425a74b4e64ae70b5def750434c P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:22.817619 24455 raft_consensus.cc:1261] T 3924b648461e463b9d7925e5671d68ae P 61e69695258f48baa984c9d395e8ce3f [term 11 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 11 index: 10. (term mismatch)
I20260809 15:47:22.817978 24455 pending_rounds.cc:76] T 3924b648461e463b9d7925e5671d68ae P 61e69695258f48baa984c9d395e8ce3f: Aborting all ops after (but not including) 9
I20260809 15:47:22.818198 24455 pending_rounds.cc:97] T 3924b648461e463b9d7925e5671d68ae P 61e69695258f48baa984c9d395e8ce3f: Aborting uncommitted NO_OP operation due to leader change: 10.10
I20260809 15:47:22.814652 24463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e4ffcf83c234ae7910ddef3ec0b4bda" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:47:22.818630 24455 raft_consensus.cc:2866] T 3924b648461e463b9d7925e5671d68ae P 61e69695258f48baa984c9d395e8ce3f [term 11 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 15:47:22.818993 24463 raft_consensus.cc:2454] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 5.
I20260809 15:47:22.815256 28532 raft_consensus.cc:504] T 3296ef50be254fc29cfc77d22a8f7a23 P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:22.821414 28532 leader_election.cc:288] T 3296ef50be254fc29cfc77d22a8f7a23 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 4 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:47:22.814388 29068 leader_election.cc:288] T ae6d2c08a7a94f58ac8dd698e3fd456c P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:47:22.803046 28447 raft_consensus.cc:3032] T 998c6cee72cf40228c6dba064c0db2da P c0f38c1908414566aa9115bb2cbdd85d [term 8 LEADER]: Stepping down as leader of term 8
W20260809 15:47:22.806272 29095 log.cc:924] Time spent T c0741ab44d99439abc3aa3bdd27ee160 P 67b3aeadfa7a432abe8af4d292e04ab9: Append to log took a long time: real 0.063s user 0.000s sys 0.004s
I20260809 15:47:22.826155 24453 raft_consensus.cc:2454] T 0a51c0baaa6e432997c9d3552af4f254 P 61e69695258f48baa984c9d395e8ce3f [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 9.
I20260809 15:47:22.807593 29107 raft_consensus.cc:504] T 294d4bcc1e7a477cb56455581277a60f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:22.823854 28447 raft_consensus.cc:729] T 998c6cee72cf40228c6dba064c0db2da P c0f38c1908414566aa9115bb2cbdd85d [term 8 LEADER]: Becoming Follower/Learner. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:47:22.830391 23282 raft_consensus.cc:3037] T d1a134ba33ea4076bbaac0ae460c17ba P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Advancing to term 8
I20260809 15:47:22.831009 28802 leader_election.cc:288] T d3e94b7e50754a81b3d72a07476cb7b9 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:22.832175 28934 raft_consensus.cc:3032] T 28e17dcb5efe45b8b6a8c42da03b4819 P c0f38c1908414566aa9115bb2cbdd85d [term 8 LEADER]: Stepping down as leader of term 8
I20260809 15:47:22.832547 28934 raft_consensus.cc:729] T 28e17dcb5efe45b8b6a8c42da03b4819 P c0f38c1908414566aa9115bb2cbdd85d [term 8 LEADER]: Becoming Follower/Learner. State: Replica: c0f38c1908414566aa9115bb2cbdd85d, State: Running, Role: LEADER
I20260809 15:47:22.830328 28447 consensus_queue.cc:260] T 998c6cee72cf40228c6dba064c0db2da P c0f38c1908414566aa9115bb2cbdd85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:22.833287 28934 consensus_queue.cc:260] T 28e17dcb5efe45b8b6a8c42da03b4819 P c0f38c1908414566aa9115bb2cbdd85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:22.834189 28934 raft_consensus.cc:3037] T 28e17dcb5efe45b8b6a8c42da03b4819 P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Advancing to term 9
I20260809 15:47:22.834306 28447 raft_consensus.cc:3037] T 998c6cee72cf40228c6dba064c0db2da P c0f38c1908414566aa9115bb2cbdd85d [term 8 FOLLOWER]: Advancing to term 9
I20260809 15:47:22.835839 28988 leader_election.cc:288] T 0acc47ce21c3422698d1ef7277c3ed63 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 10 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:22.836375 28674 consensus_queue.cc:260] T 4bcac4b24e7a4448abdc934bba5a624e P 61e69695258f48baa984c9d395e8ce3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:22.837848 23521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0acc47ce21c3422698d1ef7277c3ed63" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 10 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:47:22.838563 23521 raft_consensus.cc:2379] T 0acc47ce21c3422698d1ef7277c3ed63 P 65c36d164b0540f099d592f38307ee6c [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 in current term 10: Already voted for candidate 65c36d164b0540f099d592f38307ee6c in this term.
I20260809 15:47:22.838755 29087 leader_election.cc:288] T 16ca412a0dae4feebe85dd01bf1a9173 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:22.844201 24463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "294d4bcc1e7a477cb56455581277a60f" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:47:22.844785 24463 raft_consensus.cc:2454] T 294d4bcc1e7a477cb56455581277a60f P 61e69695258f48baa984c9d395e8ce3f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 7.
I20260809 15:47:22.846740 23282 raft_consensus.cc:1261] T d1a134ba33ea4076bbaac0ae460c17ba P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Refusing update from remote peer 65c36d164b0540f099d592f38307ee6c: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 5. (term mismatch)
I20260809 15:47:22.847761 23282 pending_rounds.cc:76] T d1a134ba33ea4076bbaac0ae460c17ba P 67b3aeadfa7a432abe8af4d292e04ab9: Aborting all ops after (but not including) 4
I20260809 15:47:22.848176 23282 pending_rounds.cc:97] T d1a134ba33ea4076bbaac0ae460c17ba P 67b3aeadfa7a432abe8af4d292e04ab9: Aborting uncommitted NO_OP operation due to leader change: 7.5
W20260809 15:47:22.847340 23470 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 259182us
I20260809 15:47:22.848569 23282 raft_consensus.cc:2866] T d1a134ba33ea4076bbaac0ae460c17ba P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 15:47:22.849519 23521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3e94b7e50754a81b3d72a07476cb7b9" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:22.849637 28971 raft_consensus.cc:504] T 99bdb93503db4de4a1cfb0161ea0c07c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:22.850013 28970 consensus_queue.cc:1035] T ba56725e42f74a1baa4c04d9660727f9 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260809 15:47:22.850379 23228 leader_election.cc:302] T 73f6f44c259547f1b05a3a434616fa54 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:22.839596 23518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "294d4bcc1e7a477cb56455581277a60f" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:22.849738 23276 raft_consensus.cc:1226] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Rejecting Update request from peer 65c36d164b0540f099d592f38307ee6c for earlier term 4. Current term is 5. Ops: []
I20260809 15:47:22.852154 23518 raft_consensus.cc:2379] T 294d4bcc1e7a477cb56455581277a60f P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 in current term 7: Already voted for candidate 65c36d164b0540f099d592f38307ee6c in this term.
I20260809 15:47:22.852623 23228 leader_election.cc:302] T f7715d71dbac447b9351fc5eb36c5e80 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters: c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:22.853696 24410 leader_election.cc:302] T 7660dfa2ef694927b8eda4100a2bcd9b P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c; no voters: c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:22.854621 24453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16ca412a0dae4feebe85dd01bf1a9173" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:47:22.851296 28970 consensus_queue.cc:1035] T 2db057fd10e149a9895dff6ef8c9f038 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.004s
I20260809 15:47:22.854938 23470 leader_election.cc:302] T ba56725e42f74a1baa4c04d9660727f9 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c; no voters: 67b3aeadfa7a432abe8af4d292e04ab9, c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:22.856588 24410 leader_election.cc:302] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c; no voters: c0f38c1908414566aa9115bb2cbdd85d
I20260809 15:47:22.859593 23280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16ca412a0dae4feebe85dd01bf1a9173" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:22.860224 23280 raft_consensus.cc:2454] T 16ca412a0dae4feebe85dd01bf1a9173 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 7.
I20260809 15:47:22.861415 23286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0da4d5283c8a46878bd39b85e5c71f66" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:22.862067 23286 raft_consensus.cc:2454] T 0da4d5283c8a46878bd39b85e5c71f66 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 7.
I20260809 15:47:22.851449 28319 consensus_queue.cc:1035] T c0741ab44d99439abc3aa3bdd27ee160 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.011s
I20260809 15:47:22.863649 28988 raft_consensus.cc:2726] T 73f6f44c259547f1b05a3a434616fa54 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260809 15:47:22.864297 28802 raft_consensus.cc:2781] T 7660dfa2ef694927b8eda4100a2bcd9b P 61e69695258f48baa984c9d395e8ce3f [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260809 15:47:22.864708 28802 raft_consensus.cc:482] T 7660dfa2ef694927b8eda4100a2bcd9b P 61e69695258f48baa984c9d395e8ce3f [term 6 FOLLOWER]: Starting leader election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:47:22.865092 28802 raft_consensus.cc:3037] T 7660dfa2ef694927b8eda4100a2bcd9b P 61e69695258f48baa984c9d395e8ce3f [term 6 FOLLOWER]: Advancing to term 7
I20260809 15:47:22.868049 23274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5c2e425a74b4e64ae70b5def750434c" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:22.868605 23274 raft_consensus.cc:2454] T c5c2e425a74b4e64ae70b5def750434c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 8.
I20260809 15:47:22.869284 28612 raft_consensus.cc:2781] T f7715d71dbac447b9351fc5eb36c5e80 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Leader election won for term 7
I20260809 15:47:22.859836 23285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae6d2c08a7a94f58ac8dd698e3fd456c" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:22.870741 24410 leader_election.cc:302] T 3d880de657e544468c82157769622a26 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c; no voters:
I20260809 15:47:22.871286 23285 raft_consensus.cc:2454] T ae6d2c08a7a94f58ac8dd698e3fd456c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 9.
I20260809 15:47:22.876322 23228 leader_election.cc:302] T 3d880de657e544468c82157769622a26 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:47:22.876565 23282 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53554 (request call id 1907) took 1212 ms. Trace:
I20260809 15:47:22.876891 23282 rpcz_store.cc:276] 0809 15:47:21.645753 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:21.646823 (+ 1070us) service_pool.cc:227] Handling call
0809 15:47:21.647082 (+ 259us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:47:22.830016 (+1182934us) spinlock_profiling.cc:243] Waited 137 ms on lock 0x7b58007d21b0. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e919bf63 00007f34e91ffaba 00007f34e92173af 00007f34e9218b4e 00007f34e9222c89 00007f34e9223556 00007f34e921fedf 00007f34e921f99e 00007f34ee1612e9 00007f34e7847684
0809 15:47:22.857727 (+ 27711us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:47:22.857874 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":315347712}
I20260809 15:47:22.848312 29086 leader_election.cc:288] T c5c2e425a74b4e64ae70b5def750434c P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:47:22.845471 29107 leader_election.cc:288] T 294d4bcc1e7a477cb56455581277a60f P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:22.882742 28451 consensus_queue.cc:260] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:22.884222 29028 raft_consensus.cc:504] T 907bacfa79d04a55a68c8dbe60056462 P 61e69695258f48baa984c9d395e8ce3f [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:22.879917 28416 raft_consensus.cc:2781] T 3d880de657e544468c82157769622a26 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260809 15:47:22.851464 28971 leader_election.cc:288] T 99bdb93503db4de4a1cfb0161ea0c07c P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:22.886795 23470 leader_election.cc:302] T 16ca412a0dae4feebe85dd01bf1a9173 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:47:22.888054 23521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99bdb93503db4de4a1cfb0161ea0c07c" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:47:22.869956 28586 raft_consensus.cc:2726] T ba56725e42f74a1baa4c04d9660727f9 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260809 15:47:22.869773 28612 raft_consensus.cc:686] T f7715d71dbac447b9351fc5eb36c5e80 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 LEADER]: Becoming Leader. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Running, Role: LEADER
I20260809 15:47:22.894683 23521 raft_consensus.cc:2379] T 99bdb93503db4de4a1cfb0161ea0c07c P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 in current term 5: Already voted for candidate 65c36d164b0540f099d592f38307ee6c in this term.
I20260809 15:47:22.895846 23470 leader_election.cc:302] T 0da4d5283c8a46878bd39b85e5c71f66 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:47:22.878010 28924 raft_consensus.cc:2781] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 61e69695258f48baa984c9d395e8ce3f [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260809 15:47:22.893209 24455 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38734 (request call id 452) took 1187 ms. Trace:
I20260809 15:47:22.880481 28319 consensus_queue.cc:1035] T 0c29937a80724ddb9d07a27534a146f3 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.020s
I20260809 15:47:22.909585 29087 raft_consensus.cc:2781] T 16ca412a0dae4feebe85dd01bf1a9173 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260809 15:47:22.907881 28612 consensus_queue.cc:237] T f7715d71dbac447b9351fc5eb36c5e80 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:22.909248 28924 raft_consensus.cc:482] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 61e69695258f48baa984c9d395e8ce3f [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:47:22.881542 28532 raft_consensus.cc:2781] T 3d880de657e544468c82157769622a26 P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260809 15:47:22.909972 29087 raft_consensus.cc:482] T 16ca412a0dae4feebe85dd01bf1a9173 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Starting leader election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
I20260809 15:47:22.912317 23470 leader_election.cc:302] T c5c2e425a74b4e64ae70b5def750434c P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:47:22.893708 28416 raft_consensus.cc:482] T 3d880de657e544468c82157769622a26 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:47:22.925315 28627 raft_consensus.cc:2781] T c5c2e425a74b4e64ae70b5def750434c P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260809 15:47:22.925729 28627 raft_consensus.cc:482] T c5c2e425a74b4e64ae70b5def750434c P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:47:22.911747 28924 raft_consensus.cc:3037] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 61e69695258f48baa984c9d395e8ce3f [term 10 FOLLOWER]: Advancing to term 11
I20260809 15:47:22.891921 28725 consensus_queue.cc:1046] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 65c36d164b0540f099d592f38307ee6c [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.031s
I20260809 15:47:22.919584 29087 raft_consensus.cc:3037] T 16ca412a0dae4feebe85dd01bf1a9173 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Advancing to term 8
I20260809 15:47:22.916309 28532 raft_consensus.cc:482] T 3d880de657e544468c82157769622a26 P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:47:22.909318 24455 rpcz_store.cc:276] 0809 15:47:21.705837 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:21.707188 (+ 1351us) service_pool.cc:227] Handling call
0809 15:47:21.719423 (+ 12235us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:47:22.819253 (+1099830us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:47:22.892930 (+ 73677us) spinlock_profiling.cc:243] Waited 1.1 s on lock 0x7b54005a05d4. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e9220f51 00007f34e921f99e 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772 00007f34e764a94c 00007f34e7649d86
0809 15:47:22.893123 (+ 193us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2524812672}
I20260809 15:47:22.924067 28416 raft_consensus.cc:3037] T 3d880de657e544468c82157769622a26 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Advancing to term 6
I20260809 15:47:22.926108 28627 raft_consensus.cc:3037] T c5c2e425a74b4e64ae70b5def750434c P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Advancing to term 9
I20260809 15:47:22.923782 28456 raft_consensus.cc:2781] T 0da4d5283c8a46878bd39b85e5c71f66 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260809 15:47:22.931207 28319 raft_consensus.cc:482] T 30c1740ef6874df0a332fbd6fd4b03ce P 67b3aeadfa7a432abe8af4d292e04ab9 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:22.940601 28532 raft_consensus.cc:3037] T 3d880de657e544468c82157769622a26 P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Advancing to term 6
I20260809 15:47:22.939536 29068 raft_consensus.cc:482] T 4b30f28311964a0b9149a50495e83d8f P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:22.945945 28456 raft_consensus.cc:482] T 0da4d5283c8a46878bd39b85e5c71f66 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260809 15:47:22.962682 29090 log.cc:924] Time spent T 4564dfc6d4094c1b9240b367ccecf3b1 P 65c36d164b0540f099d592f38307ee6c: Append to log took a long time: real 0.292s user 0.001s sys 0.000s
I20260809 15:47:22.971323 28627 raft_consensus.cc:504] T c5c2e425a74b4e64ae70b5def750434c P 65c36d164b0540f099d592f38307ee6c [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:22.973362 28627 leader_election.cc:288] T c5c2e425a74b4e64ae70b5def750434c P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 9 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:47:22.977357 28802 raft_consensus.cc:504] T 7660dfa2ef694927b8eda4100a2bcd9b P 61e69695258f48baa984c9d395e8ce3f [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:22.956272 29068 raft_consensus.cc:504] T 4b30f28311964a0b9149a50495e83d8f P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } attrs { replace: true } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } }
I20260809 15:47:22.982343 29068 leader_election.cc:288] T 4b30f28311964a0b9149a50495e83d8f P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:47:22.956646 28456 raft_consensus.cc:3037] T 0da4d5283c8a46878bd39b85e5c71f66 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Advancing to term 8
W20260809 15:47:22.990357 23470 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 78112.5us
I20260809 15:47:22.990955 23470 leader_election.cc:302] T ae6d2c08a7a94f58ac8dd698e3fd456c P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:47:22.950587 28319 raft_consensus.cc:504] T 30c1740ef6874df0a332fbd6fd4b03ce P 67b3aeadfa7a432abe8af4d292e04ab9 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:47:22.998296 23526 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44018 (request call id 375) took 1295 ms (client timeout 1719 ms). Trace:
W20260809 15:47:22.998678 23526 rpcz_store.cc:269] 0809 15:47:21.702808 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:21.703394 (+ 586us) service_pool.cc:227] Handling call
0809 15:47:22.998147 (+1294753us) spinlock_profiling.cc:243] Waited 172 ms on lock 0x7b54000f8cd4. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e92242ae 00007f34ee161c27 00007f34e7849c04 00007f34e7849b92 00007f34e7849ac1 00007f34e7849a3c 00007f34e7848cf2 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:47:22.998245 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":396284160}
I20260809 15:47:23.000046 28319 leader_election.cc:288] T 30c1740ef6874df0a332fbd6fd4b03ce P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:23.001399 24410 leader_election.cc:302] T cee02fff2c9c48a19f43eb358a89e7b0 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c; no voters: 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:47:23.003321 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30c1740ef6874df0a332fbd6fd4b03ce" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:23.007151 23276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b30f28311964a0b9149a50495e83d8f" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:23.007839 23276 raft_consensus.cc:2454] T 4b30f28311964a0b9149a50495e83d8f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 6.
I20260809 15:47:23.008689 23280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5c2e425a74b4e64ae70b5def750434c" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:47:23.009292 23280 raft_consensus.cc:3037] T c5c2e425a74b4e64ae70b5def750434c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Advancing to term 9
I20260809 15:47:23.014501 23470 leader_election.cc:302] T 4b30f28311964a0b9149a50495e83d8f P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:47:23.015229 28719 raft_consensus.cc:2781] T 4b30f28311964a0b9149a50495e83d8f P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260809 15:47:23.016239 28197 consensus_queue.cc:1035] T d1a134ba33ea4076bbaac0ae460c17ba P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260809 15:47:23.016678 28802 leader_election.cc:288] T 7660dfa2ef694927b8eda4100a2bcd9b P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 7 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:23.023653 23276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e4ffcf83c234ae7910ddef3ec0b4bda" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:23.024322 23276 raft_consensus.cc:2454] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 4.
I20260809 15:47:23.026132 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7660dfa2ef694927b8eda4100a2bcd9b" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:47:23.005542 28532 raft_consensus.cc:504] T 3d880de657e544468c82157769622a26 P 61e69695258f48baa984c9d395e8ce3f [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:23.015687 28719 raft_consensus.cc:482] T 4b30f28311964a0b9149a50495e83d8f P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:47:23.002043 29068 raft_consensus.cc:2781] T ae6d2c08a7a94f58ac8dd698e3fd456c P 65c36d164b0540f099d592f38307ee6c [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260809 15:47:23.034979 29068 raft_consensus.cc:482] T ae6d2c08a7a94f58ac8dd698e3fd456c P 65c36d164b0540f099d592f38307ee6c [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:47:23.034933 28719 raft_consensus.cc:3037] T 4b30f28311964a0b9149a50495e83d8f P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Advancing to term 7
I20260809 15:47:23.035571 29068 raft_consensus.cc:3037] T ae6d2c08a7a94f58ac8dd698e3fd456c P 65c36d164b0540f099d592f38307ee6c [term 9 FOLLOWER]: Advancing to term 10
I20260809 15:47:23.045056 23276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d880de657e544468c82157769622a26" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:23.051985 23223 leader_election.cc:302] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:47:22.992846 29028 leader_election.cc:288] T 907bacfa79d04a55a68c8dbe60056462 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 10 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:47:23.057051 23273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3296ef50be254fc29cfc77d22a8f7a23" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:47:23.058135 23273 raft_consensus.cc:3037] T 3296ef50be254fc29cfc77d22a8f7a23 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 2 FOLLOWER]: Advancing to term 4
I20260809 15:47:23.059759 23280 raft_consensus.cc:2454] T c5c2e425a74b4e64ae70b5def750434c P 67b3aeadfa7a432abe8af4d292e04ab9 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 9.
I20260809 15:47:23.060714 23470 leader_election.cc:302] T c5c2e425a74b4e64ae70b5def750434c P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:47:23.062227 29120 raft_consensus.cc:2781] T c5c2e425a74b4e64ae70b5def750434c P 65c36d164b0540f099d592f38307ee6c [term 9 FOLLOWER]: Leader election won for term 9
I20260809 15:47:23.063047 29120 raft_consensus.cc:686] T c5c2e425a74b4e64ae70b5def750434c P 65c36d164b0540f099d592f38307ee6c [term 9 LEADER]: Becoming Leader. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:47:23.064479 29120 consensus_queue.cc:237] T c5c2e425a74b4e64ae70b5def750434c P 65c36d164b0540f099d592f38307ee6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:23.018997 28873 leader_election.cc:288] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:47:23.077828 23286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "907bacfa79d04a55a68c8dbe60056462" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:23.078438 23286 raft_consensus.cc:2454] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 9.
I20260809 15:47:23.027729 29108 raft_consensus.cc:2781] T cee02fff2c9c48a19f43eb358a89e7b0 P 61e69695258f48baa984c9d395e8ce3f [term 10 FOLLOWER]: Leader election won for term 10
I20260809 15:47:23.083751 23223 leader_election.cc:302] T 0a51c0baaa6e432997c9d3552af4f254 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:47:23.081562 23273 raft_consensus.cc:2454] T 3296ef50be254fc29cfc77d22a8f7a23 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 4.
I20260809 15:47:23.082862 23526 raft_consensus.cc:2454] T 7660dfa2ef694927b8eda4100a2bcd9b P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 7.
I20260809 15:47:23.085434 24410 leader_election.cc:302] T 7660dfa2ef694927b8eda4100a2bcd9b P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c; no voters:
I20260809 15:47:23.090467 23223 leader_election.cc:302] T 294d4bcc1e7a477cb56455581277a60f P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c
I20260809 15:47:23.089777 28988 raft_consensus.cc:2781] T 0a51c0baaa6e432997c9d3552af4f254 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 9 FOLLOWER]: Leader election won for term 9
I20260809 15:47:23.086411 28319 raft_consensus.cc:2781] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260809 15:47:23.091754 28873 raft_consensus.cc:2781] T 7660dfa2ef694927b8eda4100a2bcd9b P 61e69695258f48baa984c9d395e8ce3f [term 7 FOLLOWER]: Leader election won for term 7
I20260809 15:47:23.104976 24406 leader_election.cc:302] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:47:23.105568 29028 raft_consensus.cc:2781] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260809 15:47:23.105912 29028 raft_consensus.cc:482] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:47:23.106240 29028 raft_consensus.cc:3037] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Advancing to term 6
I20260809 15:47:23.106360 24406 leader_election.cc:302] T 3296ef50be254fc29cfc77d22a8f7a23 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:47:23.107460 28879 raft_consensus.cc:2781] T 3296ef50be254fc29cfc77d22a8f7a23 P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Leader election won for term 4
I20260809 15:47:23.108129 28879 raft_consensus.cc:686] T 3296ef50be254fc29cfc77d22a8f7a23 P 61e69695258f48baa984c9d395e8ce3f [term 4 LEADER]: Becoming Leader. State: Replica: 61e69695258f48baa984c9d395e8ce3f, State: Running, Role: LEADER
I20260809 15:47:23.103211 28988 raft_consensus.cc:686] T 0a51c0baaa6e432997c9d3552af4f254 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 9 LEADER]: Becoming Leader. State: Replica: 67b3aeadfa7a432abe8af4d292e04ab9, State: Running, Role: LEADER
I20260809 15:47:23.109660 28988 consensus_queue.cc:237] T 0a51c0baaa6e432997c9d3552af4f254 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:23.109395 28879 consensus_queue.cc:237] T 3296ef50be254fc29cfc77d22a8f7a23 P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:23.112739 28197 consensus_queue.cc:1035] T 3924b648461e463b9d7925e5671d68ae P 65c36d164b0540f099d592f38307ee6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260809 15:47:23.114077 23517 raft_consensus.cc:1261] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Refusing update from remote peer 67b3aeadfa7a432abe8af4d292e04ab9: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260809 15:47:23.097048 28612 raft_consensus.cc:2781] T 294d4bcc1e7a477cb56455581277a60f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260809 15:47:23.094538 29108 raft_consensus.cc:686] T cee02fff2c9c48a19f43eb358a89e7b0 P 61e69695258f48baa984c9d395e8ce3f [term 10 LEADER]: Becoming Leader. State: Replica: 61e69695258f48baa984c9d395e8ce3f, State: Running, Role: LEADER
I20260809 15:47:23.120385 23286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "907bacfa79d04a55a68c8dbe60056462" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:47:23.127003 23286 raft_consensus.cc:3037] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 9 FOLLOWER]: Advancing to term 10
I20260809 15:47:23.130205 23517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d880de657e544468c82157769622a26" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:47:23.131018 23517 raft_consensus.cc:3037] T 3d880de657e544468c82157769622a26 P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Advancing to term 6
I20260809 15:47:23.132884 23277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d880de657e544468c82157769622a26" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:47:23.101485 28319 raft_consensus.cc:482] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
I20260809 15:47:23.121393 28971 raft_consensus.cc:482] T a3622f6e3e3641cea1960e3a31498bb7 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:23.127076 28612 raft_consensus.cc:482] T 294d4bcc1e7a477cb56455581277a60f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
I20260809 15:47:23.130622 28532 leader_election.cc:288] T 3d880de657e544468c82157769622a26 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 6 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:23.135483 28319 raft_consensus.cc:3037] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 67b3aeadfa7a432abe8af4d292e04ab9 [term 4 FOLLOWER]: Advancing to term 5
I20260809 15:47:23.151242 29100 raft_consensus.cc:482] T 763e58d6ba644eea830a6c1664d39c9b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:47:23.173059 29087 raft_consensus.cc:504] T 16ca412a0dae4feebe85dd01bf1a9173 P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:23.173236 28416 raft_consensus.cc:504] T 3d880de657e544468c82157769622a26 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:23.174055 28874 raft_consensus.cc:482] T 7baa2492df0340c4a5ac241959ce1ac3 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:47:23.175374 29120 raft_consensus.cc:482] T 998c6cee72cf40228c6dba064c0db2da P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:47:23.146291 28280 raft_consensus.cc:3032] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 61e69695258f48baa984c9d395e8ce3f [term 4 LEADER]: Stepping down as leader of term 4
I20260809 15:47:23.219455 28280 raft_consensus.cc:729] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 61e69695258f48baa984c9d395e8ce3f [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 61e69695258f48baa984c9d395e8ce3f, State: Running, Role: LEADER
I20260809 15:47:23.220773 28280 consensus_queue.cc:260] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 61e69695258f48baa984c9d395e8ce3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:23.149364 28971 raft_consensus.cc:504] T a3622f6e3e3641cea1960e3a31498bb7 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:23.186614 29087 leader_election.cc:288] T 16ca412a0dae4feebe85dd01bf1a9173 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 8 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:23.188125 24453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16ca412a0dae4feebe85dd01bf1a9173" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f"
I20260809 15:47:23.189450 23526 raft_consensus.cc:1226] T 16ca412a0dae4feebe85dd01bf1a9173 P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Rejecting Update request from peer 61e69695258f48baa984c9d395e8ce3f for earlier term 7. Current term is 8. Ops: []
I20260809 15:47:23.197271 29108 consensus_queue.cc:237] T cee02fff2c9c48a19f43eb358a89e7b0 P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:23.199110 23282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16ca412a0dae4feebe85dd01bf1a9173" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
I20260809 15:47:23.238468 23282 raft_consensus.cc:3037] T 16ca412a0dae4feebe85dd01bf1a9173 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Advancing to term 8
I20260809 15:47:23.196976 29028 raft_consensus.cc:504] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 61e69695258f48baa984c9d395e8ce3f [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:23.147446 28973 raft_consensus.cc:482] T a88421dba80d4d3388aa5eb62e23853b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:47:23.213040 23517 raft_consensus.cc:2454] T 3d880de657e544468c82157769622a26 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 6.
I20260809 15:47:23.151952 28612 raft_consensus.cc:3037] T 294d4bcc1e7a477cb56455581277a60f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Advancing to term 7
I20260809 15:47:23.147995 28525 raft_consensus.cc:482] T a88421dba80d4d3388aa5eb62e23853b P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:47:23.216284 28784 raft_consensus.cc:482] T cef3e7277d7d4760824a3436ed4a847d P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:47:23.218362 28416 leader_election.cc:288] T 3d880de657e544468c82157769622a26 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 6 election: Requested vote from peers 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:23.219103 23276 raft_consensus.cc:2379] T 3d880de657e544468c82157769622a26 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 61e69695258f48baa984c9d395e8ce3f in current term 6: Already voted for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in this term.
I20260809 15:47:23.219256 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d880de657e544468c82157769622a26" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:47:23.255522 23527 raft_consensus.cc:2379] T 3d880de657e544468c82157769622a26 P 65c36d164b0540f099d592f38307ee6c [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 in current term 6: Already voted for candidate 61e69695258f48baa984c9d395e8ce3f in this term.
I20260809 15:47:23.221714 28451 raft_consensus.cc:3037] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 65c36d164b0540f099d592f38307ee6c [term 4 FOLLOWER]: Advancing to term 5
I20260809 15:47:23.223644 29120 raft_consensus.cc:504] T 998c6cee72cf40228c6dba064c0db2da P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:23.225375 24463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d880de657e544468c82157769622a26" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f"
W20260809 15:47:23.225621 23353 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 691369us
I20260809 15:47:23.126247 28621 consensus_queue.cc:1035] T 6ced718c1a5c4da79cb8c1ebd1435b87 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.023s
W20260809 15:47:23.230896 29129 log.cc:924] Time spent T 2db057fd10e149a9895dff6ef8c9f038 P 65c36d164b0540f099d592f38307ee6c: Append to log took a long time: real 0.060s user 0.000s sys 0.000s
I20260809 15:47:23.106099 28924 raft_consensus.cc:504] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 61e69695258f48baa984c9d395e8ce3f [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:23.243659 24410 leader_election.cc:302] T 3d880de657e544468c82157769622a26 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c; no voters: 67b3aeadfa7a432abe8af4d292e04ab9
I20260809 15:47:23.240270 23286 raft_consensus.cc:2454] T 907bacfa79d04a55a68c8dbe60056462 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 10.
I20260809 15:47:23.261883 24463 raft_consensus.cc:2379] T 3d880de657e544468c82157769622a26 P 61e69695258f48baa984c9d395e8ce3f [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 in current term 6: Already voted for candidate 61e69695258f48baa984c9d395e8ce3f in this term.
I20260809 15:47:23.259163 28525 raft_consensus.cc:504] T a88421dba80d4d3388aa5eb62e23853b P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:23.272190 28280 raft_consensus.cc:3037] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Advancing to term 5
I20260809 15:47:23.228463 28874 raft_consensus.cc:504] T 7baa2492df0340c4a5ac241959ce1ac3 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:23.283038 23517 raft_consensus.cc:1261] T 6ae14a74a26f434db08c652e097385c8 P 65c36d164b0540f099d592f38307ee6c [term 9 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260809 15:47:23.260279 28784 raft_consensus.cc:504] T cef3e7277d7d4760824a3436ed4a847d P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:23.289891 23282 raft_consensus.cc:2454] T 16ca412a0dae4feebe85dd01bf1a9173 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 8.
I20260809 15:47:23.292510 23283 raft_consensus.cc:1226] T 16ca412a0dae4feebe85dd01bf1a9173 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Rejecting Update request from peer 61e69695258f48baa984c9d395e8ce3f for earlier term 7. Current term is 8. Ops: []
I20260809 15:47:23.274935 28758 raft_consensus.cc:2781] T 3d880de657e544468c82157769622a26 P 61e69695258f48baa984c9d395e8ce3f [term 6 FOLLOWER]: Leader election won for term 6
I20260809 15:47:23.291208 23470 leader_election.cc:302] T 16ca412a0dae4feebe85dd01bf1a9173 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 61e69695258f48baa984c9d395e8ce3f
I20260809 15:47:23.301843 23529 raft_consensus.cc:1261] T a3622f6e3e3641cea1960e3a31498bb7 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260809 15:47:23.281150 28873 raft_consensus.cc:686] T 7660dfa2ef694927b8eda4100a2bcd9b P 61e69695258f48baa984c9d395e8ce3f [term 7 LEADER]: Becoming Leader. State: Replica: 61e69695258f48baa984c9d395e8ce3f, State: Running, Role: LEADER
I20260809 15:47:23.254905 28973 raft_consensus.cc:504] T a88421dba80d4d3388aa5eb62e23853b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:23.215396 29100 raft_consensus.cc:504] T 763e58d6ba644eea830a6c1664d39c9b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:23.303560 29094 raft_consensus.cc:2781] T 16ca412a0dae4feebe85dd01bf1a9173 P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Leader election won for term 8
W20260809 15:47:23.342708 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 86849.3us
I20260809 15:47:23.343611 24406 leader_election.cc:302] T 907bacfa79d04a55a68c8dbe60056462 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:47:23.326903 28758 raft_consensus.cc:686] T 3d880de657e544468c82157769622a26 P 61e69695258f48baa984c9d395e8ce3f [term 6 LEADER]: Becoming Leader. State: Replica: 61e69695258f48baa984c9d395e8ce3f, State: Running, Role: LEADER
I20260809 15:47:23.346710 23282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e4ffcf83c234ae7910ddef3ec0b4bda" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9"
W20260809 15:47:23.340910 29140 log.cc:924] Time spent T c5c2e425a74b4e64ae70b5def750434c P 65c36d164b0540f099d592f38307ee6c: Append to log took a long time: real 0.110s user 0.001s sys 0.000s
I20260809 15:47:23.350478 23228 leader_election.cc:302] T 3d880de657e544468c82157769622a26 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c
I20260809 15:47:23.357623 23286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998c6cee72cf40228c6dba064c0db2da" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:23.358325 23286 raft_consensus.cc:2379] T 998c6cee72cf40228c6dba064c0db2da P 67b3aeadfa7a432abe8af4d292e04ab9 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 65c36d164b0540f099d592f38307ee6c in current term 9: Already voted for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in this term.
I20260809 15:47:23.342305 29094 raft_consensus.cc:686] T 16ca412a0dae4feebe85dd01bf1a9173 P 65c36d164b0540f099d592f38307ee6c [term 8 LEADER]: Becoming Leader. State: Replica: 65c36d164b0540f099d592f38307ee6c, State: Running, Role: LEADER
I20260809 15:47:23.349877 28532 raft_consensus.cc:2781] T 907bacfa79d04a55a68c8dbe60056462 P 61e69695258f48baa984c9d395e8ce3f [term 10 FOLLOWER]: Leader election won for term 10
I20260809 15:47:23.376922 23395 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55332 (request call id 1563) took 1474 ms. Trace:
I20260809 15:47:23.377296 23395 rpcz_store.cc:276] 0809 15:47:21.902771 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:21.942224 (+ 39453us) service_pool.cc:227] Handling call
0809 15:47:22.058810 (+116586us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:47:22.077861 (+ 19051us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:47:23.376695 (+1298834us) spinlock_profiling.cc:243] Waited 116 ms on lock 0x7b5400319cd0. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e921fa09 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:47:23.376838 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":267524352}
I20260809 15:47:23.358183 29024 raft_consensus.cc:2726] T 3d880de657e544468c82157769622a26 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260809 15:47:23.360193 29031 leader_election.cc:288] T 07fcabb569414bddbd39df6cedd24e64 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:23.384194 28874 leader_election.cc:288] T 7baa2492df0340c4a5ac241959ce1ac3 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:23.385558 28973 leader_election.cc:288] T a88421dba80d4d3388aa5eb62e23853b P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:23.381326 28532 raft_consensus.cc:686] T 907bacfa79d04a55a68c8dbe60056462 P 61e69695258f48baa984c9d395e8ce3f [term 10 LEADER]: Becoming Leader. State: Replica: 61e69695258f48baa984c9d395e8ce3f, State: Running, Role: LEADER
I20260809 15:47:23.349611 28873 consensus_queue.cc:237] T 7660dfa2ef694927b8eda4100a2bcd9b P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:23.418143 23399 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52856 (request call id 470) took 1119 ms. Trace:
I20260809 15:47:23.408833 23529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3622f6e3e3641cea1960e3a31498bb7" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:23.419775 23529 raft_consensus.cc:2454] T a3622f6e3e3641cea1960e3a31498bb7 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 7.
W20260809 15:47:23.407016 23351 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.65:35459, user_credentials={real_user=slave}} blocked reactor thread for 887459us
I20260809 15:47:23.413810 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7baa2492df0340c4a5ac241959ce1ac3" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:23.373441 28971 leader_election.cc:288] T a3622f6e3e3641cea1960e3a31498bb7 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:23.373777 29028 leader_election.cc:288] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 6 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:47:23.425866 28887 raft_consensus.cc:482] T 563daf42d7784db195ea99c3cb21b370 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:47:23.426561 28887 raft_consensus.cc:504] T 563daf42d7784db195ea99c3cb21b370 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:23.429000 23529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a823d35f4f348d1b52ba9e6d80b00d2" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:47:23.433362 28758 consensus_queue.cc:237] T 3d880de657e544468c82157769622a26 P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:23.418505 23399 rpcz_store.cc:276] 0809 15:47:22.299231 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:22.299391 (+ 160us) service_pool.cc:227] Handling call
0809 15:47:22.335609 (+ 36218us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 15:47:22.341630 (+ 6021us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:47:22.341655 (+ 25us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 15:47:22.342046 (+ 391us) log.cc:842] Serialized 26 byte log entry
0809 15:47:22.513815 (+171769us) raft_consensus.cc:1631] Marking committed up to 8
0809 15:47:22.526838 (+ 13023us) log.cc:842] Serialized 14 byte log entry
0809 15:47:23.225246 (+698408us) spinlock_profiling.cc:243] Waited 1.39 ms on lock 0x7b1800281b60. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34ef1978b1 00007f34ef1a4661 00007f34e626f4ca 00007f34ef950f99 00007f34e6133990 00007f34e62f5864 00007f34e62f5155 00007f34e92183d0 00007f34e922a576 00007f34e923b0d7 00007f34e923b058
0809 15:47:23.225389 (+ 143us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:47:23.225468 (+ 79us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 15:47:23.286724 (+ 61256us) spinlock_profiling.cc:243] Waited 15.3 ms on lock 0x7b1800281b60. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34ef1978b1 00007f34ef1a4661 00007f34e62f8638 00007f34e62fbb11 00007f34e62fbac9 00007f34e62fba59 00007f34e62fba21 00007f34e62fad1d 00007f34ef55c204 00007f34ef55c039 00007f34e62dadc6
0809 15:47:23.417943 (+131219us) raft_consensus.cc:1671] finished
0809 15:47:23.417965 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:47:23.418086 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":31837,"prepare.queue_time_us":14680,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":41,"raft.queue_time_us":717900,"raft.run_cpu_time_us":214,"raft.run_wall_time_us":7476,"spinlock_wait_cycles":38440960,"thread_start_us":154184,"threads_started":1,"wal-append.queue_time_us":189597}
I20260809 15:47:23.435422 29094 consensus_queue.cc:237] T 16ca412a0dae4feebe85dd01bf1a9173 P 65c36d164b0540f099d592f38307ee6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:23.438125 23528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a88421dba80d4d3388aa5eb62e23853b" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:23.438719 23528 raft_consensus.cc:2454] T a88421dba80d4d3388aa5eb62e23853b P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 7.
I20260809 15:47:23.406417 28719 raft_consensus.cc:504] T 4b30f28311964a0b9149a50495e83d8f P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } attrs { replace: true } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } }
I20260809 15:47:23.441071 28719 leader_election.cc:288] T 4b30f28311964a0b9149a50495e83d8f P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 7 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:47:23.442395 29152 raft_consensus.cc:482] T e71a05ba1d4d42bd95c3f85f9313399e P 61e69695258f48baa984c9d395e8ce3f [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:23.443048 29152 raft_consensus.cc:504] T e71a05ba1d4d42bd95c3f85f9313399e P 61e69695258f48baa984c9d395e8ce3f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:23.446076 23277 raft_consensus.cc:1261] T 213543cdbe814c89a17a963469922c1b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260809 15:47:23.415256 23519 raft_consensus.cc:1261] T f7715d71dbac447b9351fc5eb36c5e80 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Refusing update from remote peer 67b3aeadfa7a432abe8af4d292e04ab9: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260809 15:47:23.412906 29086 raft_consensus.cc:482] T 05ad7d8fdca64e3da527280ca5be9280 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:47:23.423425 23273 pending_rounds.cc:76] T 763e58d6ba644eea830a6c1664d39c9b P 67b3aeadfa7a432abe8af4d292e04ab9: Aborting all ops after (but not including) 4
I20260809 15:47:23.448720 23273 pending_rounds.cc:97] T 763e58d6ba644eea830a6c1664d39c9b P 67b3aeadfa7a432abe8af4d292e04ab9: Aborting uncommitted NO_OP operation due to leader change: 8.5
I20260809 15:47:23.448997 23273 raft_consensus.cc:2866] T 763e58d6ba644eea830a6c1664d39c9b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 15:47:23.449508 23286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4564dfc6d4094c1b9240b367ccecf3b1" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:23.454162 23228 leader_election.cc:302] T a3622f6e3e3641cea1960e3a31498bb7 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:47:23.411900 28924 leader_election.cc:288] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 11 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:23.457363 23277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07fcabb569414bddbd39df6cedd24e64" candidate_uuid: "c0f38c1908414566aa9115bb2cbdd85d" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:23.440881 28612 raft_consensus.cc:504] T 294d4bcc1e7a477cb56455581277a60f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:23.462306 23277 raft_consensus.cc:1261] T a3622f6e3e3641cea1960e3a31498bb7 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260809 15:47:23.462656 23286 raft_consensus.cc:1261] T 6ae14a74a26f434db08c652e097385c8 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 9 FOLLOWER]: Refusing update from remote peer c0f38c1908414566aa9115bb2cbdd85d: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260809 15:47:23.462988 28612 leader_election.cc:288] T 294d4bcc1e7a477cb56455581277a60f P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 7 election: Requested vote from peers 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:23.412272 29120 leader_election.cc:288] T 998c6cee72cf40228c6dba064c0db2da P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:47:23.424903 28919 consensus_queue.cc:1046] T 16ca412a0dae4feebe85dd01bf1a9173 P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.035s
I20260809 15:47:23.457888 28887 leader_election.cc:288] T 563daf42d7784db195ea99c3cb21b370 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:23.468149 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "294d4bcc1e7a477cb56455581277a60f" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f"
I20260809 15:47:23.468946 24463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "563daf42d7784db195ea99c3cb21b370" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
W20260809 15:47:23.464713 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 43621.7us
I20260809 15:47:23.471990 23286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e71a05ba1d4d42bd95c3f85f9313399e" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" is_pre_election: true
I20260809 15:47:23.472280 28924 raft_consensus.cc:3032] T 16ca412a0dae4feebe85dd01bf1a9173 P 61e69695258f48baa984c9d395e8ce3f [term 7 LEADER]: Stepping down as leader of term 7
I20260809 15:47:23.472903 28924 raft_consensus.cc:729] T 16ca412a0dae4feebe85dd01bf1a9173 P 61e69695258f48baa984c9d395e8ce3f [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 61e69695258f48baa984c9d395e8ce3f, State: Running, Role: LEADER
I20260809 15:47:23.474080 28924 consensus_queue.cc:260] T 16ca412a0dae4feebe85dd01bf1a9173 P 61e69695258f48baa984c9d395e8ce3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:23.469589 24463 raft_consensus.cc:2454] T 563daf42d7784db195ea99c3cb21b370 P 61e69695258f48baa984c9d395e8ce3f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 6.
I20260809 15:47:23.475939 28924 raft_consensus.cc:3037] T 16ca412a0dae4feebe85dd01bf1a9173 P 61e69695258f48baa984c9d395e8ce3f [term 7 FOLLOWER]: Advancing to term 8
I20260809 15:47:23.476608 23223 leader_election.cc:302] T 563daf42d7784db195ea99c3cb21b370 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:47:23.472652 23286 raft_consensus.cc:2454] T e71a05ba1d4d42bd95c3f85f9313399e P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 8.
I20260809 15:47:23.478406 23528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "294d4bcc1e7a477cb56455581277a60f" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c"
I20260809 15:47:23.479365 29120 raft_consensus.cc:482] T cef3e7277d7d4760824a3436ed4a847d P 65c36d164b0540f099d592f38307ee6c [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:23.480083 29120 raft_consensus.cc:504] T cef3e7277d7d4760824a3436ed4a847d P 65c36d164b0540f099d592f38307ee6c [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:47:23.483377 29167 log.cc:924] Time spent T 2a823d35f4f348d1b52ba9e6d80b00d2 P 65c36d164b0540f099d592f38307ee6c: Append to log took a long time: real 0.072s user 0.001s sys 0.000s
I20260809 15:47:23.485713 23528 raft_consensus.cc:2379] T 294d4bcc1e7a477cb56455581277a60f P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 in current term 7: Already voted for candidate 65c36d164b0540f099d592f38307ee6c in this term.
W20260809 15:47:23.480034 29158 log.cc:924] Time spent T 4564dfc6d4094c1b9240b367ccecf3b1 P 67b3aeadfa7a432abe8af4d292e04ab9: Append to log took a long time: real 0.107s user 0.000s sys 0.000s
I20260809 15:47:23.526871 23228 leader_election.cc:302] T a88421dba80d4d3388aa5eb62e23853b P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c36d164b0540f099d592f38307ee6c, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:47:23.483201 29120 leader_election.cc:288] T cef3e7277d7d4760824a3436ed4a847d P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:47:23.465720 29152 leader_election.cc:288] T e71a05ba1d4d42bd95c3f85f9313399e P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:47:23.453716 28456 raft_consensus.cc:504] T 0da4d5283c8a46878bd39b85e5c71f66 P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:23.453197 29068 raft_consensus.cc:504] T ae6d2c08a7a94f58ac8dd698e3fd456c P 65c36d164b0540f099d592f38307ee6c [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:23.452543 29086 raft_consensus.cc:504] T 05ad7d8fdca64e3da527280ca5be9280 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:23.506377 28784 leader_election.cc:288] T cef3e7277d7d4760824a3436ed4a847d P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:24.054129 28462 raft_consensus.cc:482] T 6ae14a74a26f434db08c652e097385c8 P 65c36d164b0540f099d592f38307ee6c [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:24.122965 23521 raft_consensus.cc:1226] T 0da4d5283c8a46878bd39b85e5c71f66 P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Rejecting Update request from peer c0f38c1908414566aa9115bb2cbdd85d for earlier term 7. Current term is 8. Ops: [7.4-7.4]
I20260809 15:47:24.122280 28425 raft_consensus.cc:482] T 0acc47ce21c3422698d1ef7277c3ed63 P 65c36d164b0540f099d592f38307ee6c [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:24.124622 28456 leader_election.cc:288] T 0da4d5283c8a46878bd39b85e5c71f66 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 8 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:47:24.124838 29068 leader_election.cc:288] T ae6d2c08a7a94f58ac8dd698e3fd456c P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 10 election: Requested vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:47:24.125306 29068 raft_consensus.cc:482] T 294d4bcc1e7a477cb56455581277a60f P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:23.487167 28612 raft_consensus.cc:2781] T a3622f6e3e3641cea1960e3a31498bb7 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260809 15:47:24.125725 29068 raft_consensus.cc:504] T 294d4bcc1e7a477cb56455581277a60f P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:24.123973 28425 raft_consensus.cc:504] T 0acc47ce21c3422698d1ef7277c3ed63 P 65c36d164b0540f099d592f38307ee6c [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:24.129016 28425 leader_election.cc:288] T 0acc47ce21c3422698d1ef7277c3ed63 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:47:24.129575 29086 leader_election.cc:288] T 05ad7d8fdca64e3da527280ca5be9280 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:47:23.699774 24468 raft_consensus.cc:2454] T 294d4bcc1e7a477cb56455581277a60f P 61e69695258f48baa984c9d395e8ce3f [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 7.
I20260809 15:47:23.723668 28532 consensus_queue.cc:237] T 907bacfa79d04a55a68c8dbe60056462 P 61e69695258f48baa984c9d395e8ce3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:23.669400 28719 raft_consensus.cc:482] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:24.135931 28719 raft_consensus.cc:504] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:24.102479 28586 raft_consensus.cc:482] T 953eea268e17491ca76f3a555860a512 P 65c36d164b0540f099d592f38307ee6c [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:24.053913 29087 raft_consensus.cc:482] T 99bdb93503db4de4a1cfb0161ea0c07c P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:24.082667 29100 leader_election.cc:288] T 763e58d6ba644eea830a6c1664d39c9b P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:24.145709 28586 raft_consensus.cc:504] T 953eea268e17491ca76f3a555860a512 P 65c36d164b0540f099d592f38307ee6c [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:47:24.197199 23223 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.68:34689, user_credentials={real_user=slave}} blocked reactor thread for 720621us
W20260809 15:47:24.085278 29161 log.cc:924] Time spent T 2db057fd10e149a9895dff6ef8c9f038 P c0f38c1908414566aa9115bb2cbdd85d: Append to log took a long time: real 0.634s user 0.001s sys 0.000s
I20260809 15:47:24.123529 28462 raft_consensus.cc:504] T 6ae14a74a26f434db08c652e097385c8 P 65c36d164b0540f099d592f38307ee6c [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:47:24.193301 23274 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.721s user 0.020s sys 0.018s
W20260809 15:47:24.280385 23274 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.808s user 0.020s sys 0.018s
I20260809 15:47:24.126358 28612 raft_consensus.cc:482] T a3622f6e3e3641cea1960e3a31498bb7 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:47:24.252563 29068 leader_election.cc:288] T 294d4bcc1e7a477cb56455581277a60f P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:24.253338 28719 leader_election.cc:288] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
W20260809 15:47:24.254956 28319 consensus_meta.cc:287] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 67b3aeadfa7a432abe8af4d292e04ab9: Time spent flushing consensus metadata: real 1.049s user 0.007s sys 0.000s
I20260809 15:47:24.257853 28970 raft_consensus.cc:2781] T 563daf42d7784db195ea99c3cb21b370 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260809 15:47:24.117998 28451 raft_consensus.cc:482] T d2d4f749fcfd4fb3bfb8115af3471598 P 65c36d164b0540f099d592f38307ee6c [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:23.716303 29124 raft_consensus.cc:482] T f7715d71dbac447b9351fc5eb36c5e80 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:24.174451 29087 raft_consensus.cc:504] T 99bdb93503db4de4a1cfb0161ea0c07c P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
W20260809 15:47:24.273069 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 746280us
I20260809 15:47:24.295226 28970 raft_consensus.cc:482] T 563daf42d7784db195ea99c3cb21b370 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Starting leader election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:47:24.301833 28970 raft_consensus.cc:3037] T 563daf42d7784db195ea99c3cb21b370 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Advancing to term 7
W20260809 15:47:24.302083 29172 log.cc:924] Time spent T 6ced718c1a5c4da79cb8c1ebd1435b87 P 65c36d164b0540f099d592f38307ee6c: Append to log took a long time: real 0.212s user 0.001s sys 0.000s
I20260809 15:47:24.297894 28621 raft_consensus.cc:2781] T a88421dba80d4d3388aa5eb62e23853b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260809 15:47:24.303993 28621 raft_consensus.cc:482] T a88421dba80d4d3388aa5eb62e23853b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Starting leader election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:47:24.304358 28621 raft_consensus.cc:3037] T a88421dba80d4d3388aa5eb62e23853b P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Advancing to term 8
I20260809 15:47:24.308435 28586 leader_election.cc:288] T 953eea268e17491ca76f3a555860a512 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:47:24.310338 28970 raft_consensus.cc:504] T 563daf42d7784db195ea99c3cb21b370 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:24.312587 28970 leader_election.cc:288] T 563daf42d7784db195ea99c3cb21b370 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 7 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
W20260809 15:47:24.314131 28577 log.cc:924] Time spent T ba56725e42f74a1baa4c04d9660727f9 P 61e69695258f48baa984c9d395e8ce3f: Append to log took a long time: real 0.202s user 0.000s sys 0.000s
I20260809 15:47:24.292953 28612 raft_consensus.cc:3037] T a3622f6e3e3641cea1960e3a31498bb7 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Advancing to term 8
I20260809 15:47:24.327981 23394 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55332 (request call id 1575) took 1793 ms. Trace:
I20260809 15:47:24.330276 23394 rpcz_store.cc:276] 0809 15:47:22.534620 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:22.535210 (+ 590us) service_pool.cc:227] Handling call
0809 15:47:22.535603 (+ 393us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 15:47:23.226500 (+690897us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:47:23.226530 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0809 15:47:23.227051 (+ 521us) log.cc:842] Serialized 52 byte log entry
0809 15:47:23.306695 (+ 79644us) raft_consensus.cc:1631] Marking committed up to 4
0809 15:47:23.306981 (+ 286us) log.cc:842] Serialized 14 byte log entry
0809 15:47:23.417446 (+110465us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:47:23.417514 (+ 68us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 15:47:24.327727 (+910213us) raft_consensus.cc:1671] finished
0809 15:47:24.327760 (+ 33us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:47:24.327906 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":980397,"prepare.queue_time_us":133,"prepare.run_cpu_time_us":31,"prepare.run_wall_time_us":32,"raft.queue_time_us":806142,"raft.run_cpu_time_us":267,"raft.run_wall_time_us":7476,"spinlock_wait_cycles":28487424,"thread_start_us":79166,"threads_started":1,"wal-append.queue_time_us":137285}
I20260809 15:47:24.366186 24448 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38734 (request call id 454) took 2212 ms. Trace:
I20260809 15:47:24.366567 24448 rpcz_store.cc:276] 0809 15:47:22.153282 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:22.153446 (+ 164us) service_pool.cc:227] Handling call
0809 15:47:22.404654 (+251208us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 15:47:22.511098 (+106444us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:47:24.365988 (+1854890us) spinlock_profiling.cc:243] Waited 234 ms on lock 0x7b5400534150. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e921fa09 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:47:24.366126 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":537756928}
W20260809 15:47:24.331351 28873 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.700s user 0.013s sys 0.020s
I20260809 15:47:24.371008 23517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cef3e7277d7d4760824a3436ed4a847d" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:24.371860 23517 raft_consensus.cc:2379] T cef3e7277d7d4760824a3436ed4a847d P 65c36d164b0540f099d592f38307ee6c [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 in current term 9: Already voted for candidate 65c36d164b0540f099d592f38307ee6c in this term.
I20260809 15:47:24.373793 23517 raft_consensus.cc:1261] T 953eea268e17491ca76f3a555860a512 P 65c36d164b0540f099d592f38307ee6c [term 9 FOLLOWER]: Refusing update from remote peer 67b3aeadfa7a432abe8af4d292e04ab9: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260809 15:47:24.375402 23517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "763e58d6ba644eea830a6c1664d39c9b" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:24.380477 23517 raft_consensus.cc:2396] T 763e58d6ba644eea830a6c1664d39c9b P 65c36d164b0540f099d592f38307ee6c [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 for term 10 because replica has last-logged OpId of term: 9 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 5.
W20260809 15:47:24.368888 28873 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.843s user 0.013s sys 0.020s
I20260809 15:47:24.297298 28319 raft_consensus.cc:504] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:24.406841 28319 leader_election.cc:288] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 election: Requested vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:24.299444 28451 raft_consensus.cc:504] T d2d4f749fcfd4fb3bfb8115af3471598 P 65c36d164b0540f099d592f38307ee6c [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:24.385341 28688 raft_consensus.cc:482] T f7715d71dbac447b9351fc5eb36c5e80 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:24.376578 29201 consensus_queue.cc:1035] T f7715d71dbac447b9351fc5eb36c5e80 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.016s
I20260809 15:47:24.390425 28309 raft_consensus.cc:482] T 00a1881267a14ba8b78c81040f17a014 P c0f38c1908414566aa9115bb2cbdd85d [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:24.391640 23396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3296ef50be254fc29cfc77d22a8f7a23" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:47:24.391840 28934 raft_consensus.cc:482] T 8e4ffcf83c234ae7910ddef3ec0b4bda P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:24.392254 28719 raft_consensus.cc:482] T 07fcabb569414bddbd39df6cedd24e64 P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
I20260809 15:47:24.391533 28248 raft_consensus.cc:482] T d3be1e356b254d22b995be32f1c12f6f P c0f38c1908414566aa9115bb2cbdd85d [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
I20260809 15:47:24.393698 23400 raft_consensus.cc:1226] T 1f9d6565aa4c45a3b3f01210d0eb71f7 P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Rejecting Update request from peer 61e69695258f48baa984c9d395e8ce3f for earlier term 4. Current term is 5. Ops: []
I20260809 15:47:24.394276 23397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7660dfa2ef694927b8eda4100a2bcd9b" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:47:24.395392 28523 raft_consensus.cc:482] T 4b30f28311964a0b9149a50495e83d8f P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:24.392966 28546 raft_consensus.cc:482] T 3296ef50be254fc29cfc77d22a8f7a23 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:24.393626 29058 raft_consensus.cc:482] T 0c29937a80724ddb9d07a27534a146f3 P c0f38c1908414566aa9115bb2cbdd85d [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
I20260809 15:47:24.396417 28941 raft_consensus.cc:482] T 7660dfa2ef694927b8eda4100a2bcd9b P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:24.396474 23392 raft_consensus.cc:2932] T ba56725e42f74a1baa4c04d9660727f9 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Committing config change with OpId 5.4: config changed from index -1 to 4, NON_VOTER 61e69695258f48baa984c9d395e8ce3f (127.20.202.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } attrs { replace: true } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } } }
I20260809 15:47:24.397926 28462 leader_election.cc:288] T 6ae14a74a26f434db08c652e097385c8 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:47:24.300320 29124 raft_consensus.cc:504] T f7715d71dbac447b9351fc5eb36c5e80 P 65c36d164b0540f099d592f38307ee6c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:24.399653 23402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e4ffcf83c234ae7910ddef3ec0b4bda" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:24.392298 23399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3e94b7e50754a81b3d72a07476cb7b9" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:24.393126 23393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "907bacfa79d04a55a68c8dbe60056462" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:47:24.407395 23282 raft_consensus.cc:3037] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Advancing to term 6
W20260809 15:47:24.407862 29153 log.cc:924] Time spent T 0a51c0baaa6e432997c9d3552af4f254 P 67b3aeadfa7a432abe8af4d292e04ab9: Append to log took a long time: real 1.032s user 0.005s sys 0.008s
I20260809 15:47:24.428526 28248 raft_consensus.cc:504] T d3be1e356b254d22b995be32f1c12f6f P c0f38c1908414566aa9115bb2cbdd85d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:24.435108 28248 leader_election.cc:288] T d3be1e356b254d22b995be32f1c12f6f P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:24.410972 29087 leader_election.cc:288] T 99bdb93503db4de4a1cfb0161ea0c07c P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:47:24.367955 29068 raft_consensus.cc:482] T 00a1881267a14ba8b78c81040f17a014 P 65c36d164b0540f099d592f38307ee6c [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:24.425997 23400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e4ffcf83c234ae7910ddef3ec0b4bda" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:47:24.419438 28719 raft_consensus.cc:504] T 07fcabb569414bddbd39df6cedd24e64 P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:24.418799 28309 raft_consensus.cc:504] T 00a1881267a14ba8b78c81040f17a014 P c0f38c1908414566aa9115bb2cbdd85d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:24.441216 28451 leader_election.cc:288] T d2d4f749fcfd4fb3bfb8115af3471598 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:47:24.417270 28688 raft_consensus.cc:504] T f7715d71dbac447b9351fc5eb36c5e80 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:24.441887 28719 leader_election.cc:288] T 07fcabb569414bddbd39df6cedd24e64 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:47:24.442044 28309 leader_election.cc:288] T 00a1881267a14ba8b78c81040f17a014 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:24.418648 28934 raft_consensus.cc:504] T 8e4ffcf83c234ae7910ddef3ec0b4bda P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:24.442988 28674 raft_consensus.cc:3037] T 4bcac4b24e7a4448abdc934bba5a624e P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Advancing to term 5
I20260809 15:47:24.443213 28688 leader_election.cc:288] T f7715d71dbac447b9351fc5eb36c5e80 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:24.443931 28934 leader_election.cc:288] T 8e4ffcf83c234ae7910ddef3ec0b4bda P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:24.429498 28941 raft_consensus.cc:504] T 7660dfa2ef694927b8eda4100a2bcd9b P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:24.445782 28941 leader_election.cc:288] T 7660dfa2ef694927b8eda4100a2bcd9b P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:24.447544 23397 raft_consensus.cc:2379] T 7660dfa2ef694927b8eda4100a2bcd9b P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 61e69695258f48baa984c9d395e8ce3f in current term 7: Already voted for candidate c0f38c1908414566aa9115bb2cbdd85d in this term.
I20260809 15:47:24.447927 29068 raft_consensus.cc:504] T 00a1881267a14ba8b78c81040f17a014 P 65c36d164b0540f099d592f38307ee6c [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:24.448969 23518 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56770 (request call id 1556) took 1561 ms. Trace:
I20260809 15:47:24.428408 28546 raft_consensus.cc:504] T 3296ef50be254fc29cfc77d22a8f7a23 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:24.449328 23518 rpcz_store.cc:276] 0809 15:47:22.887337 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:22.887948 (+ 611us) service_pool.cc:227] Handling call
0809 15:47:22.888272 (+ 324us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 15:47:22.934429 (+ 46157us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:47:22.934457 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 15:47:22.943554 (+ 9097us) log.cc:842] Serialized 26 byte log entry
0809 15:47:22.990484 (+ 46930us) raft_consensus.cc:1631] Marking committed up to 4
0809 15:47:22.990567 (+ 83us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:47:24.448586 (+1458019us) spinlock_profiling.cc:243] Waited 22.5 ms on lock 0x7b54000758d4. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e9220f51 00007f34e921f99e 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772 00007f34e764a94c 00007f34e7649d86
0809 15:47:24.448632 (+ 46us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 15:47:24.448684 (+ 52us) raft_consensus.cc:1671] finished
0809 15:47:24.448696 (+ 12us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:47:24.448910 (+ 214us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3137,"raft.queue_time_us":902,"raft.run_cpu_time_us":99,"raft.run_wall_time_us":5362,"spinlock_wait_cycles":51811072,"thread_start_us":64483,"threads_started":2,"wal-append.queue_time_us":46817,"wal-append.run_cpu_time_us":2229,"wal-append.run_wall_time_us":1003371}
I20260809 15:47:24.451418 28546 leader_election.cc:288] T 3296ef50be254fc29cfc77d22a8f7a23 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:24.428516 29058 raft_consensus.cc:504] T 0c29937a80724ddb9d07a27534a146f3 P c0f38c1908414566aa9115bb2cbdd85d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:24.452134 23276 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55434 (request call id 357) took 1407 ms. Trace:
I20260809 15:47:24.452553 23276 rpcz_store.cc:276] 0809 15:47:23.044389 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:23.044816 (+ 427us) service_pool.cc:227] Handling call
0809 15:47:24.451924 (+1407108us) spinlock_profiling.cc:243] Waited 173 ms on lock 0x7b5400137ed4. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e92242ae 00007f34ee161c27 00007f34e7849c04 00007f34e7849b92 00007f34e7849ac1 00007f34e7849a3c 00007f34e7848cf2 00007f34e764a94c 00007f34e7649d86 00007f34e764972f
0809 15:47:24.452070 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":398482176}
I20260809 15:47:24.453585 29058 leader_election.cc:288] T 0c29937a80724ddb9d07a27534a146f3 P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:24.456288 23521 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56766 (request call id 2257) took 1185 ms. Trace:
I20260809 15:47:24.456688 23521 rpcz_store.cc:276] 0809 15:47:23.270972 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:23.271095 (+ 123us) service_pool.cc:227] Handling call
0809 15:47:23.271435 (+ 340us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 15:47:24.123379 (+851944us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:47:24.456035 (+332656us) spinlock_profiling.cc:243] Waited 851 ms on lock 0x7b54001081d4. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34e91a5471 00007f34e9220f51 00007f34e921f99e 00007f34ee1612e9 00007f34e7847684 00007f34e7847612 00007f34e7847541 00007f34e78474bc 00007f34e7846772 00007f34e764a94c 00007f34e7649d86
0809 15:47:24.456226 (+ 191us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1957791104}
I20260809 15:47:24.424194 28319 raft_consensus.cc:482] T 998c6cee72cf40228c6dba064c0db2da P 67b3aeadfa7a432abe8af4d292e04ab9 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:24.419708 28523 raft_consensus.cc:504] T 4b30f28311964a0b9149a50495e83d8f P c0f38c1908414566aa9115bb2cbdd85d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } attrs { replace: true } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: NON_VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: true } }
I20260809 15:47:24.431522 23393 raft_consensus.cc:3037] T 907bacfa79d04a55a68c8dbe60056462 P c0f38c1908414566aa9115bb2cbdd85d [term 9 FOLLOWER]: Advancing to term 10
I20260809 15:47:24.438442 23399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a823d35f4f348d1b52ba9e6d80b00d2" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d"
I20260809 15:47:24.458868 23396 raft_consensus.cc:3037] T 3296ef50be254fc29cfc77d22a8f7a23 P c0f38c1908414566aa9115bb2cbdd85d [term 3 FOLLOWER]: Advancing to term 4
I20260809 15:47:24.459625 23402 raft_consensus.cc:2454] T 8e4ffcf83c234ae7910ddef3ec0b4bda P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 5.
W20260809 15:47:24.437352 23472 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.20.202.66:36263 timed out after 1.615s (SENT) [suppressed 25 similar messages]
I20260809 15:47:24.462203 23400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e71a05ba1d4d42bd95c3f85f9313399e" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "c0f38c1908414566aa9115bb2cbdd85d" is_pre_election: true
I20260809 15:47:24.468839 24406 leader_election.cc:302] T e71a05ba1d4d42bd95c3f85f9313399e P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:47:24.436967 28784 raft_consensus.cc:482] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:24.436474 29124 leader_election.cc:288] T f7715d71dbac447b9351fc5eb36c5e80 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:47:24.496160 23282 raft_consensus.cc:2454] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 67b3aeadfa7a432abe8af4d292e04ab9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 6.
I20260809 15:47:24.449754 29068 leader_election.cc:288] T 00a1881267a14ba8b78c81040f17a014 P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263)
I20260809 15:47:24.497848 24453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "563daf42d7784db195ea99c3cb21b370" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f"
I20260809 15:47:24.499718 24453 raft_consensus.cc:3037] T 563daf42d7784db195ea99c3cb21b370 P 61e69695258f48baa984c9d395e8ce3f [term 6 FOLLOWER]: Advancing to term 7
I20260809 15:47:24.500751 23399 raft_consensus.cc:1226] T 4bcac4b24e7a4448abdc934bba5a624e P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Rejecting Update request from peer 61e69695258f48baa984c9d395e8ce3f for earlier term 4. Current term is 5. Ops: []
I20260809 15:47:24.458374 28319 raft_consensus.cc:504] T 998c6cee72cf40228c6dba064c0db2da P 67b3aeadfa7a432abe8af4d292e04ab9 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:24.493221 23223 leader_election.cc:302] T 294d4bcc1e7a477cb56455581277a60f P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 67b3aeadfa7a432abe8af4d292e04ab9; no voters: 65c36d164b0540f099d592f38307ee6c
I20260809 15:47:24.508108 24468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e4ffcf83c234ae7910ddef3ec0b4bda" candidate_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f"
I20260809 15:47:24.508855 24468 raft_consensus.cc:2354] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 61e69695258f48baa984c9d395e8ce3f [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 5. Current term is 6.
I20260809 15:47:24.519614 23396 raft_consensus.cc:2454] T 3296ef50be254fc29cfc77d22a8f7a23 P c0f38c1908414566aa9115bb2cbdd85d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 4.
I20260809 15:47:24.522912 23393 raft_consensus.cc:2454] T 907bacfa79d04a55a68c8dbe60056462 P c0f38c1908414566aa9115bb2cbdd85d [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 10.
I20260809 15:47:24.497887 28756 raft_consensus.cc:2781] T e71a05ba1d4d42bd95c3f85f9313399e P 61e69695258f48baa984c9d395e8ce3f [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260809 15:47:24.532164 24448 raft_consensus.cc:1261] T 0a51c0baaa6e432997c9d3552af4f254 P 61e69695258f48baa984c9d395e8ce3f [term 9 FOLLOWER]: Refusing update from remote peer 67b3aeadfa7a432abe8af4d292e04ab9: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260809 15:47:24.543473 28280 raft_consensus.cc:482] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:47:24.545315 28280 raft_consensus.cc:504] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:24.464677 29196 consensus_queue.cc:1035] T 953eea268e17491ca76f3a555860a512 P 67b3aeadfa7a432abe8af4d292e04ab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.065s
I20260809 15:47:24.547384 24455 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38734 (request call id 458) took 1431 ms. Trace:
I20260809 15:47:24.547508 28280 leader_election.cc:288] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 61e69695258f48baa984c9d395e8ce3f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:24.548141 29196 raft_consensus.cc:482] T 6ae14a74a26f434db08c652e097385c8 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:24.548616 29196 raft_consensus.cc:504] T 6ae14a74a26f434db08c652e097385c8 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:24.547760 24455 rpcz_store.cc:276] 0809 15:47:23.115646 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:23.115811 (+ 165us) service_pool.cc:227] Handling call
0809 15:47:23.116270 (+ 459us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 15:47:23.304505 (+188235us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 15:47:23.347258 (+ 42753us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 15:47:23.380788 (+ 33530us) log.cc:842] Serialized 26 byte log entry
0809 15:47:23.904376 (+523588us) raft_consensus.cc:1631] Marking committed up to 10
0809 15:47:24.498523 (+594147us) log.cc:842] Serialized 14 byte log entry
0809 15:47:24.515491 (+ 16968us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:47:24.521952 (+ 6461us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 15:47:24.525340 (+ 3388us) raft_consensus.cc:1671] finished
0809 15:47:24.532076 (+ 6736us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:47:24.547324 (+ 15248us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":9465,"raft.run_cpu_time_us":277,"raft.run_wall_time_us":13039,"thread_start_us":803639,"threads_started":3,"wal-append.queue_time_us":845640}
I20260809 15:47:24.566176 24453 raft_consensus.cc:2454] T 563daf42d7784db195ea99c3cb21b370 P 61e69695258f48baa984c9d395e8ce3f [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67b3aeadfa7a432abe8af4d292e04ab9 in term 7.
I20260809 15:47:24.550384 28523 leader_election.cc:288] T 4b30f28311964a0b9149a50495e83d8f P c0f38c1908414566aa9115bb2cbdd85d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 67b3aeadfa7a432abe8af4d292e04ab9 (127.20.202.65:35459), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:24.573225 29200 raft_consensus.cc:482] T 05ad7d8fdca64e3da527280ca5be9280 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:24.574900 29200 raft_consensus.cc:504] T 05ad7d8fdca64e3da527280ca5be9280 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:24.498577 28784 raft_consensus.cc:504] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 67b3aeadfa7a432abe8af4d292e04ab9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:24.531735 28756 raft_consensus.cc:482] T e71a05ba1d4d42bd95c3f85f9313399e P 61e69695258f48baa984c9d395e8ce3f [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:47:24.531737 28873 raft_consensus.cc:482] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 61e69695258f48baa984c9d395e8ce3f [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
W20260809 15:47:24.678263 23468 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.558s user 0.009s sys 0.034s
W20260809 15:47:24.851066 23468 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.730s user 0.009s sys 0.051s
W20260809 15:47:24.716531 23472 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.66:36263, user_credentials={real_user=slave}} blocked reactor thread for 279192us
W20260809 15:47:24.740031 23228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 332252us
I20260809 15:47:24.773752 23404 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55332 (request call id 1576) took 2224 ms. Trace:
W20260809 15:47:24.599922 29184 log.cc:924] Time spent T 7660dfa2ef694927b8eda4100a2bcd9b P 61e69695258f48baa984c9d395e8ce3f: Append to log took a long time: real 0.056s user 0.000s sys 0.000s
W20260809 15:47:24.643776 28577 log.cc:924] Time spent T ba56725e42f74a1baa4c04d9660727f9 P 61e69695258f48baa984c9d395e8ce3f: Append to log took a long time: real 0.057s user 0.000s sys 0.000s
I20260809 15:47:24.586862 28756 raft_consensus.cc:3037] T e71a05ba1d4d42bd95c3f85f9313399e P 61e69695258f48baa984c9d395e8ce3f [term 8 FOLLOWER]: Advancing to term 9
I20260809 15:47:24.717957 28745 raft_consensus.cc:482] T 4bcac4b24e7a4448abdc934bba5a624e P 61e69695258f48baa984c9d395e8ce3f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:24.779920 28521 raft_consensus.cc:482] T 907bacfa79d04a55a68c8dbe60056462 P c0f38c1908414566aa9115bb2cbdd85d [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 15:47:24.874383 23223 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.202.68:34689, user_credentials={real_user=slave}} blocked reactor thread for 381226us
I20260809 15:47:24.779800 29200 leader_election.cc:288] T 05ad7d8fdca64e3da527280ca5be9280 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:24.796957 29061 raft_consensus.cc:482] T d0721a4403a440f383318a3cea9d1442 P c0f38c1908414566aa9115bb2cbdd85d [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
W20260809 15:47:24.868175 24410 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.202.67:40553, user_credentials={real_user=slave}} blocked reactor thread for 324463us
I20260809 15:47:24.679212 29191 raft_consensus.cc:482] T 16ca412a0dae4feebe85dd01bf1a9173 P 61e69695258f48baa984c9d395e8ce3f [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:24.643671 29199 raft_consensus.cc:2781] T 294d4bcc1e7a477cb56455581277a60f P 67b3aeadfa7a432abe8af4d292e04ab9 [term 7 FOLLOWER]: Leader election won for term 7
I20260809 15:47:24.855942 23404 rpcz_store.cc:276] 0809 15:47:22.548926 (+ 0us) service_pool.cc:170] Inserting onto call queue
0809 15:47:22.549921 (+ 995us) service_pool.cc:227] Handling call
0809 15:47:22.577478 (+ 27557us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 15:47:23.258343 (+680865us) raft_consensus.cc:1520] Early marking committed up to index 3
0809 15:47:23.277660 (+ 19317us) log.cc:842] Serialized 14 byte log entry
0809 15:47:23.417714 (+140054us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0809 15:47:23.453208 (+ 35494us) log.cc:842] Serialized 52 byte log entry
0809 15:47:23.453343 (+ 135us) raft_consensus.cc:1631] Marking committed up to 4
0809 15:47:23.453561 (+ 218us) log.cc:842] Serialized 14 byte log entry
0809 15:47:24.328068 (+874507us) spinlock_profiling.cc:243] Waited 3.01 ms on lock 0x7b1800266380. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34ef1978b1 00007f34ef1a4661 00007f34e62f8764 00007f34e62fbb11 00007f34e62fbac9 00007f34e62fba59 00007f34e62fba21 00007f34e62fad1d 00007f34ef55c204 00007f34ef55c039 00007f34e62dadc6
0809 15:47:24.773260 (+445192us) spinlock_profiling.cc:243] Waited 14.9 ms on lock 0x7b1800266380. stack: 00007f34e62c6033 00007f34e62c5f30 00007f34e5c24534 00007f34ef9308b4 00007f34ef930849 00007f34ef1978b1 00007f34ef1a4661 00007f34e626f4ca 00007f34ef950f99 00007f34e6133990 00007f34e62f5864 00007f34e62f5155 00007f34e92183d0 00007f34e922a576 00007f34e923b0d7 00007f34e923b058
0809 15:47:24.773440 (+ 180us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 15:47:24.773511 (+ 71us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 15:47:24.773556 (+ 45us) raft_consensus.cc:1671] finished
0809 15:47:24.773568 (+ 12us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 15:47:24.773692 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1683674,"prepare.queue_time_us":22698,"prepare.run_cpu_time_us":83,"prepare.run_wall_time_us":4809,"raft.queue_time_us":3066635,"raft.run_cpu_time_us":485,"raft.run_wall_time_us":56278,"spinlock_wait_cycles":43289088,"thread_start_us":49721,"threads_started":1,"wal-append.queue_time_us":179412,"wal-append.run_cpu_time_us":4277,"wal-append.run_wall_time_us":1011631}
I20260809 15:47:24.698308 28319 leader_election.cc:288] T 998c6cee72cf40228c6dba064c0db2da P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:24.783123 28230 raft_consensus.cc:482] T 4bcac4b24e7a4448abdc934bba5a624e P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:24.811728 29152 raft_consensus.cc:482] T 3924b648461e463b9d7925e5671d68ae P 61e69695258f48baa984c9d395e8ce3f [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:47:24.881738 24448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5712f6b6fffc4cc5a8ddfbe63e4097df" candidate_uuid: "65c36d164b0540f099d592f38307ee6c" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "61e69695258f48baa984c9d395e8ce3f" is_pre_election: true
I20260809 15:47:24.884652 23223 leader_election.cc:398] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 election: Vote denied by peer 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689) with higher term. Message: Invalid argument: T 8e4ffcf83c234ae7910ddef3ec0b4bda P 61e69695258f48baa984c9d395e8ce3f [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 67b3aeadfa7a432abe8af4d292e04ab9 for earlier term 5. Current term is 6.
I20260809 15:47:24.858825 29196 leader_election.cc:288] T 6ae14a74a26f434db08c652e097385c8 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
W20260809 15:47:24.870440 29156 log.cc:924] Time spent T ba56725e42f74a1baa4c04d9660727f9 P c0f38c1908414566aa9115bb2cbdd85d: Append to log took a long time: real 1.457s user 0.000s sys 0.021s
I20260809 15:47:24.863746 28456 raft_consensus.cc:482] T c0741ab44d99439abc3aa3bdd27ee160 P 65c36d164b0540f099d592f38307ee6c [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
I20260809 15:47:24.887547 23518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5712f6b6fffc4cc5a8ddfbe63e4097df" candidate_uuid: "61e69695258f48baa984c9d395e8ce3f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "65c36d164b0540f099d592f38307ee6c" is_pre_election: true
I20260809 15:47:24.900251 24448 raft_consensus.cc:2454] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 61e69695258f48baa984c9d395e8ce3f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c36d164b0540f099d592f38307ee6c in term 4.
I20260809 15:47:24.899698 23223 leader_election.cc:401] T 8e4ffcf83c234ae7910ddef3ec0b4bda P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
I20260809 15:47:24.905702 23518 raft_consensus.cc:2454] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61e69695258f48baa984c9d395e8ce3f in term 5.
I20260809 15:47:24.910111 23468 leader_election.cc:302] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 65c36d164b0540f099d592f38307ee6c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 65c36d164b0540f099d592f38307ee6c; no voters:
I20260809 15:47:24.916306 29085 raft_consensus.cc:2781] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260809 15:47:24.916736 29085 raft_consensus.cc:482] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 15:47:24.917062 29085 raft_consensus.cc:3037] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 65c36d164b0540f099d592f38307ee6c [term 5 FOLLOWER]: Advancing to term 6
I20260809 15:47:24.917689 29196 raft_consensus.cc:482] T 28e17dcb5efe45b8b6a8c42da03b4819 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:24.918190 29196 raft_consensus.cc:504] T 28e17dcb5efe45b8b6a8c42da03b4819 P 67b3aeadfa7a432abe8af4d292e04ab9 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65" port: 35459 } } peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } }
I20260809 15:47:24.919934 23223 leader_election.cc:302] T 563daf42d7784db195ea99c3cb21b370 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61e69695258f48baa984c9d395e8ce3f, 67b3aeadfa7a432abe8af4d292e04ab9; no voters:
I20260809 15:47:24.920262 29196 leader_election.cc:288] T 28e17dcb5efe45b8b6a8c42da03b4819 P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers c0f38c1908414566aa9115bb2cbdd85d (127.20.202.66:36263), 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553)
I20260809 15:47:24.921152 28688 raft_consensus.cc:482] T 472816994d814cb198b69e5248697440 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 61e69695258f48baa984c9d395e8ce3f)
I20260809 15:47:24.921823 29217 raft_consensus.cc:482] T d1a134ba33ea4076bbaac0ae460c17ba P 67b3aeadfa7a432abe8af4d292e04ab9 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:47:24.921650 28688 raft_consensus.cc:504] T 472816994d814cb198b69e5248697440 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:24.922968 28921 raft_consensus.cc:482] T ba56725e42f74a1baa4c04d9660727f9 P c0f38c1908414566aa9115bb2cbdd85d [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 67b3aeadfa7a432abe8af4d292e04ab9)
I20260809 15:47:24.876974 29031 raft_consensus.cc:482] T 30c1740ef6874df0a332fbd6fd4b03ce P c0f38c1908414566aa9115bb2cbdd85d [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 65c36d164b0540f099d592f38307ee6c)
I20260809 15:47:24.882287 29027 raft_consensus.cc:482] T 99bdb93503db4de4a1cfb0161ea0c07c P c0f38c1908414566aa9115bb2cbdd85d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:24.883040 28784 leader_election.cc:288] T 5712f6b6fffc4cc5a8ddfbe63e4097df P 67b3aeadfa7a432abe8af4d292e04ab9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 65c36d164b0540f099d592f38307ee6c (127.20.202.67:40553), 61e69695258f48baa984c9d395e8ce3f (127.20.202.68:34689)
I20260809 15:47:24.899480 28447 raft_consensus.cc:482] T d2d4f749fcfd4fb3bfb8115af3471598 P c0f38c1908414566aa9115bb2cbdd85d [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:24.881189 29068 raft_consensus.cc:482] T 538a228ddfb54bc6a36528d5ff07597b P 65c36d164b0540f099d592f38307ee6c [term 8 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:47:24.899132 29124 raft_consensus.cc:482] T 2a823d35f4f348d1b52ba9e6d80b00d2 P 65c36d164b0540f099d592f38307ee6c [term 10 FOLLOWER]: Starting pre-election (detected failure of leader c0f38c1908414566aa9115bb2cbdd85d)
I20260809 15:47:24.853700 28873 raft_consensus.cc:504] T b3aa6a9a52804df0ba9b0ca3c82f38c3 P 61e69695258f48baa984c9d395e8ce3f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c0f38c1908414566aa9115bb2cbdd85d" member_type: VOTER last_known_addr { host: "127.20.202.66" port: 36263 } } peers { permanent_uuid: "65c36d164b0540f099d592f38307ee6c" member_type: VOTER last_known_addr { host: "127.20.202.67" port: 40553 } } peers { permanent_uuid: "61e69695258f48baa984c9d395e8ce3f" member_type: VOTER last_known_addr { host: "127.20.202.68" port: 34689 } attrs { promote: false } }
I20260809 15:47:24.911648 28248 raft_consensus.cc:482] T 953eea268e17491ca76f3a555860a512 P c0f38c1908414566aa9115bb2cbdd85d [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 15:47:25.003754 28248 raft_consensus.cc:504] T 953eea268e17491ca76f3a555860a512 P c0f38c1908414566aa9115bb2cbdd85d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67b3aeadfa7a432abe8af4d292e04ab9" member_type: VOTER last_known_addr { host: "127.20.202.65"